cloud-fan edited a comment on issue #24991: [SPARK-28188] Materialize Dataframe API URL: https://github.com/apache/spark/pull/24991#issuecomment-511712741 I've spent more time understanding the use case, and think table cache should be a better choice here 1. disk vs memory: you can set the storage level to disk-only with more than one copy, which is more reliable than shuffle files. 2. shuffle service: it supports RDD blocks as well thanks to https://github.com/apache/spark/pull/24499 In addition, the table cache has more advantages: 1. It can work for any dataframes, even without shuffles 2. other queries can benefit from table cache automatically You do have a point that table cache is lazy, but we can add a flag to control it. e.g. `def cache(eager: Boolean = false)`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
