cloud-fan commented on issue #24991: [SPARK-28188] Materialize Dataframe API URL: https://github.com/apache/spark/pull/24991#issuecomment-511712741 I've spent more time understanding the use case, and think table cache should be a better choice here 1. disk vs memory: you can set the storage level to disk-only with more than one copy, which is more reliable than shuffle files. 2. shuffle service: it supports RDD blocks as well thanks to https://github.com/apache/spark/pull/24499 In addition, the table cache has more advantages: 1. It can work for any dataframes, even without shuffles 2. other queries can benefit from table cache implicitly You do have a point that table cache is lazy, but we can add a flag to control it. e.g. `def cache(eager: Boolean = false)`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
