vgankidi commented on issue #24991: [SPARK-28188] Materialize Dataframe API URL: https://github.com/apache/spark/pull/24991#issuecomment-508554739 @rxin Yes, materializing on a query plan without a shuffle just wastes resources. We usually recommend repartitioning the dataframe before invoking materialize to add a shuffle. I think this can be used for your use case of measuring execution time of a query plan. What name would you suggest for this api?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
