cloud-fan commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-522412152 > Then shouldn't we update the table statistics after it computes first time, and next time onwards we can directly gets from the catalog table `fallBackToHDFS` is a runtime config, which means users can change it at any time. If we update the table statistics with the HDFS size, we need a way to invalidate the table stats if `fallBackToHDFS` is set to false later. This seems over-complicated to me.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
