xuanyuanking commented on issue #23327: [SPARK-26222][SQL] Track file listing time URL: https://github.com/apache/spark/pull/23327#issuecomment-449959400 @rxin Hi Reynold, after discussing with Wenchen offline, there's a problem we need to sync with you. Is it permitted to track the file listing time by QueryPlanningTracker? - If the answer is yes, we can create phase summary at caller side and pass the result to the node by thread local QueryPlanningTracker. But in this way, we use the QueryPlanningTracker to track the per-plan node, and the key of the phase map should be a table identity in consideration of join/union scenario. That's also the reason for closing #23298. - If the answer is no, we need to pass the phase summary to scan node by another way, in this PR it is done by adding interface in FileIndex.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
