rxin commented on a change in pull request #23327: [SPARK-26222][SQL] Track
file listing time
URL: https://github.com/apache/spark/pull/23327#discussion_r242662526
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala
##########
@@ -74,12 +74,11 @@ trait FileIndex {
def partitionSchema: StructType
/**
- * Returns an optional metadata operation time, in nanoseconds, for listing
files.
+ * Returns an optional file listing phase summary.
*
- * We do file listing in query optimization (in order to get the proper
statistics) and we want
- * to account for file listing time in physical execution (as metrics). To
do that, we save the
- * file listing time in some implementations and physical execution calls it
in this method
- * to update the metrics.
+ * We call this in physical execution while we want to account for the file
listing time as
+ * metrics. If partition pruning happened in query planning, the phase also
contains this
+ * part of the cost, otherwise, it only contains file listing time of
FileIndex initialize.
*/
- def metadataOpsTimeNs: Option[Long] = None
Review comment:
I'd keep this to avoid changing too many. Just make it a function to return
a value based on fileListingPhase
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
