xuanyuanking commented on a change in pull request #23298: [SPARK-26222][SQL]
Track file listing time
URL: https://github.com/apache/spark/pull/23298#discussion_r241952397
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala
##########
@@ -72,14 +72,4 @@ trait FileIndex {
/** Schema of the partitioning columns, or the empty schema if the table is
not partitioned. */
def partitionSchema: StructType
-
- /**
- * Returns an optional metadata operation time, in nanoseconds, for listing
files.
- *
- * We do file listing in query optimization (in order to get the proper
statistics) and we want
- * to account for file listing time in physical execution (as metrics). To
do that, we save the
- * file listing time in some implementations and physical execution calls it
in this method
- * to update the metrics.
- */
- def metadataOpsTimeNs: Option[Long] = None
Review comment:
This is part of
https://github.com/apache/spark/pull/23298/commits/73bf55389466a553606d1878848c0bb018b36dc2,
here I revert the whole changes in
[SPARK-20151](https://github.com/apache/spark/pull/17476/files). Maybe use the
QueryPlanningTracker to measure the pruning phase will have a better code base
in `FileSourceScanExec`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
