Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17476#discussion_r109092194
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala
---
@@ -72,4 +72,14 @@ trait FileIndex {
/** Schema of the partitioning columns, or the empty schema if the table
is not partitioned. */
def partitionSchema: StructType
+
+ /**
+ * Returns an optional metadata operation time, in nanoseconds, for
listing files.
+ *
+ * We do file listing in query optimization (in order to get the proper
statistics) and we want
+ * to account for file listing time in physical execution (as metrics).
To do that, we save the
+ * file listing time in some implementations and physical execution
calls it in this method
+ * to update the metrics.
+ */
+ def metadataOpsTimeNs: Option[Long] = None
--- End diff --
I thought about that but there is no API level guarantee that we'd get
PrunedInMemoryFileIndex after partition pruning. It is more just a current
implementation detail. I'd rather have something that's more specified in the
API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
