ryan-johnson-databricks commented on code in PR #40885:
URL: https://github.com/apache/spark/pull/40885#discussion_r1175452917
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -203,6 +203,21 @@ trait FileFormat {
* method. Technically, a file format could choose suppress them, but that
is not recommended.
*/
def metadataSchemaFields: Seq[StructField] = FileFormat.BASE_METADATA_FIELDS
+
+ /**
+ * The extractors to use when deriving file-constant metadata columns for
this file format.
+ *
+ * By default, the value of a file-constant metadata column is obtained by
looking up the column's
+ * name in the file's metadata column value map. However, implementations
can override this method
+ * in order to provide an extractor that has access to the entire
[[PartitionedFile]] when
+ * deriving the column's value.
+ *
+ * NOTE: Extractors are lazy, invoked only if the query actually selects
their column at runtime.
+ *
+ * See also [[FileFormat.getFileConstantMetadataColumnValue]].
+ */
+ def fileConstantMetadataExtractors: Map[String, PartitionedFile => Any] =
Review Comment:
> Can we add more comments in the new API, saying that it may only handle
part of the constant metadata columns? The rest of them should be handled by a
custom FileIndex.
There was already one comment to that effect in the PR that added custom
metadata column support:
https://github.com/apache/spark/pull/40677/files#diff-2d7ec6604810e1be58ca7852ec389853c143b684894f2390ffe07f96701d2219R29
And the unit tests also are a pretty clear demonstration.
Good point that the comments are buried pretty deep tho -- I'll try to
add/expand them in FileFormat.scala as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
