cloud-fan commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1162257020
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -176,6 +186,23 @@ trait FileFormat {
* By default all field name is supported.
*/
def supportFieldName(name: String): Boolean = true
+
+ /**
+ * All fields the file format's _metadata struct defines.
+ *
+ * Each field's metadata should define [[METADATA_COL_ATTR_KEY]],
+ * [[FILE_SOURCE_METADATA_COL_ATTR_KEY]], and either
+ * [[FILE_SOURCE_CONSTANT_METADATA_COL_ATTR_KEY]] or
+ * [[FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY]] as appropriate.
+ *
+ * Constant attributes will be extracted automatically from
+ * [[PartitionedFile.extraConstantMetadataColumnValues]], while generated
metadata columns always
+ * map to some hidden/internal column the underslying reader provides.
+ *
+ * NOTE: It is not possible to change the semantics of the base metadata
fields by overriding this
+ * method. Technically, a file format could choose suppress them, but that
is not recommended.
+ */
+ def metadataSchemaFields: Seq[StructField] = FileFormat.BASE_METADATA_FIELDS
Review Comment:
Thinking about it more, how can a file source define custom constant
metadata columns? The file listing logic is shared for all file sources and I
can't think of a way to customize it for certain file sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
