cloud-fan commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1162254984
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -176,6 +186,23 @@ trait FileFormat {
* By default all field name is supported.
*/
def supportFieldName(name: String): Boolean = true
+
+ /**
+ * All fields the file format's _metadata struct defines.
+ *
+ * Each field's metadata should define [[METADATA_COL_ATTR_KEY]],
+ * [[FILE_SOURCE_METADATA_COL_ATTR_KEY]], and either
+ * [[FILE_SOURCE_CONSTANT_METADATA_COL_ATTR_KEY]] or
+ * [[FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY]] as appropriate.
+ *
+ * Constant attributes will be extracted automatically from
+ * [[PartitionedFile.extraConstantMetadataColumnValues]], while generated
metadata columns always
+ * map to some hidden/internal column the underslying reader provides.
+ *
+ * NOTE: It is not possible to change the semantics of the base metadata
fields by overriding this
+ * method. Technically, a file format could choose suppress them, but that
is not recommended.
+ */
+ def metadataSchemaFields: Seq[StructField] = FileFormat.BASE_METADATA_FIELDS
Review Comment:
I'm wondering if we should have 2 APIs:
```
def constantMetadataClolumns: Seq[StructField]
def generatedMetadataColumns: Seq[StructField]
```
Then Spark can add metadata fields which means less work for the
implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
