ryan-johnson-databricks commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1163040622
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -176,6 +186,23 @@ trait FileFormat {
* By default all field name is supported.
*/
def supportFieldName(name: String): Boolean = true
+
+ /**
+ * All fields the file format's _metadata struct defines.
+ *
+ * Each field's metadata should define [[METADATA_COL_ATTR_KEY]],
+ * [[FILE_SOURCE_METADATA_COL_ATTR_KEY]], and either
+ * [[FILE_SOURCE_CONSTANT_METADATA_COL_ATTR_KEY]] or
+ * [[FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY]] as appropriate.
+ *
+ * Constant attributes will be extracted automatically from
+ * [[PartitionedFile.extraConstantMetadataColumnValues]], while generated
metadata columns always
+ * map to some hidden/internal column the underslying reader provides.
+ *
+ * NOTE: It is not possible to change the semantics of the base metadata
fields by overriding this
+ * method. Technically, a file format could choose suppress them, but that
is not recommended.
+ */
+ def metadataSchemaFields: Seq[StructField] = FileFormat.BASE_METADATA_FIELDS
Review Comment:
At least the surface is pretty minimal (nobody needs to know the specific
metadata tags that get used): Instead of saying
```scala
StructField(name, dataType, nullable)
```
they pick one of:
```scala
FileSourceConstantMetadataStructField(name, dataType, nullable)
FileSourceGeneratedMetadataStructField(name, internalName, dataType,
nullable)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
