ryan-johnson-databricks commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1162726854
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -176,6 +186,23 @@ trait FileFormat {
* By default all field name is supported.
*/
def supportFieldName(name: String): Boolean = true
+
+ /**
+ * All fields the file format's _metadata struct defines.
+ *
+ * Each field's metadata should define [[METADATA_COL_ATTR_KEY]],
+ * [[FILE_SOURCE_METADATA_COL_ATTR_KEY]], and either
+ * [[FILE_SOURCE_CONSTANT_METADATA_COL_ATTR_KEY]] or
+ * [[FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY]] as appropriate.
+ *
+ * Constant attributes will be extracted automatically from
+ * [[PartitionedFile.extraConstantMetadataColumnValues]], while generated
metadata columns always
+ * map to some hidden/internal column the underslying reader provides.
+ *
+ * NOTE: It is not possible to change the semantics of the base metadata
fields by overriding this
+ * method. Technically, a file format could choose suppress them, but that
is not recommended.
+ */
+ def metadataSchemaFields: Seq[StructField] = FileFormat.BASE_METADATA_FIELDS
Review Comment:
Could you elaborate what we gain by splitting out two lists? For generated
columns, in particular, we _must_ use the helper object because the user should
specify the physical column name to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
