ryan-johnson-databricks commented on code in PR #40677:
URL: https://github.com/apache/spark/pull/40677#discussion_r1161702693
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala:
##########
@@ -165,6 +164,17 @@ trait FileFormat {
}
}
+ /**
+ * Create a file metadata struct column containing fields supported by the
given file format.
+ */
+ def createFileMetadataCol(): AttributeReference = {
Review Comment:
This PR intentionally moved the method _out_ of `FileFormat`, as part of the
effort to eliminate the the central code bottleneck antipattern of dumping
everything in the `FileFormat` companion object -- which currently makes it
impossible for file formats to customize any aspect of metadata handling.
Absent a specific reason to forbid file formats from customizing this behavior,
I'd prefer to allow it, so we don't have to keep coming back and changing the
code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
