dejankrak-db commented on code in PR #59305:
URL: https://github.com/apache/spark/pull/59305#discussion_r4277507788
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // Bare `"file"` resolves differently depending on context: inside a
StructField it is
+ // an external file, which is also what a schema written without any
metadata means; in a
+ // standalone parse (top-level `fromJson` or recursion outside any
StructField) it has no
+ // mode yet, as when a UDF's return type is serialized as a bare
`"file"`.
Review Comment:
Agreed, and you are right that a nested file in a UDF type should stay
unknown. The JSON form of a FileType now always carries its mode, as `"file
external"` or `"file unknown"`, in a field as well as standalone, so a nested
unknown file round-trips (f64b7071a6e). `StructField` no longer special-cases
FileType. A bare `"file"` is still read for schemas written without a mode: as
external inside a struct field and as unknown elsewhere, and a mode name this
version does not know reads as external. `inStructField` stays only for that
no-mode case. Added a test for a nested unknown file in a struct, an array and
a nested struct.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -77,7 +77,8 @@ case class StructField(
}
private[sql] def dataTypeJsonValue: JValue = {
- if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty) {
+ if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty &&
+ !dataType.existsRecursively(_.isInstanceOf[FileType])) {
Review Comment:
Follow-up: with f64b7071a6e a `FileType` writes its own mode in its JSON
form, so `StructField` has no FileType-specific handling any more and
`removeCollations` is untouched. A FILE-bearing field takes the same path as
any other field, and the test from be0bd45fa52 still covers the `MAP<STRING
COLLATE UTF8_BINARY, FILE EXTERNAL>` case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
