dejankrak-db commented on code in PR #59305:
URL: https://github.com/apache/spark/pull/59305#discussion_r4277600603
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // Bare `"file"` resolves differently depending on context: inside a
StructField it is
+ // an external file, which is also what a schema written without any
metadata means; in a
+ // standalone parse (top-level `fromJson` or recursion outside any
StructField) it has no
+ // mode yet, as when a UDF's return type is serialized as a bare
`"file"`.
Review Comment:
Note: commits I mentioned earlier in this thread were squashed. The current
commit is 9c76b11358a, and the comment above is updated to match it.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -77,7 +77,8 @@ case class StructField(
}
private[sql] def dataTypeJsonValue: JValue = {
- if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty) {
+ if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty &&
+ !dataType.existsRecursively(_.isInstanceOf[FileType])) {
Review Comment:
Note: commits I mentioned earlier in this thread were squashed. The current
commit is 9c76b11358a, and the comment above is updated to match it.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##########
@@ -77,7 +77,8 @@ case class StructField(
}
private[sql] def dataTypeJsonValue: JValue = {
- if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty) {
+ if (stringCollationMetadata.isEmpty &&
charVarcharCollationMetadata.isEmpty &&
+ !dataType.existsRecursively(_.isInstanceOf[FileType])) {
Review Comment:
Fixed in 9c76b11358a. A field whose type contains a FileType no longer goes
through `removeCollations` unless it has collation metadata: a `FileType` now
writes its own mode in its JSON form (`"file external"` or `"file unknown"`),
so `StructField` needs no FileType-specific handling and the file is the same
as on master. A FILE-bearing field takes the same path as any other field, so
`MAP<STRING COLLATE UTF8_BINARY, FILE EXTERNAL>` keeps `"string collate
UTF8_BINARY"` for its key after a JSON round trip. The `DataTypeSuite` test "an
explicit UTF8_BINARY collation next to a file survives the JSON form" covers it.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:
##########
@@ -344,6 +345,16 @@ object DataType {
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
+ case FILE_TYPE(mode) =>
+ // Bare `"file"` resolves differently depending on context: inside a
StructField it is
+ // an external file, which is also what a schema written without any
metadata means; in a
+ // standalone parse (top-level `fromJson` or recursion outside any
StructField) it has no
+ // mode yet, as when a UDF's return type is serialized as a bare
`"file"`.
Review Comment:
Agreed, a nested file in a UDF type should stay unknown. Fixed in
9c76b11358a: the JSON form of a FileType now always carries its mode, as `"file
external"` or `"file unknown"`, in a struct field as well as standalone, so a
nested unknown file round-trips. `StructField` no longer special-cases
FileType. A bare `"file"` is still read for schemas written without a mode: as
external inside a struct field and as unknown elsewhere. Any other mode name is
rejected with `INVALID_JSON_DATA_TYPE`. `inStructField` stays only for that
no-mode case. `DataTypeSuite` has a test for a nested unknown file in a struct,
an array and a nested struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
