rich7420 commented on code in PR #7106:
URL: https://github.com/apache/datafusion-comet/pull/7106#discussion_r4347705818
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -330,6 +330,21 @@ fn parquet_convert_array_impl(
Ok(Arc::new(string_array))
}
+ // Unannotated BINARY read as a string. Spark's cast replaces
malformed UTF-8 with
+ // U+FFFD, where Arrow's safe cast below would return NULL. Top-level
columns don't
+ // reach this arm (#7032). The reader keeps a file's ARROW:schema
types, so an Arrow
+ // writer's `LargeBinary` or `BinaryView` arrives as is; Spark's cast
takes only
+ // `Binary`, and converting `LargeBinary` reuses its value buffer.
+ (Binary | LargeBinary | BinaryView, Utf8) => {
Review Comment:
Could nested `Dictionary(_, Binary | LargeBinary)` leaves from an
ARROW:schema still miss this arm and get NULL instead of U+FFFD? Maybe unpack
them to `Binary` here, with a `List<Dictionary<Int32, Binary>>` test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
