andygrove opened a new issue, #7469:
URL: https://github.com/apache/datafusion-comet/issues/7469
### Describe the bug
On DataFusion 56, the native Parquet scan decodes a nested Parquet BINARY
value that is read as a string straight into an Arrow `Utf8` array, without
checking that it is valid UTF-8. Nested values here means list elements and
struct fields. An invalid value then gives a `StringArray` that holds invalid
UTF-8. In a debug build (and the CI profile, which enables debug assertions)
the parquet reader panics. In a release build the array is unsound for every
string kernel that reads it.
On current `main`, which is on DataFusion 55.2, #7106 decodes these values
with Spark's lossy decode, so an ill-formed sequence becomes U+FFFD. The
DataFusion 56 upgrade (#6410, draft PR #7249) undoes that.
`nested_arrow_binary_types_read_as_string_decode_invalid_utf8` in
`native/core/src/parquet/parquet_exec.rs` shows it. A list of `[b"one",
[0xff]]` read as `array<string>` panics in
`parquet/src/arrow/buffer/offset_buffer.rs:139`:
```
InvalidArgumentError("Invalid UTF8 sequence at string index 1 (3..4):
invalid utf-8 sequence of 1 bytes from index 0")
```
It panics for plain `Binary` elements too, not only the Arrow-specific
`LargeBinary` and `BinaryView`, so files written by Spark are affected whenever
a nested BINARY column is read as a string, through a read schema or
`spark.sql.parquet.binaryAsString`.
### Cause
apache/datafusion#25193 made `apply_file_schema_type_coercions` recurse into
nested types. When the table schema has `Utf8` where the file has `Binary`,
`LargeBinary` or `BinaryView`, DataFusion now rewrites the nested file field to
`Utf8` and hands that schema to the parquet reader. DataFusion 55.2 did this
only for top-level fields, so nested values reached Comet's
`parquet_convert_array` as binary and #7106's decode.
The parquet reader validates UTF-8 only when the column carries the `UTF8`
annotation (`validate_utf8 = desc.converted_type() == ConvertedType::UTF8` in
`parquet/src/arrow/array_reader/byte_array.rs`), which an unannotated BINARY
column does not. DataFusion's own docs on `coerce_data_type` note that the
conversion "hands back an unvalidated string array".
This is the nested form of #7032's top-level gap, which already exists on
`main`.
### Steps to reproduce
Run `nested_arrow_binary_types_read_as_string_decode_invalid_utf8` on the
`datafusion-56` branch.
### Expected behavior
An ill-formed sequence in a nested BINARY value read as a string decodes to
U+FFFD, as it does on `main`.
### Additional context
- Possible fixes: have the parquet reader validate UTF-8 when the target
Arrow type is a string, whatever the annotation; let callers opt out of the
binary-to-string coercion in DataFusion; or keep nested binary file types out
of the coercion in Comet and convert them with Comet's decoder.
- Until a fix lands, the test is ignored on the DataFusion 56 branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
