adriangb opened a new issue, #25509:
URL: https://github.com/apache/datafusion/issues/25509
**Describe the bug**
`datafusion.execution.parquet.binary_as_string = true` changes a Parquet
`Binary` column to a string type in the schema that DataFusion gives to the
Parquet reader. If the column holds bytes that are not valid UTF-8, the reader
returns an invalid string array and reports no error. The first operation that
decodes those bytes then stops the process with SIGSEGV. The user gets no
message and no error code from the query.
This is not limited to unusual schemas. A top level column is sufficient.
**To Reproduce**
```shell
datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'"
-c "set datafusion.execution.parquet.binary_as_string = true" -c "SELECT b FROM
'bin.parquet'"
```
```
DataFusion CLI v54.0.0
+-------+
| count |
+-------+
| 1 |
+-------+
1 row(s) fetched.
Elapsed 0.010 seconds.
0 row(s) fetched.
Elapsed 0.001 seconds.
```
The exit code is 139. The macOS crash report shows `SIGSEGV` with the fault
in `comfy_table::Row::max_content_widths`, which measures the width of the
invalid string. With `SELECT upper(b)` the fault moves to
`alloc::str::to_uppercase`.
Measured on the released `datafusion-cli` 54.0.0 and on a debug build of
`main` at 4e907557ad (arrow and parquet 59.3.0):
| Column type and target | Build | Result |
| --- | --- | --- |
| `Binary` to `Utf8View` (defaults) | release | SIGSEGV |
| `Binary` to `Utf8View` (defaults) | debug | non-unwinding panic from the
`core::str::validations` UB check, then abort |
| `Binary` to `Utf8` (`schema_force_view_types = false`) | release | SIGSEGV
|
| `Binary` to `Utf8` | debug | panic in
`parquet-59.3.0/src/arrow/buffer/offset_buffer.rs:141`: `Invalid UTF8 sequence
at string index 0` |
| `LargeBinary` to `LargeUtf8` | release | SIGSEGV |
| `BinaryView` to `Utf8View` | release | SIGSEGV |
`count(*)`, `octet_length(b)` and the byte-wise comparison `b = 'ok'` all
succeed, because they do not decode UTF-8. The invalid array moves through the
plan without a sign of a problem.
An explicit cast is safe, which shows the difference:
```shell
datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'"
-c "SELECT arrow_cast(b, 'Utf8') FROM 'bin.parquet'"
```
```
Error: Arrow error: Invalid argument error: Encountered non UTF-8 data:
invalid utf-8 sequence of 1 bytes from index 0
```
Valid multibyte UTF-8 in a `Binary` column reads correctly with the option
on.
**Expected behavior**
The query must fail with an error, in the same way as the explicit cast.
DataFusion must not give an invalid string array to the rest of the plan.
**Additional context**
The cause is in parquet-rs. `ByteArrayColumnValueDecoder::new` selects UTF-8
validation from the physical Parquet annotation only:
```rust
// parquet/src/arrow/array_reader/byte_array.rs:192
let validate_utf8 = desc.converted_type() == ConvertedType::UTF8;
```
A plain BYTE_ARRAY column has no `UTF8` annotation, so the target Arrow type
that DataFusion supplies through `ArrowReaderOptions::with_schema` has no
effect. `OffsetBuffer::into_array` then uses
`ArrayDataBuilder::build_unchecked` in release builds. Upstream report:
PLACEHOLDER_ARROW
Found while reviewing https://github.com/apache/datafusion/pull/25342. That
PR does not introduce the problem. It applies the same override to map
children, so it makes more reads reach this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
