andygrove opened a new issue, #12063:
URL: https://github.com/apache/arrow-rs/issues/12063
### Describe the bug
A Parquet BINARY column that has no `UTF8` (`STRING`) annotation can be read
as `Utf8` by supplying an Arrow schema with `ArrowReaderOptions::with_schema`.
The reader then builds the string array without checking that the bytes are
valid UTF-8. When they are not:
- in a debug build, the reader panics in `OffsetBuffer::into_array`
(`parquet/src/arrow/buffer/offset_buffer.rs:139`,
`ArrayDataBuilder::build().unwrap()`)
- in a release build, it returns a `StringArray` that holds invalid UTF-8
(`validate_full()` fails). That is unsound for any code that reads the values
as `&str`.
The byte array readers decide whether to validate from the Parquet
annotation alone:
```rust
let validate_utf8 = desc.converted_type() == ConvertedType::UTF8;
```
in `byte_array.rs`, `byte_view_array.rs` and `byte_array_dictionary.rs`
under `parquet/src/arrow/array_reader/`. They ignore the Arrow type being
produced, so a supplied string type over an unannotated column skips validation.
DataFusion hit this when it coerces BINARY file columns to the string type
of a table schema (apache/datafusion#27065).
### To Reproduce
With `arrow`/`parquet` 60.1.0 and `bytes`:
```rust
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, BinaryArray};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use bytes::Bytes;
use parquet::arrow::arrow_reader::{ArrowReaderOptions,
ParquetRecordBatchReaderBuilder};
use parquet::arrow::ArrowWriter;
fn main() {
// A BINARY column (no UTF8 annotation) whose second value is not valid
UTF-8.
let values: ArrayRef =
Arc::new(BinaryArray::from(vec![b"one".as_slice(), &[0xff]]));
let batch = RecordBatch::try_from_iter([("c", values)]).unwrap();
let mut buf = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buf, batch.schema(),
None).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
// Read it back as Utf8 through a supplied schema.
let schema = Arc::new(Schema::new(vec![Field::new("c", DataType::Utf8,
false)]));
let mut reader = ParquetRecordBatchReaderBuilder::try_new_with_options(
Bytes::from(buf),
ArrowReaderOptions::new().with_schema(schema),
)
.unwrap()
.build()
.unwrap();
let batch = reader.next().unwrap().unwrap();
let column = batch.column(0);
println!("type: {}", column.data_type());
println!("validate_full: {:?}", column.to_data().validate_full());
}
```
Debug build:
```
thread 'main' panicked at
parquet-60.1.0/src/arrow/buffer/offset_buffer.rs:139:48:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Invalid
UTF8 sequence at string index 1 (3..4): invalid utf-8 sequence of 1 bytes from
index 0")
```
Release build:
```
type: Utf8
validate_full: Err(InvalidArgumentError("Invalid UTF8 sequence at string
index 1 (3..4): invalid utf-8 sequence of 1 bytes from index 0"))
```
### Expected behavior
Reading into a string type (`Utf8`, `LargeUtf8`, `Utf8View`, or a dictionary
of one of them) always validates UTF-8, whatever the column's annotation, and
returns an error for invalid bytes, as it already does for a column annotated
`UTF8`.
### Additional context
A possible fix is to validate when the column is annotated `UTF8` or the
target Arrow type is a string type. Columns that are annotated `UTF8` pay
nothing extra, and an unannotated column read as binary still skips the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
