adriangb opened a new issue, #11140: URL: https://github.com/apache/arrow-rs/issues/11140
**Describe the bug** The Arrow reader accepts a schema that changes a BYTE_ARRAY column from `Binary` to `Utf8`, `LargeUtf8` or `Utf8View`. It does not validate the bytes as UTF-8. It returns `Ok` with an array that breaks the string invariant. `ByteArrayColumnValueDecoder::new` takes the decision from the physical annotation only: ```rust // parquet/src/arrow/array_reader/byte_array.rs:192 let validate_utf8 = desc.converted_type() == ConvertedType::UTF8; ``` A plain BYTE_ARRAY column has no `UTF8` annotation, so `validate_utf8` stays `false`. The target Arrow type that comes from `ArrowReaderOptions::with_schema` has no effect on this decision. `OffsetBuffer::into_array` then builds the array with `ArrayDataBuilder::build_unchecked` in release builds (`parquet/src/arrow/buffer/offset_buffer.rs:141`). The caller gets an array that it can not use safely. Any operation that decodes the bytes as UTF-8 is undefined behavior. Apache DataFusion has a `binary_as_string` option that uses this override, and a simple query over such a file stops the process with SIGSEGV. See https://github.com/apache/datafusion/issues/25509. **To Reproduce** `Cargo.toml`: ```toml [dependencies] arrow = "59.3.0" parquet = "59.3.0" bytes = "1" ``` `src/main.rs`: ```rust use std::sync::Arc; use arrow::array::{Array, ArrayRef, BinaryArray, RecordBatch}; use arrow::datatypes::{DataType, Field, Schema}; use bytes::Bytes; use parquet::arrow::arrow_reader::{ArrowReaderOptions, ParquetRecordBatchReaderBuilder}; use parquet::arrow::arrow_writer::{ArrowWriter, ArrowWriterOptions}; fn write() -> Bytes { // One BYTE_ARRAY column, no UTF8 annotation, one value of 0xff. let schema = Arc::new(Schema::new(vec![Field::new("b", DataType::Binary, true)])); let array = Arc::new(BinaryArray::from(vec![Some(&[0xffu8][..])])) as ArrayRef; let batch = RecordBatch::try_new(Arc::clone(&schema), vec![array]).unwrap(); let mut buf = Vec::new(); // Skip the advisory ARROW:schema key. The result is the same with it. let options = ArrowWriterOptions::new().with_skip_arrow_metadata(true); let mut writer = ArrowWriter::try_new_with_options(&mut buf, schema, options).unwrap(); writer.write(&batch).unwrap(); writer.close().unwrap(); Bytes::from(buf) } fn read_as(data: Bytes, target: DataType) { let schema = Arc::new(Schema::new(vec![Field::new("b", target.clone(), true)])); let options = ArrowReaderOptions::new().with_schema(schema); let mut reader = ParquetRecordBatchReaderBuilder::try_new_with_options(data, options) .unwrap() .build() .unwrap(); let batch = reader.next().unwrap().unwrap(); let column = batch.column(0); println!(" read Ok, data_type = {}", column.data_type()); match column.to_data().validate_full() { Ok(()) => println!(" validate_full: Ok"), Err(e) => println!(" validate_full: Err({e})"), } } fn main() { let data = write(); for target in [DataType::Utf8, DataType::LargeUtf8, DataType::Utf8View] { println!("target = {target:?}"); read_as(data.clone(), target); } } ``` `cargo run --release`: ``` target = Utf8 read Ok, data_type = Utf8 validate_full: Err(Invalid argument error: Invalid UTF8 sequence at string index 0 (0..1): invalid utf-8 sequence of 1 bytes from index 0) target = LargeUtf8 read Ok, data_type = LargeUtf8 validate_full: Err(Invalid argument error: Invalid UTF8 sequence at string index 0 (0..1): invalid utf-8 sequence of 1 bytes from index 0) target = Utf8View read Ok, data_type = Utf8View validate_full: Err(Invalid argument error: Encountered non-UTF-8 data at index 0: invalid utf-8 sequence of 1 bytes from index 0) ``` In a debug build, the `Utf8` and `LargeUtf8` targets panic in the reader, because `into_array` uses `build().unwrap()` when `debug_assertions` is on: ``` thread 'main' panicked at parquet-59.3.0/src/arrow/buffer/offset_buffer.rs:141:48: called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Invalid UTF8 sequence at string index 0 (0..1): invalid utf-8 sequence of 1 bytes from index 0)") ``` The `Utf8View` target has no such check. A debug build also returns the invalid array for that target. **Expected behavior** The reader must validate UTF-8 when the target Arrow type is a string type, and not only when the Parquet column carries the `UTF8` annotation. If the bytes are not valid UTF-8, the reader must return an error. **Additional context** Reproduced with 59.3.0. The same gate is in 59.2.0. Valid multibyte UTF-8 in a `Binary` column reads correctly on all three targets, so the override is useful. Only the validation is missing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
