AndreaBozzo opened a new issue, #11579:
URL: https://github.com/apache/arrow-rs/issues/11579
**Describe the bug**
The CSV reader can return `Utf8` arrays whose values are not valid UTF-8,
through safe APIs.
`RecordDecoder::flush` (`arrow-csv/src/reader/records.rs`) validates UTF-8
once, over the concatenated bytes of every field in the batch.
`StringRecord::get` then slices each field with `get_unchecked`, relying on
"Parsing produces offsets at valid byte boundaries". That does not hold when
invalid bytes on either side of a delimiter or line terminator join into a
valid sequence: the concatenation passes `from_utf8`, while each field on its
own is not UTF-8.
**To Reproduce**
arrow-csv 60.0.0 (the code is unchanged on `main`):
```rust
use std::io::Cursor;
use std::sync::Arc;
use arrow_array::{Array, StringArray};
use arrow_csv::ReaderBuilder;
use arrow_schema::{DataType, Field, Schema};
fn main() {
// Two rows of one byte each, invalid alone; 0xCD 0x9C joined is U+035C.
let data: &[u8] = b"\xCD\n\x9C\n";
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Utf8,
true)]));
let mut reader = ReaderBuilder::new(schema)
.with_header(false)
.build(Cursor::new(data))
.unwrap();
let batch = reader.next().unwrap().unwrap(); // expected an error
let column =
batch.column(0).as_any().downcast_ref::<StringArray>().unwrap();
println!("validate_full: {:?}", column.to_data().validate_full());
for row in 0..column.len() {
let value: &str = column.value(row);
println!(
"row {row}: bytes {:02X?}, valid UTF-8: {}",
value.as_bytes(),
std::str::from_utf8(value.as_bytes()).is_ok()
);
}
}
```
Output:
```
validate_full: Err(InvalidArgumentError("incomplete utf-8 byte sequence from
index 0"))
row 0: bytes [CD], valid UTF-8: false
row 1: bytes [9C], valid UTF-8: false
```
Two fields of one row behave the same (`b"\xCD,\x9C\n"` with a two-column
schema).
**Expected behavior**
An error, as for other invalid input: `Encountered invalid UTF-8 data for
line ... and field ...`.
**Additional context**
Using such a value as `&str` is undefined behaviour; in a debug build,
`str::trim` on it trips the `unreachable_unchecked` precondition in
`core::str::validations`. Found by fuzzing a downstream CSV profiler.
A possible fix: after `from_utf8` succeeds in `flush`, require every field
offset to satisfy `data.is_char_boundary(offset)`, and report the first one
that does not with the same line and field error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
