andygrove opened a new pull request, #12064: URL: https://github.com/apache/arrow-rs/pull/12064
# Which issue does this PR close? - Closes #12063. # Rationale for this change Reading a Parquet BINARY column that has no `UTF8` annotation as a string type, through `ArrowReaderOptions::with_schema`, decodes the bytes into a string array without checking that they are valid UTF-8. For invalid bytes this panics in a debug build and returns an invalid `StringArray` in a release build. The byte array decoders decided whether to validate from the column's annotation alone, ignoring the Arrow type being produced. A dictionary of strings over such a column had a related problem. The dictionary decoder also took its value type from the annotation, so `Dictionary<_, Utf8>` came back with `Binary` values, and `Dictionary<_, Utf8View>` panicked in `convert_values_to_view`. # What changes are included in this PR? - `ColumnValueDecoder` (crate-private) gets a `set_validate_utf8` method, which does nothing by default. `GenericRecordReader::set_validate_utf8` makes the record reader call it on every value decoder it creates. - `make_byte_array_reader`, `make_byte_view_array_reader` and `make_byte_array_dictionary_reader` set it when the target type is `Utf8`, `LargeUtf8` or `Utf8View`, or a dictionary of one of them. - The byte array and byte view decoders then validate UTF-8 as they already do for annotated columns. The dictionary decoder also decodes the values as `Utf8`/`LargeUtf8`. Annotated columns, and unannotated columns read as binary, behave as before, so neither pays for an extra check. # Are these changes tested? `test_read_binary_as_string_types_validates_utf8` replaces `test_read_non_utf8_binary_as_utf8`, which expected the debug-build panic. It writes an unannotated BINARY column, both dictionary encoded and plain, and reads it as each of the six string types. Valid UTF-8 reads as strings of the requested type, and invalid UTF-8 returns a `non UTF-8 data` error. Before this change, the invalid case panicked for `Utf8`, `LargeUtf8` and `Utf8View`, and the valid case returned `Binary` values or panicked for the dictionary types. # Are there any significant user-facing changes? Reading invalid UTF-8 from an unannotated BINARY column as a string now returns an error instead of an invalid array. A dictionary of strings over such a column now has string values. # AI usage The code, tests and this description were written with an AI assistant (Claude Code). The bug, the failing cases and the fix were verified by running the new test against the code before and after the change, and by running the `parquet` crate's tests and clippy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
