alamb opened a new pull request, #12348: URL: https://github.com/apache/arrow-rs/pull/12348
# Which issue does this PR close? - Part of #11140. Stacked on #11144: the first commits here are that PR plus two small review follow-ups, and only the last commit is new. Draft until #11144 merges. # Rationale for this change After #12161 the dictionary reader validates UTF-8 on a different layer from the other two byte array readers. `DictionaryDecoder` takes its `validate_utf8` flag and `value_type` from the converted type, so for an unannotated `BYTE_ARRAY` column read as `Dictionary(_, Utf8)` it decodes without validation into `Binary` values, and `ByteArrayDictionaryReader::consume_batch` validates afterwards by converting the values with `StringArray::try_from_binary`. The `ColumnValueDecoder::set_validate_utf8` hint added in #11144 reaches this decoder through `set_page_reader` and is silently dropped, because the trait method has a no-op default. That works today, but only because of the reader-level conversion. Nothing in the decoder records that it relies on another mechanism, and removing the conversion would reopen the hole without a compile error. # What changes are included in this PR? - `DictionaryDecoder` overrides `set_validate_utf8`. The flag now sets both validation and the decoded value type, so dictionary pages are validated once in `set_dict` and built directly as `Utf8`/`LargeUtf8`, and fallback pages are validated while decoding by `try_push`, the same way the plain reader does it. - `make_byte_array_dictionary_reader` derives the flag from the requested dictionary value type and passes it to the record reader, like `make_byte_array_reader` and `make_byte_view_array_reader` do. - `byte_array_value_type` takes the flag instead of the column descriptor. - `convert_dictionary_values` only converts to the view types. The `try_from_binary` arms are unreachable now and are removed. The per-page cache from #12161 is kept for the `Utf8View` conversion. - `test_read_non_utf8_binary_as_utf8` asserts the exact error message again, since all paths now report `encountered non UTF-8 data`. # Are these changes tested? - New `test_validate_utf8_follows_requested_type` in `byte_array_dictionary.rs` checks, at the decoder level, that without the hint an unannotated column builds `Binary` values with no validation, and with it valid data builds `Utf8` values while invalid data is rejected in both a dictionary page and a PLAIN fallback page. - The existing end-to-end tests in `tests/schema.rs` cover all six dictionary targets over five encodings, including the split-character input. `cargo test -p parquet --features arrow --lib -- arrow::` (621 tests) and `cargo clippy -p parquet --all-targets --features arrow -- -D warnings` pass under the pinned 1.100.0 toolchain. # Are there any user-facing changes? No. Output types and error messages for the dictionary reader are unchanged, except that a split character read through a dictionary target now reports `encountered non UTF-8 data` like the other readers instead of `Split UTF-8 codepoint`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
