alamb opened a new pull request, #12348:
URL: https://github.com/apache/arrow-rs/pull/12348

   # Which issue does this PR close?
   
   - Part of #11140. Stacked on #11144: the first commits here are that PR plus 
two small review follow-ups, and only the last commit is new. Draft until 
#11144 merges.
   
   # Rationale for this change
   
   After #12161 the dictionary reader validates UTF-8 on a different layer from 
the other two byte array readers. `DictionaryDecoder` takes its `validate_utf8` 
flag and `value_type` from the converted type, so for an unannotated 
`BYTE_ARRAY` column read as `Dictionary(_, Utf8)` it decodes without validation 
into `Binary` values, and `ByteArrayDictionaryReader::consume_batch` validates 
afterwards by converting the values with `StringArray::try_from_binary`. The 
`ColumnValueDecoder::set_validate_utf8` hint added in #11144 reaches this 
decoder through `set_page_reader` and is silently dropped, because the trait 
method has a no-op default.
   
   That works today, but only because of the reader-level conversion. Nothing 
in the decoder records that it relies on another mechanism, and removing the 
conversion would reopen the hole without a compile error.
   
   # What changes are included in this PR?
   
   - `DictionaryDecoder` overrides `set_validate_utf8`. The flag now sets both 
validation and the decoded value type, so dictionary pages are validated once 
in `set_dict` and built directly as `Utf8`/`LargeUtf8`, and fallback pages are 
validated while decoding by `try_push`, the same way the plain reader does it.
   - `make_byte_array_dictionary_reader` derives the flag from the requested 
dictionary value type and passes it to the record reader, like 
`make_byte_array_reader` and `make_byte_view_array_reader` do.
   - `byte_array_value_type` takes the flag instead of the column descriptor.
   - `convert_dictionary_values` only converts to the view types. The 
`try_from_binary` arms are unreachable now and are removed. The per-page cache 
from #12161 is kept for the `Utf8View` conversion.
   - `test_read_non_utf8_binary_as_utf8` asserts the exact error message again, 
since all paths now report `encountered non UTF-8 data`.
   
   # Are these changes tested?
   
   - New `test_validate_utf8_follows_requested_type` in 
`byte_array_dictionary.rs` checks, at the decoder level, that without the hint 
an unannotated column builds `Binary` values with no validation, and with it 
valid data builds `Utf8` values while invalid data is rejected in both a 
dictionary page and a PLAIN fallback page.
   - The existing end-to-end tests in `tests/schema.rs` cover all six 
dictionary targets over five encodings, including the split-character input.
   
   `cargo test -p parquet --features arrow --lib -- arrow::` (621 tests) and 
`cargo clippy -p parquet --all-targets --features arrow -- -D warnings` pass 
under the pinned 1.100.0 toolchain.
   
   # Are there any user-facing changes?
   
   No. Output types and error messages for the dictionary reader are unchanged, 
except that a split character read through a dictionary target now reports 
`encountered non UTF-8 data` like the other readers instead of `Split UTF-8 
codepoint`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to