dayan02dev opened a new pull request, #12333:
URL: https://github.com/apache/arrow-rs/pull/12333

   # Which issue does this PR close?
   
   - Closes #7243
   
   # Rationale for this change
   
   When a batch of a dictionary column can't keep one dictionary (it reaches 
the next row group's dictionary, or a PLAIN fallback page), the reader converts 
the keys it has buffered into plain values. Null slots padded before that point 
still hold some earlier key, and each was converted into a full copy of that 
key's value. The file in the issue (a 4 MB value, then 600 nulls) fails with 
`index overflow decoding byte array`. A single row group fails too when a 
`RowSelection` splits the batch ahead of a PLAIN page, so not merging row 
groups would not avoid it. Below 2 GiB the copies stay in the returned batch: a 
1 MB value followed by 400 nulls comes back as a 406 MB batch.
   
   This is a different trigger from #7973 and #11478, where the decoded strings 
in a batch really exceed the `i32` offset range. Here the data is small and 
only the copies overflow.
   
   # What changes are included in this PR?
   
   - `DictionaryBuffer::Dict` keeps the validity mask that `pad_nulls` 
receives, and `spill_values` writes an empty slot for each null, as the comment 
there suggested. Reads without nulls record nothing.
   - Rebuilding a dictionary from plain values hashes and deduplicates only the 
valid slots; null slots get key 0. On its own, the first change made the 
converted batch slower to rebuild (its empty slots alternate randomly with real 
values), so this keeps it fast. Batches decoded from plain pages with nulls 
benefit too.
   - Converting to `FixedSizeBinary` zero-fills empty slots. A non-nullable 
`Dictionary<_, FixedSizeBinary>` field under a nullable struct has no null 
buffer for them, so the conversion failed with `Nested array size mismatch`, 
already the case for plain pages on main.
   
   # Are these changes tested?
   
   Yes. `test_dictionary_fallback_with_nulls` reads a 64 KiB value, 100 nulls 
and two short values across two row groups, and from one row group through a 
row selection that splits the read before a PLAIN page. It checks the values, 
that the batch holds the large value once (on main the two batches use 6.6 MB 
and 3.3 MB), and that the rebuilt dictionary has only the three distinct 
values. `test_dictionary_buffer` now checks the bytes a spill produces, and 
`test_fixed_size_binary_in_dict_dictionary_disabled` also round-trips the 
column inside a nullable struct, which fails on main with `Nested array size 
mismatch`. The issue's program now reads its 602 rows; it isn't a test because 
failing it takes a 2 GiB allocation.
   
   `arrow_reader` benchmarks, main → this PR, median of 6 back-to-back runs per 
case: StringDictionary `optional, half NULLs` 180.5 → 157.5 µs; `mandatory, no 
NULLs` 132.7 → 133.3 µs and `optional, no NULLs` 134.1 → 133.5 µs; the four 
PlainStringToDictionary cases within 1.5%. A temporary bench case reading plain 
pages into a dictionary (not included in this PR) gave ×0.96 at 10% nulls, 
×0.74 at 50% and ×0.41 at 90%.
   
   # Are there any significant user-facing changes?
   
   No API changes. Dictionaries rebuilt from plain values no longer get an 
empty-string entry that only null rows reference.
   
   # AI usage disclosure
   
   The analysis, fix, tests and benchmark numbers in this PR were produced with 
an AI coding agent (Claude Code). I read the full diff and ran the new tests 
locally; they fail on main and pass with this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to