Rich-T-kid opened a new issue, #12196:
URL: https://github.com/apache/arrow-rs/issues/12196
Parent: #12195
### Problem
`DictionaryBuffer::as_keys_with_bit_width` keeps the dictionary-preserving
path only when the incoming dictionary has the same allocation as the one
already stored in the buffer.
```text
current dictionary Arc ----+
+-- same pointer? --> append keys
incoming dictionary Arc ---+
+-- different pointer? --> spill values
```
Two column chunks can contain dictionaries with identical values but
different allocations:
```text
dictionary A = ["mobile", "desktop", "tablet"]
dictionary B = ["mobile", "desktop", "tablet"]
contents: equal
pointers: different
```
If a batch crosses that boundary, the reader expands all existing keys into
byte values and later rebuilds the same dictionary.
### Proposed change
When dictionary pointers differ, detect the case where the dictionaries have
identical type, offsets, and values. If they are equal, retain the existing
dictionary and append the incoming keys without remapping or spilling.
```text
different pointers
|
| compare or fingerprint unique values
v
identical contents? -- yes --> reuse existing key space
|
no
v
normal merge/spill path
```
Any fingerprint must be confirmed by equality before treating the key spaces
as interchangeable. The existing pointer check remains the first and cheapest
test.
### Requirements
- Cover distinct `ArrayRef` allocations with identical dictionary contents.
- Preserve the fallback for dictionaries that differ in value, order, type,
or length.
- Preserve key bounds and null handling.
- Benchmark batches crossing row groups or column chunks with identical
dictionaries.
- Show that the additional comparison does not regress the same-pointer fast
path.
- Compare the cost of checking the unique dictionary values with the avoided
expansion of row values.
This only handles exactly equal dictionaries. Dictionaries with overlapping
or reordered values still require the remapping work tracked by #12195.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
