princejain756 opened a new pull request, #11904: URL: https://github.com/apache/arrow-rs/pull/11904
# Which issue does this PR close? - Closes #11901 # Rationale for this change `RowConverter` encodes every value in a dictionary's values array for each batch, so encoding cost grows with dictionary size rather than row count. The issue reports a 1M-value dictionary costing ~7ms to convert 8192 rows versus ~50us for the equivalent Utf8 array. The same dictionary-retention problem is tracked downstream in apache/datafusion#26696, where hash aggregate output slices keep the whole dictionary. # What changes are included in this PR? - When a dictionary column's values outnumber its rows, `Codec::encoder` gathers only the referenced values via `MutableArrayData` (no new dependencies) and stores a per-row remap in `Encoder::Dictionary`. - `row_lengths` and the encode path translate each key through the remap when present. `encode_dictionary_values` keeps its signature and handles the un-compacted path as before. - New benchmark case `4096 string_dictionary_sparse_keys(1M values)`: 4096 rows keyed into 1024 of 1M strings. # Are these changes tested? New unit tests cover byte equality against an equivalent small dictionary, null keys, round-trip decoding, and sliced arrays. Benchmark results for the new case, this branch vs main: - `convert_columns`: 9.86ms -> 0.152ms (~65x) - `convert_columns_prepared`: 9.77ms -> 0.155ms (~63x) - `append_rows`: 9.61ms -> 0.150ms (~64x) - `convert_rows`: 51.0us -> 61.3us (decode path is unchanged) # Are there any significant user-facing changes? No. Row encodings are unchanged; this only avoids encoding dictionary values that no row references. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
