Rich-T-kid opened a new pull request, #12043: URL: https://github.com/apache/arrow-rs/pull/12043
## Why Closes #11901. `RowConverter` currently encodes every value in a dictionary before converting a batch. For an 8,192-row slice backed by a large dictionary, the work therefore scales with the retained values array rather than the rows being converted. This is a smaller follow-up to #11904. It avoids the hash map and per-row remap used there, and uses the benchmark crossover requested in [review](https://github.com/apache/arrow-rs/pull/11904#discussion_r4393788975) to keep small and dense dictionaries on the existing path. ## What changed For dictionaries with at least 32K values and at least four values per row, gather one value per input row and encode that row-aligned array. All other dictionaries keep the existing dictionary-key lookup path. The benchmark sweep put the crossover between 20K and 24.6K values. The 32K / 4x guard is deliberately conservative. ## Results Compared current `main` at `b231cf15ef` with this branch at `65afa03a07`. Both binaries used the same benchmark grid and separate target directories. These are medians of three paired local runs; negative values mean lower time. | Dictionary values | `convert_columns` | prepared converter | `append_rows` | |---:|---:|---:|---:| | 10K | -1.1% | +0.3% | -0.5% | | 32K | -26.2% | -25.3% | -25.3% | | 100K | -71.5% | -71.6% | -71.8% | | 1M | -96.9% | -97.0% | -97.0% | The 10K case remains on the existing path. The first enabled case is about 1.34x faster, and the 1M case is about 33x faster. ## Test coverage The focused test covers a sliced 100K-value dictionary with repeated high keys and a null key. It checks byte-for-byte row equality against an equivalent compact dictionary and round-trip decoding. All `arrow-row` unit tests and doctests pass; fmt and clippy are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
