Rich-T-kid opened a new pull request, #12008: URL: https://github.com/apache/arrow-rs/pull/12008
# Which issue does this PR close? - Closes #11885 # Rationale for this change Merging short slices of large dictionaries currently allocates and scans state sized to each complete values array. This makes concat and interleave cost grow with unreferenced dictionary values rather than selected rows. # What changes are included in this PR? Use sparse referenced-key collection and remapping when a dictionary has more than 16 values per selected row. Smaller dictionaries retain the existing dense bitmap and vector path. # Are these changes tested? Added coverage for the sparse merge path and its concat and interleave consumers. On the 12 x 682-row reproduction from #11885 (Apple M4 Pro): | Values per input | main | this PR | | ---: | ---: | ---: | | 100,000 | 188.09 us | 146.76 us (-22.0%) | | 1,000,000 | 446.71 us | 148.94 us (-66.7%) | The existing 1,024-row dense and sparse dictionary concat benchmarks show no regression. # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
