Rich-T-kid opened a new pull request, #12223: URL: https://github.com/apache/arrow-rs/pull/12223
# Which issue does this PR close? - Closes #11563 # Rationale for this change Sparse indexed copies of `StringArray` currently materialize an intermediate array with `take` and then copy the selected strings again into the coalescer. This makes very sparse `push_batch_with_indices` workloads spend most of their time on an avoidable allocation and copy. # What changes are included in this PR? This adds direct indexed copying for sparse variable-width byte arrays. It gathers selected values and nulls into the existing byte-array builder after checking offset overflow. Selections above the measured crossover keep the existing `take` fallback, including mixed schemas where primitive columns can still use their own indexed path. I used Codex to help implement the change and run the local benchmark matrix. I reviewed the final code and benchmark results. # Are these changes tested? Unit tests cover the direct and fallback paths, including nulls, sliced arrays, repeated indices, and out-of-order indices. Local `coalesce_kernels` measurements at 8,192 input rows: | Case | `main` | This PR | Change | | --- | ---: | ---: | ---: | | `single_utf8`, 0.1%, no nulls | 13.90 ms | 6.53 ms | 53% faster | | `single_utf8`, 0.1%, 10% nulls | 17.67 ms | 6.38 ms | 64% faster | | `mixed_utf8`, 0.1%, no nulls | 5.92 ms | 3.24 ms | 45% faster | | `mixed_utf8`, 0.1%, 10% nulls | 10.73 ms | 5.71 ms | 47% faster | The 1%, 10%, and 80% cases retain the existing fallback and showed no material regression. # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
