Rich-T-kid opened a new issue, #12341: URL: https://github.com/apache/arrow-rs/issues/12341
The RunArray specialization from #12218 optimizes contiguous `copy_rows`, but `BatchCoalescer::push_batch_with_indices` can still materialize an intermediate taken array before copying it into the accumulator. A prototype added a RunArray-specific `copy_rows_by_indices` implementation that maps logical indices to physical runs, retains the referenced value ranges, and merges consecutive output rows that reference the same physical run. This can avoid the intermediate `take` allocation without adding specialized filter or interleave dispatch. Local `coalesce_kernels` results for `single_ree` against the exact #12218 merge base: | Null density | Selectivity | Base | Prototype | Speedup | | ---: | ---: | ---: | ---: | ---: | | 0 | 0.001 | 21.157 ms | 2.887 ms | 7.33x | | 0 | 0.01 | 4.905 ms | 1.877 ms | 2.61x | | 0 | 0.1 | 2.955 ms | 1.811 ms | 1.63x | | 0 | 0.8 | 3.006 ms | 1.797 ms | 1.67x | | 0.1 | 0.001 | 24.861 ms | 3.072 ms | 8.09x | | 0.1 | 0.01 | 8.660 ms | 2.074 ms | 4.18x | | 0.1 | 0.1 | 3.597 ms | 2.022 ms | 1.78x | | 0.1 | 0.8 | 3.557 ms | 2.023 ms | 1.76x | The prototype was removed from #12218 to keep that PR focused. A follow-up should preserve the existing generic fallback and cover repeated and out-of-order indices, nullable values, sliced RunArrays, target-batch splitting, Int16/Int32/Int64 run ends, and run-end overflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
