zhuqi-lucas commented on issue #10774: URL: https://github.com/apache/arrow-rs/issues/10774#issuecomment-5448118582
Tried this and the coalesce approach doesn't help here: its single-pass path only fires for all-inline views. When views reference data buffers, `copy_rows_by_filter_from` does the same filter-then-remap two passes as today's code (the comment there even says "match the filter kernel"). Measured where the time actually goes in `consume_batch` (clickbench bench, Q12): `filter` ~6%, `concat` ~50% — and the concat cost is the per-view `buffer_index` remap, not string data (`str_len=20` and `128` concat at the same speed; all-inline is 2.7x faster). I implemented three single-pass alternatives (`interleave`, index-based gather, run-based gather with the remap fused in) — all measurably slower: at ~80% selectivity nothing beats filter's run-based bulk copy. What did show up is a quadratic: `cleanup_consumed_batches` rescans batch ids from 0 and takes the cache write lock on every `consume_batch`. Fix in #10901. The real opportunity the numbers point at: the single-slice path (filter only) shares the source buffers and does **zero** remapping — 0.058 ns/row vs 0.498 for the multi-slice path, which is only 12.8% of calls but covers 50% of all rows. Rather than a fused kernel, aligning the cache batch size/boundaries with `consume_batch`'s span (`row_count` reaches ~22k vs `batch_size=8192`, hence the 7-slice concats) would make the zero-remap path the common case. Worth pursuing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
