Dandandan opened a new pull request, #24454: URL: https://github.com/apache/datafusion/pull/24454
## Which issue does this PR close? - N/A ## Rationale for this change When every probe row of a hash join matches exactly once, the emitted probe indices are a contiguous run `[k, k+1, ..., k+n-1]`. Taking a probe-side column by those indices copies it element by element to reproduce a range it already has. Slicing it is zero-copy. This is common in TPC-H: 11 joins across the suite run at `avg_fanout=100%, probe_hit_rate=100%` (q2, q5, q7, q8, q9, q10, q11, q13, q15, q18, q21). Detecting a contiguous *run* rather than an identity range matters, because output is emitted in `batch_size` chunks, so the indices usually start partway into the probe batch. ## What changes are included in this PR? `contiguous_run_start()` in `joins/utils.rs`, used by `build_batch_from_indices` to slice probe-side columns instead of gathering them. Computed once per batch and reused for every probe-side column. Falls back to `take` when there are nulls (unmatched rows in outer joins) or any gap; the scan short-circuits on the first mismatch, so sparse index vectors typically pay O(1). One tradeoff worth flagging for review: a sliced array shares the parent buffer, so `get_array_memory_size()` on the output reports the parent batch's size, and each output chunk pins one probe batch. ## Are these changes tested? Yes — a unit test covers runs starting at zero and mid-batch, gaps, repeats, descending order, nulls and empty input. Existing join tests pass (1039 in `joins::`). ## Are there any user-facing changes? No API change. I have not been able to measure a reliable speedup: this machine is too loaded to resolve an effect this size, and my A/B attempts did not survive their own negative controls. Posting as a draft for that reason — the change removes work and is correctness-tested, but the benchmark claim is unproven. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
