sunchao commented on code in PR #25584:
URL: https://github.com/apache/datafusion/pull/25584#discussion_r4520769491
##########
datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs:
##########
@@ -912,6 +1003,122 @@ impl BitwiseSortMergeJoinStream {
}
}
+ /// Return a bounded slice length, yielding periodically even if every
+ /// child batch is immediately ready. Pausing the timer excludes scheduling
+ /// time from the join's own work.
+ async fn summary_slice_len(&mut self, remaining: usize) -> usize {
+ if self.existence_summary.as_ref().unwrap().rows_until_yield == 0 {
+ self.stop_join_time();
+ tokio::task::yield_now().await;
+ self.start_join_time();
+ self.existence_summary.as_mut().unwrap().rows_until_yield =
+ SUMMARY_WORK_BUDGET;
+ }
+ let state = self.existence_summary.as_mut().unwrap();
+ let len = remaining.min(state.rows_until_yield);
+ state.rows_until_yield -= len;
+ len
+ }
+
+ /// Consume the whole inner key group into owned summary state. Continue
+ /// draining after saturation so input errors and group boundaries retain
+ /// their ordinary behavior.
+ async fn summarize_inner_key_group(&mut self) -> Result<()> {
Review Comment:
The rebased code recognizes the comparison once per stream, reuses the
existing traversal, and bounds the scan for early matches. I kept the
common-path controls in the PR description. The first matched run showed +5.2%
mean time on the unfiltered control with head processes on opposite sides of
base. In a follow-up with eight paired runs pinned to one CPU, head took less
time in all eight pairs (geometric mean head/base 0.938). Q13 was also lower in
all eight pairs (0.911), and both revisions returned the same 900,000 rows.
These are controls for overhead; the unfiltered case bypasses the new
algorithm. [Operator and Q27
results](https://github.com/sunchao/arrow-datafusion/actions/runs/42738952097)
ยท [Paired controls and Q13
parity](https://github.com/sunchao/arrow-datafusion/actions/runs/42743163590).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
