sunchao commented on code in PR #25584:
URL: https://github.com/apache/datafusion/pull/25584#discussion_r4520462937
##########
datafusion/physical-plan/benches/sort_merge_join.rs:
##########
@@ -248,5 +251,167 @@ fn bench_smj(c: &mut Criterion) {
group.finish();
}
-criterion_group!(benches, bench_smj);
+/// Compare execution with summaries enabled and disabled in the same binary.
+/// Inputs are already sorted: SQL versions of these EXISTS/NOT EXISTS queries
+/// also measure sorting and depend on the optimizer's choice of join
algorithm.
+/// These cases isolate the residual semi/anti join, including output
collection.
+fn bench_existence_summary(c: &mut Criterion) {
+ let rt = Runtime::new().unwrap();
+ let mut group = c.benchmark_group("sort_merge_join_existence_summary");
+ group.sample_size(10);
+ group.warm_up_time(std::time::Duration::from_millis(500));
+ group.measurement_time(std::time::Duration::from_secs(2));
+
+ // Vary group count, rows per side, residual selectivity, and join type.
+ // The early-witness case controls for the generic join's short circuit;
+ // the small-group case measures the cost of repeatedly resetting
summaries.
+ for (name, groups, probe_rows, inner_rows, distinct, op, kind) in [
Review Comment:
The matched run now includes the first-row and third-row range/inequality
controls, singleton no-match, and seven-row groups with shared and isolated
pools. The singleton Utf8 no-match case moved from 36.87 ms to 0.958 ms; the
Utf8 first-row range controls were within 4% of base. The adverse unfiltered
control is recorded in the PR description and remains under discussion on the
common-path thread. [Full
results](https://github.com/sunchao/arrow-datafusion/actions/runs/42738952097).
##########
datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs:
##########
@@ -707,6 +721,127 @@ impl BitwiseSortMergeJoinStream {
}
}
+ /// Try initial probes or an exact summary, updating the matched bits.
+ /// Return true when this outer slice needs no ordinary filter evaluation.
+ /// Keep this work separate from the asynchronous spill and batch
traversal.
+ #[inline(never)]
+ fn try_apply_summary(
+ &mut self,
+ outer_slice: &RecordBatch,
+ summary: &mut Option<Box<CachedSummary>>,
+ allow_summary: &mut Option<bool>,
+ ) -> Result<bool> {
+ let outer_group_start = self.outer_offset;
+ let outer_group_len = outer_slice.num_rows();
+ let filter = self.filter.as_ref().unwrap();
+
+ // Probe a small prefix before paying for min/max. This cost heuristic
+ // preserves early witnesses without adding an unconditional inner
+ // scan. A single outer row also stays on the ordinary path, which
+ // already visits each inner row at most once.
+ if summary.is_none()
+ && outer_group_len > 1
+ && *allow_summary.get_or_insert_with(|| {
+ self.inner_key_buffer
+ .iter()
+ .map(RecordBatch::num_rows)
+ .sum::<usize>()
+ >= 7
+ })
+ {
+ if let Some(comparison) = SemiAntiComparison::try_new(
+ filter,
+ self.outer_is_left,
+ self.outer.schema().as_ref(),
+ self.inner.schema().as_ref(),
+ )? {
+ let mut remaining = 2;
Review Comment:
Implemented the doubling-prefix scan in 9608e5bd01 and added the
third-witness fixture. On the same runner, the Utf8 case took 0.632 ms on base
and 0.627 ms on head. Int64 took 0.570 ms and 0.565 ms. Both revisions used
identical fixtures. [Measurements and raw
samples](https://github.com/sunchao/arrow-datafusion/actions/runs/42738952097).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
