SubhamSinghal opened a new pull request, #24956:
URL: https://github.com/apache/datafusion/pull/24956
## Which issue does this PR close?
Part of #17427.
## Rationale for this change
`PiecewiseMergeJoinExec`'s metrics accounting had several gaps left over
from earlier PRs in the epic:
- The classic-join stream (`Left`/`Right`/`Full`/`Inner`) never routed
`poll_next` through `BaselineMetrics::record_poll`, so `output_rows`,
`output_bytes`, and `output_batches` stayed at `0` in `EXPLAIN ANALYZE`
regardless of how many rows the join actually produced.
- `join_time` was never measured on either the classic or existence-join
stream — only `build_time` was timed, so `elapsed_compute` understated total
operator cost for large probe sides.
- `probe_hit_rate`/`avg_fanout` exist on the shared
`BuildProbeJoinMetrics` struct (populated by `HashJoinExec`) but PWMJ never
populated them, always showing `N/A (0/0)`.
## What changes are included in this PR?
- `ClassicPWMJStream::poll_next` now calls
`self.join_metrics.baseline.record_poll(poll)`, matching the existence-join
stream.
- `join_time` is timed around the actual comparison work:
`resolve_classic_join` and the `ProcessUnmatched` bitmap/take pass on the
classic path; `extreme_key` + `mark_matched_buffered_rows` on the existence
path.
- `probe_hit_rate`/`avg_fanout` are populated for classic join, using the
range size (`buffered_len - buffer_idx`) already computed at each match as the
fanout.
- `probe_hit_rate` is populated for existence join: a streamed batch
counts as a hit if its extreme key lowers the shared watermark, a miss
otherwise. `avg_fanout` is intentionally left unset for existence join — its
watermark-based semantics don't have a natural per-row fanout equivalent.
- Added a metrics-focused test to each stream module:
`classic_join::tests::inner_join_records_output_and_probe_metrics` and
`existence_join::tests::probe_hit_rate_counts_batches_that_advance_the_watermark`,
both hand-derived and verified against actual runs.
## Are these changes tested?
Yes — two new unit tests
## Are there any user-facing changes?
`EXPLAIN ANALYZE` on a `PiecewiseMergeJoinExec` plan now reports accurate
`output_rows`/`output_bytes`/`output_batches`/`join_time`/`probe_hit_rate`/`avg_fanout`
instead of zeros/`N/A`. No API or behavior change to query results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
