jayzhan211 opened a new issue, #24962:
URL: https://github.com/apache/datafusion/issues/24962
### Describe the bug
`HashJoinExec` and `NestedLoopJoinExec` report `EmissionType::Incremental`
for `JoinType::LeftSemi`, but neither operator emits any `LeftSemi` output
during the probe phase. All output rows are produced from the build-side
visited bitmap after the probe side is exhausted, which is
`EmissionType::Final` behaviour.
The comment next to the classification says the incremental group is the
joins that "only need to generate matched rows from the probe side". That is
not true for `LeftSemi` in these operators, which output build-side rows.
## Where
- `HashJoinExec::compute_properties` and
`NestedLoopJoinExec::compute_properties` both place `LeftSemi` in the
`EmissionType::Incremental` arm, alongside `Inner`, `Right`, `RightSemi`,
`RightAnti` and `RightMark`.
- `adjust_indices_by_join_type` in `joins/utils.rs` returns empty index
arrays for `LeftSemi | LeftAnti | LeftMark` during the probe phase, so nothing
is emitted while probing.
- `need_produce_result_in_final` includes `LeftSemi`, and
`get_final_indices_from_shared_bitmap` produces the matched build rows for it
once the probe side is done.
So the operator itself already treats `LeftSemi` as "produce in final", and
only the reported plan property disagrees.
## Why it matters
1. **Wrong plan property.** Anything that inspects `pipeline_behavior()`
gets a misleading answer for `LeftSemi` hash and nested loop joins.
2. **Unbounded probe side is not rejected.** With a bounded build side and
an unbounded, incrementally emitting probe side, the join is labelled
`Incremental` and passes pipeline checks, yet it can never emit a row because
it waits for probe exhaustion. `LeftAnti` and `LeftMark` in the same situation
are labelled `Both`; `LeftSemi` should get the same treatment at minimum.
3. **Inconsistent predicates.** `maintains_input_order` reasons about which
joins emit from the bitmap at the end and includes `LeftSemi`, while the
emission classification excludes it. Two different predicates for the same
underlying fact invite future drift.
### To Reproduce
```rust
// bounded left, unbounded incremental right, join_type = LeftSemi
let join = HashJoinExec::try_new(left, right, on, None, &JoinType::LeftSemi,
None,
PartitionMode::CollectLeft, NullEquality::NullEqualsNothing)?;
assert_eq!(join.pipeline_behavior(), EmissionType::Incremental); // passes
today
// but executing it produces no output until `right` ends, which never
happens
```
### Expected behavior
`LeftSemi` should be reported as `EmissionType::Final` for `HashJoinExec`
and `NestedLoopJoinExec`, since the operators produce nothing before the probe
side completes. If `Final` turns out to have unwanted planning side effects,
`Both` is still more accurate than `Incremental` and matches how `LeftAnti` and
`LeftMark` are handled.
Ideally the classification and `maintains_input_order` derive from the same
predicate (`need_produce_result_in_final` or a renamed equivalent) so they
cannot disagree.
### Additional context
`SortMergeJoinExec` is unaffected. It reports `Incremental` unconditionally
and does emit `LeftSemi` rows as it streams.
Surfaced during review of #24957, where the comment was reworded to
"everything else is emitted incrementally", which made the mismatch more
visible. The classification itself predates that PR and is unchanged by it.
#24957 keeps the existing `Incremental` label for `LeftSemi` to avoid a
behaviour change inside a refactor; this issue tracks fixing the label
separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
