jayzhan211 opened a new pull request, #26016:
URL: https://github.com/apache/datafusion/pull/26016
## Which issue does this PR close?
- Follow-up to #26015 (stacked on it; review only the last commit until it
merges).
## Rationale for this change
In #26015 a hash join partition that falls back to a sort-merge join reports
itself to the dynamic filter like a canceled partition, so the probe rows
routed
to it are not pruned at all. Its join key bounds are still exact, though:
only
the membership test (hash table / IN list) is missing. This keeps the bounds
pruning for such partitions.
## What changes are included in this PR?
- New `PushdownStrategy::Unknown`: "this partition may hold rows, but there
is
no structure to test membership against". It contributes its bounds and no
membership term. It is deliberately distinct from `Empty`, which proves the
partition holds nothing and gives it a `false` branch; reporting `Empty`
here
would silently drop matches.
- The fallback computes min/max bounds (and whether any key is NULL) while
feeding the build side into the sort, and reports `Unknown` with them.
- `HashJoinStream::schedule_build_report` is extracted so the hash and the
fallback paths share the report.
## What is the testing strategy for this PR?
- `shared_bounds`: two tests for the pruning semantics of `Unknown` (keeps
routed rows, still applies its bounds).
- `exec.rs`: `sort_merge_fallback_keeps_the_dynamic_filter_permissive` now
asserts a key outside the build side's range is pruned and keys inside are
kept. Swapping `Unknown` for `Empty` makes it fail.
## Are there any user-facing changes?
No. Less data is scanned on the probe side when a partition falls back with
dynamic filter pushdown enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
