CalvinKirs opened a new pull request, #69662:
URL: https://github.com/apache/doris/pull/69662
### What problem does this PR solve?
Issue Number: None
Related PR: #28010
Problem Summary:
A hidden delete-sign Filter can break a useful runtime-filter chain in a
three-table join. Consider this reduced physical plan (Project and Distribute
nodes omitted):
```text
Join (e.bloc_id = b.id) RF1: b.id -> e.bloc_id
Join (f.agent_code = e.agent_code) RF0: e.agent_code -> f.agent_code
Scan fact f
Filter (e.__DORIS_DELETE_SIGN__ = 0)
Scan agent e receives RF1
Filter (b.bloc_code = 'keep')
Scan bloc b
```
The selective bloc predicate keeps RF1. The agent scan is then recognized as
an effective source because it receives RF1, but `visitPhysicalFilter`
previously discarded that information when the Filter contained only hidden
columns. The inner join could therefore prune RF0 based on unknown or
non-selective static join-column statistics, losing the filtering chain and
increasing fact-table scans.
Reuse the existing `propagateEffectiveSrc` helper before classifying visible
predicates. Hidden-only Filters inherit the child's existing REF/NATIVE type; a
visible predicate still marks its Filter NATIVE. A delete-sign predicate with
no effective child source still does not keep an otherwise ineffective RF.
HashAggregate's existing helper behavior is unchanged.
Add planner unit tests covering known/non-selective and unknown statistics,
hidden-only negative controls, and a visible-predicate control. Add EXPLAIN
regression coverage for the retained chain and both controls.
### Release note
Fix excessive fact-table scans caused by pruning effective chained runtime
filters across hidden delete-sign Filters.
### Check List (For Author)
- Test
- [x] Regression test
- `rf_prune_hidden_filter` and existing `rf_prune`: both passed
using an isolated FE with a mocked BE (planner/EXPLAIN validation).
- [x] Unit Test
- `RuntimeFilterPrunerTest`: 5 passed; existing `RuntimeFilterTest`:
50 passed.
- With the production fix reverted, both new parameterized chain
cases failed (expected 2 RFs, actual 1); the other three controls passed.
- FE Checkstyle: zero violations; whitespace check passed.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Preserve existing effective-source information through hidden
Filters so useful chained RFs survive pruning. Hidden predicates alone remain
non-selective.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Validation limits
The regression checks exercise planning with a mocked BE; real BE data
filtering and production performance were not benchmarked. The existing
effective-source policy remains heuristic, including decoupled RF dependencies.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
