englefly opened a new pull request, #70516: URL: https://github.com/apache/doris/pull/70516
### Background Runtime filters from an ancestor hash join can already reach scans through lower joins. Equality expansion currently adds targets across INNER/SEMI joins, but does not derive targets on the non-preserved side of an OUTER join. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: In `(L LEFT JOIN R ON L.k = R.k) JOIN E ON L.k = E.k`, the runtime filter built from `E.k` can filter `L.k`, while `R` still scans and builds rows that cannot match any needed `L` row. This change derives an additional target on `R.k`, and handles RIGHT OUTER joins symmetrically. The existing RF producer, target registration and translation paths are reused; most of the diff adds coverage for plan generation and execution semantics. ### Details - **Change map:** 31 lines of production changes, 275 lines of test-source changes, and 206 lines of automatically generated expected results. - **Before and after:** An ancestor RF can now reach both the preserved join key and the corresponding non-preserved key. Expansion is directional and limited to ordinary equality between direct column references with identical source/probe/target types. Direct aliases retain column metadata; projected computed expressions are excluded. FULL OUTER, mark joins and decoupled RF expansion are excluded, and existing NULL-safe producer restrictions remain in effect. - **Configs and defaults:** Reuses the existing `expand_runtime_filter_by_inner_join` session setting, which defaults to true. No configuration or protocol fields are added. - **Behavior change:** Eligible queries receive additional RF consumers. Join results, including unmatched preserved rows, remain unchanged. - **Impact on existing users:** Additional filtering can reduce rows entering shuffle and hash build. Storage-read and elapsed-time improvements depend on selectivity, storage layout and RF arrival time; no workload-level performance improvement is claimed here. ### Testing - `bash run-fe-ut.sh --run org.apache.doris.nereids.postprocess.RuntimeFilterTest`: 57 tests passed. - `MVN_OPT='-Dmaven.build.cache.enabled=false' FE_MAVEN_THREADS=4 bash build.sh --fe --exclude-obs-dependencies -j4`: FE build passed, including Checkstyle with zero violations. Verified that the deployed FE jar contains the new expansion guard. - `bash run-regression-test.sh --conf regression-test/conf/regression-conf-gj29031.groovy --run -d query_p0/runtime_filter -s outer_join_runtime_filter -genOut`: generated expected output on a local single-BE cluster. All eight expansion-disabled/enabled result pairs are identical; the plans show the added address/phone scan targets. - `bash run-regression-test.sh --conf regression-test/conf/regression-conf-gj29031.groovy --run -d query_p0/runtime_filter,correctness_p0 -s outer_join_runtime_filter,test_runtime_filter_outer_join_nullable_side`: both suites passed against the saved output. Coverage includes IN, BLOOM, MIN_MAX and IN_OR_BLOOM, plus empty non-preserved inputs and empty producers. - No workload-level performance benchmark was run. ### Notes for Reviewers Self-review checkpoints: - Goal and scope: production changes are confined to `RuntimeFilterPushDownVisitor#visitPhysicalHashJoin` and imports; `RuntimeFilterGenerator` and `RuntimeFilterTranslator` need no changes. - Concurrency: standalone ancestor-build and descendant-build pipelines remain independent. Execution coverage waits for the added RF consumers; non-blocking context propagation is also tested. - Lifecycle: reuses `withNewProbeExpression`, preserving the builder, source, filter type, expression order, NDV and non-blocking requirement; no new ownership or static initialization is introduced. - Parallel paths: covers LEFT/RIGHT OUTER joins, reversed ON operands, nested joins, aliases and shared CTE consumers. Existing shared-producer eligibility checks remain in place. Nested-loop join expansion is outside this hash-equality change. - Conditional checks: comments explain the directional proof, identical-type restriction and rejection of expressions normalized into projected slots. - Coverage and results: FE tests check target registration and serialized consumers; ordered regression results compare expansion disabled/enabled and cover NULLs, duplicates, composite-key cross combinations, empty producers and empty non-preserved inputs. Expected output is generated by the regression runner. - Observability and cost: uses existing Explain RF targets and execution RF counters; target registration and legacy grouping remain unchanged. Expansion follows child edges and uses existing per-scan deduplication. - Compatibility, persistence and writes: no Thrift/protobuf IDs, FE/BE variables, storage formats, metadata journals, transactions or data-write paths change. ### Release note Runtime filter expansion can now derive filters from the preserved side to matching keys on the non-preserved side of LEFT/RIGHT OUTER joins, reducing unnecessary downstream processing for eligible queries. ### Check List (For Author) - Test: See Testing. - Behavior changed: Yes (see Details). - Does this need documentation: No; optimizer enhancement using an existing setting. - Pre-submission status: Ready. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Implementation, tests and PR description. - Human review: The user reviewed the optimization approach; the generated diff awaits human review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
