924060929 opened a new pull request, #70666:
URL: https://github.com/apache/doris/pull/70666

   ### Background
   
   Nereids stores equivalent plans in Memo groups. Regular logical and physical 
expressions connect these groups into a directed acyclic graph, and statistics 
derivation visits children before their parents. When an inserted expression 
already exists in another group, `Memo.mergeGroup` combines the groups and 
redirects their parent expressions.
   
   ### What problem does this PR solve?
   
   Issue Number: close #70174
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary: A one-key join of two tables with correlated `EXISTS` row 
policies can exhaust the FE heap during planning with sampled statistics. 
Project merging finds an existing expression in an indirect descendant group, 
but the old merge guard checks only immediate parent-child relationships. This 
change rejects a merge whenever either group can reach the other through 
regular logical or physical expressions. Most of the diff adds unit and SQL 
regression coverage; valid merges with shared children and enforcers remain 
supported.
   
   ### Details
   
   - **Change map:** 39 changed lines in `Memo.java`; 108 unit-test lines, 180 
SQL-regression lines, and 26 automatically generated result lines.
   - **Frequency:** Deterministic with the sampled-statistics, single-key 
reproducer in #70174 on 4.1.5, for both plain `EXISTS` and `OR EXISTS` 
policies. No production incident-frequency estimate.
   - **Root cause:** `ApplyRuleJob -> Memo.copyIn -> insertGroupExpression -> 
mergeGroup` contracts an ancestor and an indirect descendant. A reduced example 
is `G3(Project k) -> G2(Project 1,k) -> G1(Project k) -> G0(Scan k)`: inserting 
`Project(k,G0)` into G3 finds G1, and merging G3 into G1 creates `G1 -> G2 -> 
G1`. `DeriveStatsJob` then keeps scheduling unfinished children; a reproduced 2 
GiB OOM heap contained about 41.9 million `DeriveStatsJob` objects, with only 
2,561 Memo expressions.
   - **Why existing tests missed it:** Existing merge tests cover ordinary 
merges and direct relationships, but do not validate the indirect cycle 
produced when project merging rediscovers an expression in a descendant group.
   - **Fix:** Check reachability in both directions before mutating either 
group. Use an iterative traversal with a visited set to handle shared 
descendants without recursive stack growth. Enforcers are stored separately and 
intentionally excluded because their self-reference is valid.
   - **Other paths and branches:** The common entry also protects recursive 
parent-expression deduplication. Tests cover direct and deeper logical 
dependencies, an indirect physical dependency, and a permitted merge with 
shared children and an enforcer. This PR targets master; the same patch was 
independently built and exercised on the 4.1.5 source base. The separately 
reported cloud warmup-timeout behavior is outside this group-cycle fix.
   - **Behavior change:** Merges that would create an indirect dependency cycle 
are refused, as direct-cycle merges already were. No configuration, RPC, or 
persisted format changes. Reachability checks are linear in the reachable 
groups and expression edges; the SQL timings below are functional validation, 
not a planner-throughput benchmark.
   
   ### Testing
   
   - Before the fix, the reduced `testIndirectProjectMergeMustNotCreateCycle` 
fails `MemoValidator` on the 4.1.5 base. With the patch, `MemoTest` runs 42 
tests with 0 failures, 0 errors, and 1 existing skipped test.
   - Master: `MemoTest`, `GroupExpressionTest`, and `StructInfoMapTest` through 
`run-fe-ut.sh`: 47 tests, 0 failures, 0 errors, 1 existing skipped test. The 
final test initialization uses the existing `MemoTestUtils` factory to supply 
master's statement context.
   - Clean master FE build with Checkstyle: 
`MVN_OPT='-Dmaven.build.cache.enabled=false' DISABLE_BUILD_UI=ON 
DISABLE_BE_JAVA_EXTENSIONS=ON ./build.sh --fe --clean -j4 --output 
output/fixed-master`.
   - Patched 4.1.5 FE with original 4.1.5 BE, ARM64, 2 GiB FE heap: the 
unmodified public issue's single-key `EXPLAIN`, sampled statistics, and no 
disabled rules complete in 0.591 s (plain policy) and 0.297 s (OR policy). Both 
previously exhausted the same heap.
   - The 4.1.5 SQL regression output was generated by `run-regression-test.sh`, 
inspected, then checked by a second run. It covers both policy forms, one/two 
keys, admin bypass, a granted reader, and a rejected reader; granted/admin 
cases have nonempty expected rows.
   - Master FE with original 4.1.5 BE, ARM64, 2 GiB FE heap: after rerunning 
sampled analysis, the public single-key EXPLAIN completes in 0.081 s (plain 
policy) and 0.167 s (OR policy), with no disabled rules and FE survival 
checked. The full master SQL suite could not execute against this older BE: its 
first fixture INSERT is rejected with `Unsupported exec type in pipeline: 
Invalid plan node type`. Full matching-version master FE/BE regression remains 
for CI; this is not counted as a passing master runtime suite.
   
   ### Release note
   
   Fix FE out-of-memory failures during planning when equivalent Memo groups 
have an indirect dependency, including joins protected by correlated EXISTS row 
policies.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No.
   - Pre-submission status: Partially verified: local master unit tests, FE 
build and Checkstyle, and the patched 4.1.5 end-to-end regression passed. Full 
matching-version master FE/BE SQL regression is pending CI (see Testing).
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex.
   - Generated: Production fix, unit tests, SQL regression, and this 
description. The result file was generated by the regression runner.
   - Human review: Pending author review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to