Doris-Breakwater commented on issue #70174: URL: https://github.com/apache/doris/issues/70174#issuecomment-6622267946
Initial triage: this warrants high-priority investigation by the FE/Nereids optimizer maintainers because a small `EXPLAIN` reportedly exhausts the FE heap. The reproduction and controls already narrow the investigation substantially. The plain-`EXISTS` failure predates 4.1.5; the additional `OR EXISTS` failure should be tracked as a possible expansion of that existing failure path. I reviewed the exact source commits `917b20be83878e98f94e3ebf7efccc80346e3e50` and `b10073ad9ca17cd5685c4dd3b3ef650f256376d0`. This is a source-based assessment; I have not independently run the SQL reproduction or confirmed the OOM allocation path. **What the release source establishes** 1. **There is a concrete memo-growth candidate beyond simply swapping the same two joins.** `SemiJoinSemiJoinTransposeProject` constructs a projection over the old bottom join's right input using only the slots needed by the outer projection and bottom join predicates. If the predicates no longer require any right-input slots, `CBOUtils.newProjectIfNeeded` can construct an empty projection. `LogicalProject` replaces that with `Alias(TinyIntLiteral(1))`, and `Alias(Expression)` allocates a fresh `ExprId`. Alias identity participates in projection equality, and memo deduplication compares the operator and child groups. Consequently, repeated construction of these existence-only projections can produce distinct memo expressions despite equivalent meaning. This is a specific mechanism to inspect in a rule trace, **not yet proof that this reproduction reaches it repeatedly**. Sources: [transpose rule](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/ doris/nereids/rules/exploration/join/SemiJoinSemiJoinTransposeProject.java#L103-L124), [projection helper](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/CBOUtils.java#L46-L60), [empty-projection handling](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java#L84-L101), [alias identity](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java#L55-L62). There are termination mechanisms: the transpose sets `hasLAsscom`, its matcher checks reorder context, and memo insertion deduplicates existing expressions. Thus “the two joins swap forever” needs evidence showing how fresh projections, changed child groups, or other transformations sustain the sequence. Disabling several different rules can interrupt the same sequence without identifying its initiating defect. 2. **The job queue has a separate amplification mechanism worth measuring.** In [ApplyRuleJob.execute](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/ApplyRuleJob.java#L73-L139), `deriveStatsJobs` is allocated before the matched-plan loop; every iteration pushes the entire accumulated list without clearing it. If each of N matches adds one job, this schedules N(N+1)/2 stack entries, including repeated references to earlier jobs. The stats-derived guard can skip their later execution, but does not prevent those queue entries from being retained first. Also, [pattern matching eagerly assembles matching plan combinations](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/GroupExpressionMatching.java#L55-L195). These paths can consume time/memory within one scheduler job. The memo expression limit does not bound all of these allocations. Their actual contribution here requires allocation/queue evidence. 3. **The referenced mark-join change is a plausible trigger, not an established cause of the older failure.** `d6fdff126d0` (#66482 / #67280) is present in the reported 4.1.5 commit and absent from the reported 4.1.1 commit. Its inference feeds [SubqueryToApply](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java#L507-L525), which explicitly drops the mark slot when conversion to a plain semi join is allowed. That supports the reported plan-shape difference. The transpose change from #66574 adds a guard against a dangling mark slot; it does not address empty-projection identity or queue growth. #69814 (`f2156159c19`, backport of #69748) is absent from both reported commits and changes constant-project pushdown. It deserves an isolated A/B test, but there is no basis yet to call it a fix for this OOM. 4. **The two timeout concerns are distinct.** [SimpleJobScheduler](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/scheduler/SimpleJobScheduler.java#L35-L53) checks only between jobs; matching, transformation, insertion, and queue construction can run without an internal deadline check. It also truncates elapsed time to seconds and uses `>`, so a dump exactly at 5 seconds with a 5-second setting is not itself evidence of a missed check. The later OOM still merits investigation of job duration and GC pressure. For cloud mode, the reported timeout substitution is supported directly by the source: [waitForAutoStart](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java#L1701-L1743) reaches the wrapper even for `NORMAL`; [the wrapper](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java#L1869-L1900) sets `warmup=true` without restoring it. Both the scheduler and `RewriteJob.checkTimeout` then select `auto_start_wait_to_resume_times` instead of the session timeout. With that configuration at 300, this explains the 300-versus-60-second threshold if this path was reached. Please file a separate linked issue so timeout accounting/restoration and optimizer memory growth can be tested independently. **What remains unproven, and the most useful next evidence** - Capture `EXPLAIN REWRITTEN PLAN` as `rp_reader` for one key versus two keys, sampled versus full statistics, and both policy forms on the two reported versions. This [returns before cascades optimization](https://github.com/apache/doris/blob/917b20be838/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java#L297-L308). It should establish whether the correlation predicates disappear, which join conditions remain, and where mark slots differ. - Export the actual table/column statistics used in each comparison: row counts, NDVs, null counts, min/max, and `hot_values`, plus effective planner settings. `FilterEstimation` consumes hot-value frequencies, so they can change estimates; their causal role here is unconfirmed. In particular, the reported 4.1.1 failure with null `hot_values` means populated hot values are not a necessary condition across both versions. - On an isolated reproduction FE, collect the complete OOM stack and GC log, several timestamped planner thread samples spanning the timeout, and a short allocation recording or heap histogram. Instrument rule applications with input/output group IDs, projection `ExprId`s, memo size, match count, and pending-job count. This distinguishes sustained new-expression creation, finite combinatorial enumeration, and duplicate stats-job scheduling. A single `DeriveStatsJob` stack sample cannot distinguish them. - Turn the supplied matrix into regression coverage, including bounded planning/queue growth and timeout behavior. Test #69814 independently; test cloud `NORMAL`, actual resume, and exception paths against the intended timeout budget. **Interim mitigation:** the reported single-rule disable is the most targeted tested workaround. Prefer session scope where operationally practical, retaining any existing disabled-rule entries, and validate both planning latency and execution plans. The reported allowed/denied checks are useful; extend them to the admin policy form before broader rollout. Full statistics are another reported bypass, but do not establish that later sampled auto-analysis cannot reintroduce the trigger. Neither increasing FE heap nor changing statistics should be presented as a root-cause fix. Breakwater-GitHub-Analysis-Slot: slot_41c01197ea78 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
