Doris-Breakwater commented on issue #70679: URL: https://github.com/apache/doris/issues/70679#issuecomment-6944245498
Breakwater-GitHub-Analysis-Slot: slot_475cdaa28e80 **Initial assessment:** The reported stack and MAT screenshots support an FE planning-memory amplification investigation. Source inspection confirms a mechanism that can propagate very large `tblRefIds` lists through shared CTE producers, exchanges, and joins. The incident-specific trigger remains unconfirmed without the SQL/plan or list contents. The allocation failing in `HashJoinNode` can be the final amplification point rather than the original source of growth. **1. How the lists can grow** I checked the local source at Apache Doris **4.1.0**, commit `5960d4cea0e6`, and **4.1.5-release**, commit `917b20be8387`: - The [PlanNode copy constructor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java#L198-L206) allocates a new list containing all child `tblRefIds`; it copies references, not new `TupleId` objects. - The [Nereids HashJoinNode constructor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java#L108-L119) concatenates both child lists without deduplication. - An [ExchangeNode](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/ExchangeNode.java#L65-L102) first inherits that full list through its superclass constructor. When an output descriptor exists, `updateTupleIds()` replaces `tupleIds` but **preserves the inherited `tblRefIds` and appends another ID**. Repeating this call appends again, so this branch is not idempotent for `tblRefIds`. The null-descriptor branch clears all three collections before repopulating them. - In [CTE consumer/distribution translation](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java#L1490-L1559), a consumer returns the existing producer's multicast fragment. [Distribution translation](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java#L355-L434) constructs an exchange over its root, then updates that exchange with the consumer sink's output descriptor. Thus a consumer can carry the producer's accumulated ancestry rather than only its own output identity. **Hypothesis to verify:** successive CTEs join multiple references to earlier materialized CTEs. This forms a compact shared plan graph, but concatenating ancestry lists can effectively expand its paths exponentially. Ordinary copying over an unshared tree alone does not establish exponential growth. For illustration, if each level joins two consumers of the preceding producer, and both exchange updates have non-null output descriptors, the code permits `L(next) = 2 * (L(previous) + 2)`. Starting at one entry, 25 levels yield **167,772,156 entries**; a subsequent copy-and-append with 1.5× ArrayList growth can allocate capacity **251,658,234**. These are close to the reported array sizes. This is checked arithmetic derived from the code, **not a Doris SQL reproduction or proof of the actual CTE topology**. The non-clearing append alone adds one entry per call; copying the accumulated lists across shared dependencies is the potential multiplier. **2. Relevant upstream change and release status** There is a directly relevant merged change: [PR #57764, “remove useless code in PlanNode”](https://github.com/apache/doris/pull/57764), commit [`9433adea8a81`](https://github.com/apache/doris/commit/9433adea8a81297a8eef1a95fb47ca930d9a8574). Its diff removes the `tblRefIds` field, copy-constructor propagation, hash-join concatenation, and exchange appends. Current master also lacks those operations. This removes this particular allocation mechanism; the PR was a cleanup and does not itself demonstrate a reproduction of this incident. The relevant operations remain in **4.1.0, 4.1.5-release, and current branch-4.1** (checked at `e1039662cb13`). Therefore I cannot identify a verified fixed **4.1 release**, and lack of reproduction on 4.1.5 does not demonstrate a fix. Maintainers should assess a focused backport of the metadata removal and its dependencies, with a regression test, rather than assume a version upgrade resolves this case. A search of the inspected 4.1.0 FE production sources found `getTblRefIds()` callers only propagating these lists, which further supports reviewing whether the metadata can be removed. **3. Evidence needed to confirm the trigger** Please provide sanitized material preserving the CTE reference graph and join structure: 1. SQL plus table/view DDL, exact FE build/commit, JDK version, and relevant session settings, particularly `enable_cte_materialize` and `inline_cte_referenced_threshold`. State whether the same SQL, settings, schema, and statistics were tested on 4.1.5. 2. For the largest MAT arrays, the precise field path, ideally `PlanNode.tblRefIds -> ArrayList.elementData`, the list's **size versus backing-array length**, and a bounded sample/frequency summary of the contained tuple IDs. Retained heap and array capacity alone do not identify the list's logical contents. 3. Node IDs/types and child relationships for the largest exchanges/joins, correlated with CTE producers and consumers. On an isolated diagnostic run, record only list sizes and output tuple IDs at exchange construction/update and before join concatenation; do not stringify enormous lists. Locate the first unexpectedly large list and determine whether its growth follows the recurrence above. 4. The query ID, timestamp-aligned FE/audit/GC logs, full planning-thread stack, and concurrent planning count. MAT paths to GC roots can distinguish objects retained by the active planning thread from objects retained after planning. The screenshots show an active translator thread, so they do not by themselves prove a persistent leak. As noted in the report, retained-heap totals may overlap. **4. Interim containment and maintainer next steps** - Prevent automatic resubmission of the offending SQL. A SQL text/hash block rule can reject a known query before planning: in [4.1.0 StmtExecutor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L745-L753), the regex/hash check precedes `planner.plan()`. Validate rule matching and scope with the actual submitted SQL. - Test splitting the dependency chain into separate statements that persist intermediate results to tables, then scan those tables. This cuts planner metadata ancestry at statement boundaries. Merely adding another CTE does not provide that boundary. Validate result semantics and intermediate-data cost. - Do not treat disabling CTE materialization as a generally safe workaround: [CTEInline](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInline.java#L90-L119) deep-copies the producer for each inlined consumer, potentially expanding the actual plan instead. Test any such change only with the reproducer on an isolated FE. `EXPLAIN` can also reach translation and should be tested there first. - The inspected exchange/join copy operations have no per-query allocation-budget check. Lower BE execution-memory limits, optimizer timeouts, or a larger FE heap should not be presented as a guaranteed guard against these Java array allocations. A controlled planning-memory/metadata limit would need explicit implementation and tests before allocation. Suggested triage: **FE / Nereids planning memory, with FE availability impact; reproducer needed**. Next, confirm the CTE graph and duplicate-ID growth, reduce it to a small regression case, and evaluate the upstream removal on the affected branch. No product code was modified, and no SQL reproduction or end-to-end fix validation has been performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
