Doris-Breakwater commented on issue #70679:
URL: https://github.com/apache/doris/issues/70679#issuecomment-6944245498

   Breakwater-GitHub-Analysis-Slot: slot_475cdaa28e80
   
   **Initial assessment:** The reported stack and MAT screenshots support an FE 
planning-memory amplification investigation. Source inspection confirms a 
mechanism that can propagate very large `tblRefIds` lists through shared CTE 
producers, exchanges, and joins. The incident-specific trigger remains 
unconfirmed without the SQL/plan or list contents. The allocation failing in 
`HashJoinNode` can be the final amplification point rather than the original 
source of growth.
   
   **1. How the lists can grow**
   
   I checked the local source at Apache Doris **4.1.0**, commit `5960d4cea0e6`, 
and **4.1.5-release**, commit `917b20be8387`:
   
   - The [PlanNode copy 
constructor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java#L198-L206)
 allocates a new list containing all child `tblRefIds`; it copies references, 
not new `TupleId` objects.
   - The [Nereids HashJoinNode 
constructor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java#L108-L119)
 concatenates both child lists without deduplication.
   - An 
[ExchangeNode](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/planner/ExchangeNode.java#L65-L102)
 first inherits that full list through its superclass constructor. When an 
output descriptor exists, `updateTupleIds()` replaces `tupleIds` but 
**preserves the inherited `tblRefIds` and appends another ID**. Repeating this 
call appends again, so this branch is not idempotent for `tblRefIds`. The 
null-descriptor branch clears all three collections before repopulating them.
   - In [CTE consumer/distribution 
translation](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java#L1490-L1559),
 a consumer returns the existing producer's multicast fragment. [Distribution 
translation](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java#L355-L434)
 constructs an exchange over its root, then updates that exchange with the 
consumer sink's output descriptor. Thus a consumer can carry the producer's 
accumulated ancestry rather than only its own output identity.
   
   **Hypothesis to verify:** successive CTEs join multiple references to 
earlier materialized CTEs. This forms a compact shared plan graph, but 
concatenating ancestry lists can effectively expand its paths exponentially. 
Ordinary copying over an unshared tree alone does not establish exponential 
growth.
   
   For illustration, if each level joins two consumers of the preceding 
producer, and both exchange updates have non-null output descriptors, the code 
permits `L(next) = 2 * (L(previous) + 2)`. Starting at one entry, 25 levels 
yield **167,772,156 entries**; a subsequent copy-and-append with 1.5× ArrayList 
growth can allocate capacity **251,658,234**. These are close to the reported 
array sizes. This is checked arithmetic derived from the code, **not a Doris 
SQL reproduction or proof of the actual CTE topology**. The non-clearing append 
alone adds one entry per call; copying the accumulated lists across shared 
dependencies is the potential multiplier.
   
   **2. Relevant upstream change and release status**
   
   There is a directly relevant merged change: [PR #57764, “remove useless code 
in PlanNode”](https://github.com/apache/doris/pull/57764), commit 
[`9433adea8a81`](https://github.com/apache/doris/commit/9433adea8a81297a8eef1a95fb47ca930d9a8574).
 Its diff removes the `tblRefIds` field, copy-constructor propagation, 
hash-join concatenation, and exchange appends. Current master also lacks those 
operations. This removes this particular allocation mechanism; the PR was a 
cleanup and does not itself demonstrate a reproduction of this incident.
   
   The relevant operations remain in **4.1.0, 4.1.5-release, and current 
branch-4.1** (checked at `e1039662cb13`). Therefore I cannot identify a 
verified fixed **4.1 release**, and lack of reproduction on 4.1.5 does not 
demonstrate a fix. Maintainers should assess a focused backport of the metadata 
removal and its dependencies, with a regression test, rather than assume a 
version upgrade resolves this case. A search of the inspected 4.1.0 FE 
production sources found `getTblRefIds()` callers only propagating these lists, 
which further supports reviewing whether the metadata can be removed.
   
   **3. Evidence needed to confirm the trigger**
   
   Please provide sanitized material preserving the CTE reference graph and 
join structure:
   
   1. SQL plus table/view DDL, exact FE build/commit, JDK version, and relevant 
session settings, particularly `enable_cte_materialize` and 
`inline_cte_referenced_threshold`. State whether the same SQL, settings, 
schema, and statistics were tested on 4.1.5.
   2. For the largest MAT arrays, the precise field path, ideally 
`PlanNode.tblRefIds -> ArrayList.elementData`, the list's **size versus 
backing-array length**, and a bounded sample/frequency summary of the contained 
tuple IDs. Retained heap and array capacity alone do not identify the list's 
logical contents.
   3. Node IDs/types and child relationships for the largest exchanges/joins, 
correlated with CTE producers and consumers. On an isolated diagnostic run, 
record only list sizes and output tuple IDs at exchange construction/update and 
before join concatenation; do not stringify enormous lists. Locate the first 
unexpectedly large list and determine whether its growth follows the recurrence 
above.
   4. The query ID, timestamp-aligned FE/audit/GC logs, full planning-thread 
stack, and concurrent planning count. MAT paths to GC roots can distinguish 
objects retained by the active planning thread from objects retained after 
planning. The screenshots show an active translator thread, so they do not by 
themselves prove a persistent leak. As noted in the report, retained-heap 
totals may overlap.
   
   **4. Interim containment and maintainer next steps**
   
   - Prevent automatic resubmission of the offending SQL. A SQL text/hash block 
rule can reject a known query before planning: in [4.1.0 
StmtExecutor](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L745-L753),
 the regex/hash check precedes `planner.plan()`. Validate rule matching and 
scope with the actual submitted SQL.
   - Test splitting the dependency chain into separate statements that persist 
intermediate results to tables, then scan those tables. This cuts planner 
metadata ancestry at statement boundaries. Merely adding another CTE does not 
provide that boundary. Validate result semantics and intermediate-data cost.
   - Do not treat disabling CTE materialization as a generally safe workaround: 
[CTEInline](https://github.com/apache/doris/blob/5960d4cea0e6d2a08f62b45a07a662395d2bbcfc/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInline.java#L90-L119)
 deep-copies the producer for each inlined consumer, potentially expanding the 
actual plan instead. Test any such change only with the reproducer on an 
isolated FE. `EXPLAIN` can also reach translation and should be tested there 
first.
   - The inspected exchange/join copy operations have no per-query 
allocation-budget check. Lower BE execution-memory limits, optimizer timeouts, 
or a larger FE heap should not be presented as a guaranteed guard against these 
Java array allocations. A controlled planning-memory/metadata limit would need 
explicit implementation and tests before allocation.
   
   Suggested triage: **FE / Nereids planning memory, with FE availability 
impact; reproducer needed**. Next, confirm the CTE graph and duplicate-ID 
growth, reduce it to a small regression case, and evaluate the upstream removal 
on the affected branch. No product code was modified, and no SQL reproduction 
or end-to-end fix validation has been performed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to