1186683677 opened a new issue, #70679:
URL: https://github.com/apache/doris/issues/70679

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Version
   
   Apache Doris 4.1.0
   Component: Frontend (FE), Nereids Planner
   FE JVM heap size: 16 GB
   Garbage Collector: G1GC
   
   ### What's Wrong?
   
   We encountered a Java OutOfMemoryError in the Doris FE while planning a 
complex SQL query involving multiple CTEs and JOINs.
   
   The OOM occurred during Nereids physical plan translation, specifically when 
the HashJoinNode constructor attempted to merge the tblRefIds lists from its 
child nodes.
   
   **OOM stack trace:**
   java.lang.OutOfMemoryError: Java heap space
       at java.util.Arrays.copyOf
       at java.util.ArrayList.grow
       at java.util.ArrayList.addAll
       at org.apache.doris.planner.HashJoinNode.<init>(HashJoinNode.java:119)
       at 
org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:1751)
       at 
org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:2253)
       at 
org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalCTEProducer(PhysicalPlanTranslator.java:1538)
       at 
org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalCTEAnchor(PhysicalPlanTranslator.java:1492)
       at 
org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:594)
   
   **Eclipse MAT heap dump analysis:**
   
   - 51 ExchangeNode instances retained approximately 2.01 GB of heap memory.
   - 25 HashJoinNode instances retained approximately 1.34 GB.
   - Extremely large Object[] arrays were found, including arrays with 
approximately 167 million and 251 million elements.
   - The OOM occurred while expanding an ArrayList during HashJoinNode 
construction.
   
   These retained heap values may overlap and should not be added together.
   
   The relevant code in HashJoinNode.java is:
       tblRefIds.addAll(outer.getTblRefIds());
       tblRefIds.addAll(inner.getTblRefIds());
   
   It appears that tblRefIds may have grown to an unexpectedly large size 
during plan translation.
   However, we have not yet identified the exact operation responsible for this 
growth.
   
   **FE JVM OOM Log (fe.out)**
   java.lang.OutOfMemoryError: Java heap space
   Dumping heap to /app/doris/fe/log/java_pid3369.hprof ...
   Heap dump file created [15385255901 bytes in 103.946 secs]
    java.lang.OutOfMemoryError: Java heap space
   -XX:OnOutOfMemoryError="kill -9 %p"
   Executing /bin/sh -c "kill -9 3369"...
   
   The FE encountered a Java heap OutOfMemoryError.
   
   The JVM successfully generated a heap dump (approximately 14.33 GiB) and 
then executed the configured OnOutOfMemoryError command to terminate the FE 
process.
   
   The heap dump was analyzed using Eclipse MAT. The analysis identified 
unusually large arrays associated with ExchangeNode and HashJoinNode during 
Nereids physical plan translation.
   
   ### What You Expected?
   
   The Doris FE should be able to handle complex SQL query planning without 
excessive memory amplification in internal plan node metadata.
   
   If a query generates an excessively large execution plan, the FE should 
ideally terminate the query with a controlled error rather than exhausting the 
JVM heap and potentially affecting FE availability.
   
   ### How to Reproduce?
   
   The issue occurred in our production environment when executing a complex 
SQL query containing multiple CTEs and JOINs.
   
   Observed sequence:
   
   1. Submit a complex SQL query involving multiple CTEs and JOINs to Doris 
4.1.0.
   2. The Nereids planner starts translating the physical plan into 
PlanFragment objects.
   3. Multiple ExchangeNode and HashJoinNode instances are created.
   4. During HashJoinNode construction, the planner attempts to merge large 
tblRefIds lists.
   5. ArrayList.addAll() triggers an array expansion, resulting in 
OutOfMemoryError.
   
   We have not yet prepared a minimal reproducible SQL example.
   
   ### Anything Else?
   
   We investigated the relevant source code in Doris 4.1.0 and 4.1.5.
   The HashJoinNode constructor contains the same tblRefIds.addAll() operations 
in both versions.
   We also noticed the following implementation in 
ExchangeNode.updateTupleIds():
   
   if (outputTupleDesc != null) {
       tupleIds.clear();
       tupleIds.add(outputTupleDesc.getId());
       tblRefIds.add(outputTupleDesc.getId());
       nullableTupleIds.add(outputTupleDesc.getId());
   }
   
   In this branch, tblRefIds is appended to without being cleared. We are not 
sure whether this contributes to the observed memory growth.
   
   <img width="1486" height="903" alt="MAT_Leak_Suspects1" 
src="https://github.com/user-attachments/assets/a5ec2ae7-3bbe-4d57-b945-bf93f7cda30c";
 />
   <img width="505" height="640" alt="MAT_Leak_Suspects2" 
src="https://github.com/user-attachments/assets/7d6efe78-60bd-42d7-acee-2804f33e5739";
 />
   
   We would appreciate guidance from the community on the following questions:
   
   1. Could repeated copying or propagation of tblRefIds between plan nodes 
cause excessive memory amplification?
   2. Is the behavior of ExchangeNode.updateTupleIds() expected?
   3. Is this a known issue that has already been fixed in a newer Doris 
release?
   4. Are there any recommended workarounds or safeguards to prevent a single 
complex query from exhausting the FE heap?
   
   **Additional notes:**
   We have not reproduced the issue on Doris 4.1.5.
   The exact root cause has not yet been confirmed.
   Eclipse MAT analysis results and sanitized stack traces can be provided for 
further investigation.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to