1186683677 opened a new issue, #70679: URL: https://github.com/apache/doris/issues/70679
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.0 Component: Frontend (FE), Nereids Planner FE JVM heap size: 16 GB Garbage Collector: G1GC ### What's Wrong? We encountered a Java OutOfMemoryError in the Doris FE while planning a complex SQL query involving multiple CTEs and JOINs. The OOM occurred during Nereids physical plan translation, specifically when the HashJoinNode constructor attempted to merge the tblRefIds lists from its child nodes. **OOM stack trace:** java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf at java.util.ArrayList.grow at java.util.ArrayList.addAll at org.apache.doris.planner.HashJoinNode.<init>(HashJoinNode.java:119) at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:1751) at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:2253) at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalCTEProducer(PhysicalPlanTranslator.java:1538) at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalCTEAnchor(PhysicalPlanTranslator.java:1492) at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:594) **Eclipse MAT heap dump analysis:** - 51 ExchangeNode instances retained approximately 2.01 GB of heap memory. - 25 HashJoinNode instances retained approximately 1.34 GB. - Extremely large Object[] arrays were found, including arrays with approximately 167 million and 251 million elements. - The OOM occurred while expanding an ArrayList during HashJoinNode construction. These retained heap values may overlap and should not be added together. The relevant code in HashJoinNode.java is: tblRefIds.addAll(outer.getTblRefIds()); tblRefIds.addAll(inner.getTblRefIds()); It appears that tblRefIds may have grown to an unexpectedly large size during plan translation. However, we have not yet identified the exact operation responsible for this growth. **FE JVM OOM Log (fe.out)** java.lang.OutOfMemoryError: Java heap space Dumping heap to /app/doris/fe/log/java_pid3369.hprof ... Heap dump file created [15385255901 bytes in 103.946 secs] java.lang.OutOfMemoryError: Java heap space -XX:OnOutOfMemoryError="kill -9 %p" Executing /bin/sh -c "kill -9 3369"... The FE encountered a Java heap OutOfMemoryError. The JVM successfully generated a heap dump (approximately 14.33 GiB) and then executed the configured OnOutOfMemoryError command to terminate the FE process. The heap dump was analyzed using Eclipse MAT. The analysis identified unusually large arrays associated with ExchangeNode and HashJoinNode during Nereids physical plan translation. ### What You Expected? The Doris FE should be able to handle complex SQL query planning without excessive memory amplification in internal plan node metadata. If a query generates an excessively large execution plan, the FE should ideally terminate the query with a controlled error rather than exhausting the JVM heap and potentially affecting FE availability. ### How to Reproduce? The issue occurred in our production environment when executing a complex SQL query containing multiple CTEs and JOINs. Observed sequence: 1. Submit a complex SQL query involving multiple CTEs and JOINs to Doris 4.1.0. 2. The Nereids planner starts translating the physical plan into PlanFragment objects. 3. Multiple ExchangeNode and HashJoinNode instances are created. 4. During HashJoinNode construction, the planner attempts to merge large tblRefIds lists. 5. ArrayList.addAll() triggers an array expansion, resulting in OutOfMemoryError. We have not yet prepared a minimal reproducible SQL example. ### Anything Else? We investigated the relevant source code in Doris 4.1.0 and 4.1.5. The HashJoinNode constructor contains the same tblRefIds.addAll() operations in both versions. We also noticed the following implementation in ExchangeNode.updateTupleIds(): if (outputTupleDesc != null) { tupleIds.clear(); tupleIds.add(outputTupleDesc.getId()); tblRefIds.add(outputTupleDesc.getId()); nullableTupleIds.add(outputTupleDesc.getId()); } In this branch, tblRefIds is appended to without being cleared. We are not sure whether this contributes to the observed memory growth. <img width="1486" height="903" alt="MAT_Leak_Suspects1" src="https://github.com/user-attachments/assets/a5ec2ae7-3bbe-4d57-b945-bf93f7cda30c" /> <img width="505" height="640" alt="MAT_Leak_Suspects2" src="https://github.com/user-attachments/assets/7d6efe78-60bd-42d7-acee-2804f33e5739" /> We would appreciate guidance from the community on the following questions: 1. Could repeated copying or propagation of tblRefIds between plan nodes cause excessive memory amplification? 2. Is the behavior of ExchangeNode.updateTupleIds() expected? 3. Is this a known issue that has already been fixed in a newer Doris release? 4. Are there any recommended workarounds or safeguards to prevent a single complex query from exhausting the FE heap? **Additional notes:** We have not reproduced the issue on Doris 4.1.5. The exact root cause has not yet been confirmed. Eclipse MAT analysis results and sanitized stack traces can be provided for further investigation. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
