[
https://issues.apache.org/jira/browse/IMPALA-14622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044587#comment-18044587
]
Quanlong Huang commented on IMPALA-14622:
-----------------------------------------
[~zwhtx] [~weizisheng] Thanks for the experiments!
It seems we are cloning too many Exprs. In a heap dump of a 16GB heap space,
there are lots of SlotRef and different subclasses of Expr:
{noformat}
Class Name | Objects | Shallow
Heap | Retained Heap
-----------------------------------------------------------------------------------------------------
java.lang.Thread | 13 |
1,560 | 13,347,108,648
java.util.ArrayList | 108,622,533 |
2,606,940,792 | 12,094,365,720
java.lang.Object[] | 64,036,078 |
2,482,585,920 | 11,890,923,176
org.apache.impala.analysis.InlineViewRef | 98,044 |
14,902,688 | 11,334,385,680
org.apache.impala.analysis.SelectStmt | 117,695 |
14,123,400 | 11,174,590,352
org.apache.impala.analysis.CaseExpr | 6,176,132 |
444,681,504 | 10,928,766,928
org.apache.impala.analysis.BinaryPredicate | 6,571,283 |
630,843,168 | 10,491,219,608
org.apache.impala.analysis.FunctionCallExpr | 8,326,931 |
732,769,928 | 8,943,405,792
org.apache.impala.analysis.ExprSubstitutionMap | 611,668 |
14,680,032 | 8,598,724,296
org.apache.impala.analysis.Subquery | 19,620 |
1,412,640 | 8,403,545,480
org.apache.impala.analysis.FromClause | 117,693 |
2,824,632 | 8,325,121,368
org.apache.impala.analysis.CastExpr | 9,887,834 |
791,026,720 | 7,099,646,832
org.apache.impala.analysis.ArithmeticExpr | 4,944,014 |
355,969,008 | 6,749,317,736
org.apache.impala.analysis.MultiAggregateInfo | 58,878 |
3,768,192 | 5,701,383,032
org.apache.impala.analysis.SlotRef | 33,443,450 |
2,675,476,000 | 5,497,622,304
org.apache.impala.analysis.UnionStmt | 2 |
224 | 4,939,017,272
org.apache.impala.analysis.SetOperationStmt$SetOperand| 24 |
768 | 4,939,008,984
org.apache.impala.catalog.View | 8 |
1,408 | 4,770,676,808
org.apache.impala.analysis.AggregateInfo | 117,756 |
6,594,336 | 3,481,888,784
org.apache.impala.analysis.SelectList | 117,694 |
2,824,656 | 2,548,232,584
org.apache.impala.analysis.SelectListItem | 2,016,690 |
64,534,080 | 2,528,612,952
java.util.LinkedHashMap | 137,315 |
7,689,640 | 1,874,526,240
org.apache.impala.analysis.Analyzer$GlobalState | 2 |
432 | 1,857,534,472
java.util.LinkedHashMap$Entry | 1,644,145 |
65,765,800 | 1,851,097,368
java.util.HashMap | 830,849 |
39,880,752 | 696,429,656
java.util.HashMap$Node[] | 832,801 |
151,697,232 | 674,016,680
org.apache.impala.analysis.NumericLiteral | 6,438,456 |
463,568,832 | 618,112,344
java.util.HashMap$Node | 14,912,597 |
477,203,104 | 522,326,904
org.apache.impala.analysis.AnalyticExpr | 762,177 |
67,071,576 | 492,416,864
org.apache.impala.analysis.Path | 1,633,789 |
104,562,496 | 455,907,176
org.apache.impala.analysis.SlotDescriptor | 980,629 |
70,605,288 | 418,395,912{noformat}
The stacktrace when the planner thread hits OOM (it's still cloning some Exprs):
{noformat}
Thread 0x3ce632a90
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at java.util.ArrayList.<init>(I)V (ArrayList.java:153)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1267)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.CaseExpr.<init>(Lorg/apache/impala/analysis/CaseExpr;)V
(CaseExpr.java:170)
at
org.apache.impala.analysis.CaseExpr.clone()Lorg/apache/impala/analysis/Expr;
(CaseExpr.java:463)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.FunctionCallExpr.<init>(Lorg/apache/impala/analysis/FunctionCallExpr;)V
(FunctionCallExpr.java:233)
at
org.apache.impala.analysis.FunctionCallExpr.clone()Lorg/apache/impala/analysis/Expr;
(FunctionCallExpr.java:888)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at
org.apache.impala.analysis.ExprSubstitutionMap.clone()Lorg/apache/impala/analysis/ExprSubstitutionMap;
(ExprSubstitutionMap.java:209)
at
org.apache.impala.analysis.QueryStmt.<init>(Lorg/apache/impala/analysis/QueryStmt;)V
(QueryStmt.java:532)
at
org.apache.impala.analysis.SelectStmt.<init>(Lorg/apache/impala/analysis/SelectStmt;)V
(SelectStmt.java:1717)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/SelectStmt;
(SelectStmt.java:1814)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/QueryStmt;
(SelectStmt.java:71)
at
org.apache.impala.analysis.InlineViewRef.<init>(Lorg/apache/impala/analysis/InlineViewRef;)V
(InlineViewRef.java:135)
at
org.apache.impala.analysis.InlineViewRef.clone()Lorg/apache/impala/analysis/TableRef;
(InlineViewRef.java:600)
at
org.apache.impala.analysis.FromClause.clone()Lorg/apache/impala/analysis/FromClause;
(FromClause.java:189)
at
org.apache.impala.analysis.SelectStmt.<init>(Lorg/apache/impala/analysis/SelectStmt;)V
(SelectStmt.java:1719)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/SelectStmt;
(SelectStmt.java:1814)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/QueryStmt;
(SelectStmt.java:71)
at
org.apache.impala.analysis.Subquery.<init>(Lorg/apache/impala/analysis/Subquery;)V
(Subquery.java:65)
at
org.apache.impala.analysis.Subquery.clone()Lorg/apache/impala/analysis/Subquery;
(Subquery.java:170)
at
org.apache.impala.analysis.Subquery.clone()Lorg/apache/impala/analysis/Expr;
(Subquery.java:37)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.Predicate.<init>(Lorg/apache/impala/analysis/Predicate;)V
(Predicate.java:51)
at
org.apache.impala.analysis.BinaryPredicate.<init>(Lorg/apache/impala/analysis/BinaryPredicate;)V
(BinaryPredicate.java:159)
at
org.apache.impala.analysis.BinaryPredicate.clone()Lorg/apache/impala/analysis/Expr;
(BinaryPredicate.java:432)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.CaseExpr.<init>(Lorg/apache/impala/analysis/CaseExpr;)V
(CaseExpr.java:170)
at
org.apache.impala.analysis.CaseExpr.clone()Lorg/apache/impala/analysis/Expr;
(CaseExpr.java:463)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at
org.apache.impala.analysis.ExprSubstitutionMap.clone()Lorg/apache/impala/analysis/ExprSubstitutionMap;
(ExprSubstitutionMap.java:209)
at
org.apache.impala.analysis.SelectStmt.<init>(Lorg/apache/impala/analysis/SelectStmt;)V
(SelectStmt.java:1730)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/SelectStmt;
(SelectStmt.java:1814)
at
org.apache.impala.analysis.SelectStmt.clone()Lorg/apache/impala/analysis/QueryStmt;
(SelectStmt.java:71)
at
org.apache.impala.analysis.Subquery.<init>(Lorg/apache/impala/analysis/Subquery;)V
(Subquery.java:65)
at
org.apache.impala.analysis.Subquery.clone()Lorg/apache/impala/analysis/Subquery;
(Subquery.java:170)
at
org.apache.impala.analysis.Subquery.clone()Lorg/apache/impala/analysis/Expr;
(Subquery.java:37)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.CastExpr.<init>(Lorg/apache/impala/analysis/CastExpr;)V
(CastExpr.java:126)
at
org.apache.impala.analysis.CastExpr.clone()Lorg/apache/impala/analysis/Expr;
(CastExpr.java:474)
at
org.apache.impala.analysis.Expr.cloneList(Ljava/util/List;)Ljava/util/List;
(Expr.java:1269)
at org.apache.impala.analysis.Expr.<init>(Lorg/apache/impala/analysis/Expr;)V
(Expr.java:462)
at
org.apache.impala.analysis.ArithmeticExpr.<init>(Lorg/apache/impala/analysis/ArithmeticExpr;)V
(ArithmeticExpr.java:100)
at
org.apache.impala.analysis.ArithmeticExpr.clone()Lorg/apache/impala/analysis/Expr;
(ArithmeticExpr.java:296)
at
org.apache.impala.analysis.Expr.trySubstitute(Lorg/apache/impala/analysis/ExprSubstitutionMap;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/analysis/Expr;
(Expr.java:1140)
at
org.apache.impala.analysis.Expr.trySubstituteList(Ljava/lang/Iterable;Lorg/apache/impala/analysis/ExprSubstitutionMap;Lorg/apache/impala/analysis/Analyzer;Z)Ljava/util/List;
(Expr.java:1180)
at
org.apache.impala.analysis.Expr.substituteList(Ljava/lang/Iterable;Lorg/apache/impala/analysis/ExprSubstitutionMap;Lorg/apache/impala/analysis/Analyzer;Z)Ljava/util/List;
(Expr.java:1188)
at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.buildResultExprs()V
(SelectStmt.java:1237)
at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze()V
(SelectStmt.java:359)
at
org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(Lorg/apache/impala/analysis/SelectStmt$SelectAnalyzer;)V
(SelectStmt.java:285)
at
org.apache.impala.analysis.SelectStmt.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(SelectStmt.java:277)
at
org.apache.impala.analysis.SetOperationStmt$SetOperand.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(SetOperationStmt.java:105)
at
org.apache.impala.analysis.SetOperationStmt.analyzeOperands(Lorg/apache/impala/analysis/Analyzer;)V
(SetOperationStmt.java:412)
at
org.apache.impala.analysis.SetOperationStmt.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(SetOperationStmt.java:330)
at
org.apache.impala.analysis.UnionStmt.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(UnionStmt.java:49)
at
org.apache.impala.analysis.WithClause.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(WithClause.java:83)
at
org.apache.impala.analysis.QueryStmt.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(QueryStmt.java:165)
at
org.apache.impala.analysis.SelectStmt.analyze(Lorg/apache/impala/analysis/Analyzer;)V
(SelectStmt.java:276)
at
org.apache.impala.analysis.AnalysisContext$AnalysisDriverImpl.analyze()Lorg/apache/impala/analysis/AnalysisContext$AnalysisResult;
(AnalysisContext.java:565)
at
org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(Lorg/apache/impala/service/CompilerFactory;Lorg/apache/impala/analysis/ParsedStatement;Lorg/apache/impala/analysis/StmtMetadataLoader$StmtTableCache;Lorg/apache/impala/authorization/AuthorizationChecker;Z)Lorg/apache/impala/analysis/AnalysisContext$AnalysisResult;
(AnalysisContext.java:496)
at
org.apache.impala.service.Frontend.doCreateExecRequest(Lorg/apache/impala/service/CompilerFactory;Lorg/apache/impala/service/Frontend$PlanCtx;Ljava/util/List;Lorg/apache/impala/util/EventSequence;)Lorg/apache/impala/thrift/TExecRequest;
(Frontend.java:2989)
at
org.apache.impala.service.Frontend.getTExecRequest(Lorg/apache/impala/service/CompilerFactory;Lorg/apache/impala/service/Frontend$PlanCtx;Lorg/apache/impala/util/EventSequence;)Lorg/apache/impala/thrift/TExecRequest;
(Frontend.java:2553)
at
org.apache.impala.service.Frontend.getTExecRequestWithFallback(Lorg/apache/impala/service/Frontend$PlanCtx;Lorg/apache/impala/util/EventSequence;)Lorg/apache/impala/thrift/TExecRequest;
(Frontend.java:2419)
at
org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/service/Frontend$PlanCtx;)Lorg/apache/impala/thrift/TExecRequest;
(Frontend.java:2106)
at org.apache.impala.service.JniFrontend.createExecRequest([B)[B
(JniFrontend.java:175){noformat}
> Multi sub-query and union all in one query cause coordinator jvm oom
> --------------------------------------------------------------------
>
> Key: IMPALA-14622
> URL: https://issues.apache.org/jira/browse/IMPALA-14622
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.3.0
> Reporter: Weizisheng
> Priority: Major
> Attachments: demo.sql, image-2025-12-11-15-19-04-753.png
>
>
> With demo.sql attached, coordinator will run out of heap memory of the
> coordinator process.
> The following log brushing in impalad.INFO
> {code:java}
> I1209 11:00:20.028505 2072263 MetastoreShim.java:655]
> e449e840948f713e:fb6ff8ba00000000] Table
> demo_table_1210,#bucket=-1,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=null{code}
> And eventually, coordinator aborted.
> {code:java}
> I1209 11:04:38.965866 3295030 JvmPauseMonitor.java:209] Detected pause in JVM
> or host machine (eg GC): pause of approximately 5679ms
> GC pool 'G1 Young Generation' had collection(s): count=1 time=6045ms
> I1209 11:05:33.816803 4099651 impala-server.cc:1363] Found local timezone
> "Asia/Shanghai".
> I1209 11:05:47.246533 3295030 JvmPauseMonitor.java:209] Detected pause in JVM
> or host machine (eg GC): pause of approximately 5468ms
> GC pool 'G1 Young Generation' had collection(s): count=4 time=14537ms
> GC pool 'G1 Old Generation' had collection(s): count=1 time=53666ms
> OOM caused by java heap space occurred, allocate size: 376 bytes, type:
> Ljava/lang/Thread;
> garbage-first heap total 31424512K, used 31369210K [0x0000000082000000,
> 0x0000000800000000)
> region size 16384K, 0 young (0K), 0 survivors (0K)
> Metaspace used 72157K, capacity 72727K, committed 73908K, reserved
> 1116160K
> class space used 6947K, capacity 7159K, committed 7296K, reserved
> 1048576K
> current stack trace:
> F1209 11:33:31.436828 5235 jni-util.cc:241]
> fc45f25066e555ad:4b2c8b5c00000000] Check failed: rc == 0 (-2 vs. 0) Unable to
> get JVM {code}
> It happens for coordinator both with -Xmx8G and -Xmx32G.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
