[
https://issues.apache.org/jira/browse/IMPALA-14622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044339#comment-18044339
]
Quanlong Huang commented on IMPALA-14622:
-----------------------------------------
[~weizisheng] Thanks for reporting the issue! The query is quite complex. Are
you able to run it in other engines?
I can reproduce the OOM locally. Note that the new calcite planner is able to
generate the plan but takes around one hour:
{noformat}
Query Compilation: 1h14m
- Metadata of all 1 tables cached: 259.580ms (259.580ms)
- Analysis finished: 485.468ms (225.888ms)
- Authorization finished (noop): 485.686ms (218.014us)
- Starting optimization: 6s272ms (5s786ms)
- Expanded plan: 6s544ms (271.898ms)
- Coerced plan: 7s141ms (597.005ms)
- Created optimized plan pre join: 48m2s (47m54s)
- Created optimized join plan: 56m40s (8m38s)
- Created optimized plan post join: 1h11m (15m8s)
- Created final Impala convert plan: 1h14m (2m39s)
- Single node plan created: 1h14m (2s817ms)
- Runtime filters computed: 1h14m (363.882ms)
- Distributed plan created: 1h14m (32.722ms)
- Planning finished: 1h14m (1s615ms){noformat}
So I'm not sure if this is a normal query that we should support.
Note that there are lots of CTEs in the query. I think there are room for
improvements. CC [~MikaelSmith], [~scarlin], [~rizaon]
> Multi sub-query and union all in one query cause coordinator jvm oom
> --------------------------------------------------------------------
>
> Key: IMPALA-14622
> URL: https://issues.apache.org/jira/browse/IMPALA-14622
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.3.0
> Reporter: Weizisheng
> Priority: Major
> Attachments: demo.sql
>
>
> With demo.sql attached, coordinator will run out of heap memory of the
> coordinator process.
> The following log brushing in impalad.INFO
> {code:java}
> I1209 11:00:20.028505 2072263 MetastoreShim.java:655]
> e449e840948f713e:fb6ff8ba00000000] Table
> demo_table_1210,#bucket=-1,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=null{code}
> And eventually, coordinator aborted.
> {code:java}
> I1209 11:04:38.965866 3295030 JvmPauseMonitor.java:209] Detected pause in JVM
> or host machine (eg GC): pause of approximately 5679ms
> GC pool 'G1 Young Generation' had collection(s): count=1 time=6045ms
> I1209 11:05:33.816803 4099651 impala-server.cc:1363] Found local timezone
> "Asia/Shanghai".
> I1209 11:05:47.246533 3295030 JvmPauseMonitor.java:209] Detected pause in JVM
> or host machine (eg GC): pause of approximately 5468ms
> GC pool 'G1 Young Generation' had collection(s): count=4 time=14537ms
> GC pool 'G1 Old Generation' had collection(s): count=1 time=53666ms
> OOM caused by java heap space occurred, allocate size: 376 bytes, type:
> Ljava/lang/Thread;
> garbage-first heap total 31424512K, used 31369210K [0x0000000082000000,
> 0x0000000800000000)
> region size 16384K, 0 young (0K), 0 survivors (0K)
> Metaspace used 72157K, capacity 72727K, committed 73908K, reserved
> 1116160K
> class space used 6947K, capacity 7159K, committed 7296K, reserved
> 1048576K
> current stack trace:
> F1209 11:33:31.436828 5235 jni-util.cc:241]
> fc45f25066e555ad:4b2c8b5c00000000] Check failed: rc == 0 (-2 vs. 0) Unable to
> get JVM {code}
> It happens for coordinator both with -Xmx8G and -Xmx32G.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
