[
https://issues.apache.org/jira/browse/IMPALA-11842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Rawat updated IMPALA-11842:
------------------------------------
Priority: Critical (was: Major)
> Improve memory estimation for streaming aggregate operator
> ----------------------------------------------------------
>
> Key: IMPALA-11842
> URL: https://issues.apache.org/jira/browse/IMPALA-11842
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Priority: Critical
>
> Streaming aggregate operator can over estimate the peak memory and as a
> result Impala could request max memory allowed by admission controller. This
> impacts query concurrency and also causes unnecessary scaling.
> Looking at some cases in the following profile snippet, the estimated peak
> memory (8.15 GB) is *20X* from actual peak memory (416.07 MB).
> {code:java}
> Estimated Per-Host Mem: 247067273376
> Request Pool: root.default
> Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB)
> impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB)
> impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB)
> impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB)
> coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00
> MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34
> GB) impala-executor-003-7:27010(1.35 GB)
> Per Host Number of Fragment Instances: impala-executor-003-5:27010(37)
> impala-executor-003-4:27010(38) impala-executor-003-6:27010(37)
> impala-executor-003-0:27010(38) impala-executor-003-8:27010(38)
> impala-executor-003-2:27010(37) impala-executor-003-1:27010(38)
> coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1)
> impala-executor-003-3:27010(37) impala-executor-003-9:27010(37)
> impala-executor-003-7:27010(38)
> Latest admission queue reason: Not enough memory available on host
> impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06
> GB was available.
> Admission result: Admitted (queued)
> Initial admission queue reason: waited 83020 ms, reason: Not enough
> memory available on host impala-executor-003-5:27010. Needed 50.00 GB but
> only 33.06 GB out of 83.06 GB was available.
> Cluster Memory Admitted: 500.10 GB
> Executor Group: root.default-group-002
> ExecSummary:
> Operator #Hosts #Inst Avg Time Max Time #Rows Est.
> #Rows Peak Mem Est. Peak Mem Detail
> -----------------------------------------------------------------------------------------------------------------------------------------
> F04:ROOT 1 1 146.192us 146.192us
> 4.01 MB 4.00 MB
> 11:MERGING-EXCHANGE 1 1 3.297ms 3.297ms 100
> 100 1.88 MB 234.53 KB UNPARTITIONED
> F03:EXCHANGE SENDER 10 120 4.196ms 374.168ms
> 7.52 KB 0
> 05:TOP-N 10 120 22.184ms 1s028ms 12.00K
> 100 16.00 KB 1.56 KB
> 10:AGGREGATE 10 120 4m2s 12m14s 499.98K
> 487.66K 2.34 MB 10.00 MB FINALIZE
> 09:EXCHANGE 10 120 5s336ms 7s799ms 22.11B
> 487.66K 10.40 MB 3.09 MB HASH(vendor_id)
> F02:EXCHANGE SENDER 10 120 28s199ms 48s974ms
> 4.16 MB 0
> 04:AGGREGATE 10 120 1m17s 1m34s 22.11B
> 487.66K 21.02 MB 10.00 MB STREAMING
> 08:AGGREGATE 10 120 12m29s 22m36s 50.00B
> 50.00B 3.85 GB 10.87 GB
> 07:EXCHANGE 10 120 10s165ms 12s246ms 50.00B
> 50.00B 11.69 MB 12.34 MB HASH(vendor_id,purchase_id)
> F00:EXCHANGE SENDER 10 120 1m28s 1m49s
> 4.16 MB 0
> 03:AGGREGATE 10 120 2m34s 5m5s 50.00B
> 50.00B 416.07 MB 8.15 GB STREAMING
> 02:HASH JOIN 10 120 1m17s 1m40s 50.00B
> 50.00B 46.12 KB 0 INNER JOIN, BROADCAST
> |--F05:JOIN BUILD 10 10 557.800ms 613.307ms
> 408.02 MB 408.00 MB
> | 06:EXCHANGE 10 10 88.770ms 102.048ms 5.00M
> 5.00M 16.11 MB 10.10 MB BROADCAST
> | F01:EXCHANGE SENDER 5 5 190.673ms 212.904ms
> 75.23 KB 0
> | 00:SCAN HDFS 5 5 774.428ms 965.865ms 5.00M
> 5.00M 12.92 MB 64.00 MB tab.product
> 01:SCAN HDFS 10 120 5m48s 13m14s 50.00B
> 50.00B 32.92 MB 88.00 MB tab.pli {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
