[
https://issues.apache.org/jira/browse/IMPALA-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868047#comment-17868047
]
Abhishek Rawat commented on IMPALA-13251:
-----------------------------------------
Do you have "impala.admission-control.clamp-mem-limit-query-option" configured
for the Pool? If this is configured to False then that MEM_LIMIT query option
will override the configured min/max resource pool limits
Some details in the section "Minimum Query Memory Limit and Maximum Query
Memory Limit" :
https://impala.apache.org/docs/build/html/topics/impala_admission.html#admission_memory
> Planner uses session MEM_LIMIT with Max MEM_LIMIT set on Resource Pool
> ----------------------------------------------------------------------
>
> Key: IMPALA-13251
> URL: https://issues.apache.org/jira/browse/IMPALA-13251
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.3.0
> Reporter: Andrey Popkov
> Priority: Minor
> Labels: memlimit, memory-bug, planner, resourcepool
>
> impalad version 4.3.0
> Query is INSERT SELECT FROM WHERE
> with no joins, functions, orders etc
> fails with OOM
> Query MEM_LIMIT is set to 35Gb
> Resource Pool max query MEM_LIMIT range is 20Gb
> Cluster has 14 hosts and Admission Control is enabled
> so we see in query profile:
> Query Options (set by configuration): MEM_LIMIT=37580963840, MT_DOP=8, ..
> Query Options (set by configuration and planner): MEM_LIMIT=37580963840,
> MT_DOP=8, ..
> Cluster Memory Admitted: 280.00 GB
> Max Per-Host Resource Reservation: Memory=128.00MB Threads=8
> Per-Host Resource Estimates: Memory=123.54GB
> Query(x:y): memory limit exceeded. Limit=20.00 GB Reservation=192.25 MB
> ReservationLimit=16.00 GB OtherMemory=19.81 GB Total=20.00 GB Peak=20.00 GB
> The same query with MEM_LIMIT=0 and MT_DOP=0 is successful with:
> Cluster Memory Admitted 280.00 GB
> Max Per-Host Resource Reservation: Memory=16.00MB Threads=2
> Per-Host Resource Estimates: Memory=35.02GB
> The question is why planner uses 35G but not 20G
> And perhaps, it has impact on memory utilization
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
