[
https://issues.apache.org/jira/browse/IMPALA-13940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18070985#comment-18070985
]
Michael Smith commented on IMPALA-13940:
----------------------------------------
Profiles seem to use a surprisingly large amount of memory. I see 275 MB
Untracked Memory added each run; downloading the thrift profile was 409 KB,
text was 5 MB, and JSON was 25 MB.
> Excessive memory use when running targeted-perf's primitive_many_fragments
> --------------------------------------------------------------------------
>
> Key: IMPALA-13940
> URL: https://issues.apache.org/jira/browse/IMPALA-13940
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Blocker
> Attachments: primitive_many_fragments_100.sql
>
>
> When running targeted-perf's primitive_many_fragments test case in a loop in
> a single impala-shell session, I see memory continually increase until it
> hits the memory limit:
> {noformat}
> Query state can be monitored at:
> http://joemcdonnell-22743:25000/query_plan?query_id=514ae852aad5406e:aa67c15900000000
> 2025-04-07 17:14:59 [Exception] ERROR: Query
> 514ae852aad5406e:aa67c15900000000 failed:
> Failed to get minimum memory reservation of 1.11 GB on daemon
> joemcdonnell-22743:27000 for query 514ae852aad5406e:aa67c15900000000 due to
> following error: Memory limit exceeded: Could not allocate memory while
> trying to increase reservation.
> Query(514ae852aad5406e:aa67c15900000000) could not allocate 1.11 GB without
> exceeding limit.
> Error occurred on backend joemcdonnell-22743:27000
> Memory left in process limit: 2.05 GB
> Query(514ae852aad5406e:aa67c15900000000): Reservation=0 ReservationLimit=9.60
> GB OtherMemory=0 Total=0 Peak=0
> Memory is likely oversubscribed. Reducing query concurrency or configuring
> admission control may help avoid this error.
> {noformat}
> [https://github.com/apache/impala/blob/master/testdata/workloads/targeted-perf/queries/primitive_many_fragments.test]
> Untracked Memory: Total=9.05 GB
> Oddly enough, it doesn't fail if I run each query in a separate impala-shell
> invocation. I will attach a repro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
