[
https://issues.apache.org/jira/browse/IMPALA-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796952#comment-17796952
]
Michael Smith edited comment on IMPALA-12635 at 12/14/23 11:56 PM:
-------------------------------------------------------------------
That seems reasonable. We can revert [https://gerrit.cloudera.org/#/c/20697/]
for now; I probably mislabeled it since the cache will stop using it soon and
the memory impacts process memory limit. Tracking LLVM bytes allocated was
added in IMPALA-4397/IMPALA-3259 and has been under-counting it since then. I'm
not clear how we account for codegen memory in planning (I'm not sure we do),
it likely needs to be updated with that change.
was (Author: JIRAUSER288956):
That seems reasonable. We can revert [https://gerrit.cloudera.org/#/c/20697/]
for now; I probably mislabeled it since it has larger consequences than codegen
cache size. Tracking LLVM bytes allocated was added in IMPALA-4397/IMPALA-3259
and has been under-counting it since then. I'm not clear how we account for
codegen memory in planning (I'm not sure we do), it likely needs to be updated
with that change.
> Tests fail with "Memory is likely oversubscribed"
> -------------------------------------------------
>
> Key: IMPALA-12635
> URL: https://issues.apache.org/jira/browse/IMPALA-12635
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 1.4.2
> Reporter: Andrew Sherman
> Assignee: Wenzhe Zhou
> Priority: Critical
>
> 3 different tests failed trying to allocate memory
> {code}
> E OperationalError: Failed to get minimum memory reservation of 94.11 MB on
> daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for
> query a94be0ce8a16884d:cef55b1a00000000 due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(a94be0ce8a16884d:cef55b1a00000000) could not allocate 94.11 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.24 GB
> E Query(a94be0ce8a16884d:cef55b1a00000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> {code}
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Failed to get minimum memory reservation of 4.03 MB on
> daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for
> query 3e4b9c2bd24a8c30:5db8f3e100000000 due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(3e4b9c2bd24a8c30:5db8f3e100000000) could not allocate 4.03 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.20 GB
> E Query(3e4b9c2bd24a8c30:5db8f3e100000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> {code}
> E OperationalError: Failed to get minimum memory reservation of 284.18 MB
> on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> for query 054cab730f0605ca:6e70e3dd00000000 due to following error: Memory
> limit exceeded: Could not allocate memory while trying to increase
> reservation.
> E Query(054cab730f0605ca:6e70e3dd00000000) could not allocate 284.18 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.34 GB
> E Query(054cab730f0605ca:6e70e3dd00000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> [~wzhou] I'm randomly picking on you to own this. This has only happened once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
