[
https://issues.apache.org/jira/browse/IMPALA-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796952#comment-17796952
]
Michael Smith commented on IMPALA-12635:
----------------------------------------
That seems reasonable. We can revert [https://gerrit.cloudera.org/#/c/20697/]
for now; I probably mislabeled it since it has larger consequences than codegen
cache size. Tracking LLVM bytes allocated was added in IMPALA-4397/IMPALA-3259
and has been under-counting it since then. I'm not clear how we account for
codegen memory in planning (I'm not sure we do), it likely needs to be updated
with that change.
> Tests fail with "Memory is likely oversubscribed"
> -------------------------------------------------
>
> Key: IMPALA-12635
> URL: https://issues.apache.org/jira/browse/IMPALA-12635
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 1.4.2
> Reporter: Andrew Sherman
> Assignee: Wenzhe Zhou
> Priority: Critical
>
> 3 different tests failed trying to allocate memory
> {code}
> E OperationalError: Failed to get minimum memory reservation of 94.11 MB on
> daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for
> query a94be0ce8a16884d:cef55b1a00000000 due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(a94be0ce8a16884d:cef55b1a00000000) could not allocate 94.11 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.24 GB
> E Query(a94be0ce8a16884d:cef55b1a00000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> {code}
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Failed to get minimum memory reservation of 4.03 MB on
> daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000 for
> query 3e4b9c2bd24a8c30:5db8f3e100000000 due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(3e4b9c2bd24a8c30:5db8f3e100000000) could not allocate 4.03 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.20 GB
> E Query(3e4b9c2bd24a8c30:5db8f3e100000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> {code}
> E OperationalError: Failed to get minimum memory reservation of 284.18 MB
> on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> for query 054cab730f0605ca:6e70e3dd00000000 due to following error: Memory
> limit exceeded: Could not allocate memory while trying to increase
> reservation.
> E Query(054cab730f0605ca:6e70e3dd00000000) could not allocate 284.18 MB
> without exceeding limit.
> E Error occurred on backend
> impala-ec2-centos79-m6i-4xlarge-xldisk-0682.vpc.cloudera.com:27000
> E Memory left in process limit: 1.34 GB
> E Query(054cab730f0605ca:6e70e3dd00000000): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
> [~wzhou] I'm randomly picking on you to own this. This has only happened once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
