[
https://issues.apache.org/jira/browse/IMPALA-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753261#comment-16753261
]
ASF subversion and git services commented on IMPALA-7941:
---------------------------------------------------------
Commit 7aec1b6db042d88c04a501e24afaa25d4794cccb in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7aec1b6 ]
IMPALA-7941: part 2/2: use cgroups memory limit
This uses the functionality from part 1 to detect the CGroups memory
limit and use it to set a lower process memory limit if needed.
min(system memory, cgroups memory limit) is used instead of
system memory to determine the memory limit. Behaviour of
processes without a memory limit set via CGroups is unchanged.
The default behaviour of using 80% of the memory limit detected
is still in effect. This seems like an OK default, but may
lead to some amount of wasted memory.
Modify containers to have a default JVM heap size of 2GB and
--mem_limit_includes_jvm, so that the automatically configured
memory limit makes more sense.
start-impala-cluster.py is modified to exercise all of this.
Testing:
Tested a containerised cluster manually on my system, which has
32GB of RAM. Here's the breakdown from the memz/ page showing
the JVM heap and auto-configured memory limit.
Process: Limit=7.31 GB Total=1.94 GB Peak=1.94 GB
JVM: max heap size: Total=1.78 GB
JVM: non-heap committed: Total=35.56 MB
Buffer Pool: Free Buffers: Total=0
Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=0
Control Service Queue: Limit=50.00 MB Total=0 Peak=0
Data Stream Service Queue: Limit=374.27 MB Total=0 Peak=0
Data Stream Manager Early RPCs: Total=0 Peak=0
TCMalloc Overhead: Total=12.20 MB
Untracked Memory: Total=121.31 MB
Change-Id: Ie9fb4fb936a46fc194a204391d03c07c8c7fba21
Reviewed-on: http://gerrit.cloudera.org:8080/12262
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Determine process memory limit from cgroups limit, if set
> ---------------------------------------------------------
>
> Key: IMPALA-7941
> URL: https://issues.apache.org/jira/browse/IMPALA-7941
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: admission-control, resource-management
> Attachments: allocmem.cc
>
>
> We currently take the hardcoded config value, or the system physical memory
> if that is not set. We could be smarter when running in a container by
> sniffing out the cgroups setting.
> Java did something along this lines:
> https://www.opsian.com/blog/java-on-docker/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
