Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/10928 )
Change subject: IMPALA-7811: optionally count JVM heap towards process mem limit
......................................................................
IMPALA-7811: optionally count JVM heap towards process mem limit
Adds a flag --mem_limit_includes_jvm that alters memory accounting to
include the amount of memory we think that the JVM is likely to use.
By default this flag is false, so behaviour is unchanged.
We're not ready to change the default but I want to check this in to
enable experimentation.
Two metrics are counted towards the process limit:
* The maximum JVM heap size. We count this because the JVM memory
consumption can expand up to this threshold at any time.
* JVM non-heap committed memory. This can be a non-trivial amount of
memory (e.g. I saw 150MB on one production cluster). There isn't a
hard upper bound on this memory that I know of but should not
grow rapidly.
This requires adjustments in a couple of other places:
* Admission control previous assumed that all of the process memory
limit was available to queries (an assumption that is not strictly
true because of untracked memory, etc, but close enough). However,
the JVM heap makes a large part of the process limit unusable to
queries, so we should only admit up to "process limit - max JVM heap
size" per node.
* The buffer pool is now a percentage of the remaining process limit
after the JVM heap, instead of the total process limit.
Currently, end-to-end tests fail if run with this flag for two reasons:
* The default JVM heap size is 1/4 of physical memory, which means that
essentially all of the process memory limit is consumed by the JVM
heaps when we running 3 impala daemons per host, unless -Xmx is
explicitly set.
* If the heap size is limited to 1-2GB like below, then most tests pass
but TestInsert.test_insert_large_string fails because IMPALA-4865
lets it create giant strings that eat up all the JVM heap.
start-impala-cluster.py \
--impalad_args=--mem_limit_includes_jvm=true --jvm_args="-Xmx1g"
Testing:
Add a custom cluster test that uses the new option and validates the
the memory consumption values.
Change-Id: I39dd715882a32fc986755d573bd46f0fd9eefbfc
Reviewed-on: http://gerrit.cloudera.org:8080/10928
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M common/thrift/StatestoreService.thrift
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_admission_controller.py
A tests/custom_cluster/test_jvm_mem_tracking.py
M www/backends.tmpl
14 files changed, 201 insertions(+), 58 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/10928
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I39dd715882a32fc986755d573bd46f0fd9eefbfc
Gerrit-Change-Number: 10928
Gerrit-PatchSet: 16
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <pooja.nilange...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
