Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/23542 )
Change subject: IMPALA-14493: Cap memory usage of global admission service ...................................................................... IMPALA-14493: Cap memory usage of global admission service The global admission service can experience OOM errors under high concurrency because its process memory tracker is inaccurate and doesn't account for all memory allocations. Ensuring memory tracker accurately accounts for every allocation could be difficult, this patch uses a simpler solution to introduce a hard memory cap using tcmalloc statistics, which accurately reflect the true process memory usage. If a new query is submitted while tcmalloc memory usage is over the process limit, the query will be rejected immediately to protect from OOM. Adds a new flag enable_admission_service_mem_safeguard allowing this feature to be enabled or disabled. By default, this feature is turned on Tests: Added test test_admission_service_low_mem_limit. Passed exhaustive tests. Change-Id: I2ee2c942a73fcd69358851fc2fdc0fc4fe531c73 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M tests/custom_cluster/test_admission_controller.py 4 files changed, 57 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/23542/2 -- To view, visit http://gerrit.cloudera.org:8080/23542 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ee2c942a73fcd69358851fc2fdc0fc4fe531c73 Gerrit-Change-Number: 23542 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
