Hello Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11157
to look at the new patch set (#16).
Change subject: IMPALA-7349: Add Admission control support for automatically
setting per host memory limit for a query
......................................................................
IMPALA-7349: Add Admission control support for automatically setting
per host memory limit for a query
With this patch the per host memory limit of a query is automatically
set using the mem_limit set in the query options and the mem_estimate
calculated by the planner based on the following pseudo code:
if mem_limit is set in query options:
use that and if 'clamp-mem-limit-query-option' is true:
enforce the min/max query mem limits defined in the pool config.
else:
mem_limit = max(mem_estiamte,
min_mem_limit_required_to_accomodate_largest_initial_reservation)
finally, enforce min/max query mem limits defined in the pool
config on this value.
This calculated mem limit will also be used for admission accounting
and consequently for admission control. Moreover, three new pool
configuration options have been added to enable this behaviour:
"min-query-mem-limit" & "max-query-mem-limit" => help
clamp the per host memory limit for a query. If both these limits
are not configured, then the estimates from planning are not used
as a memory limit and only used for making admission decisions.
Moreover the estimates will no longer have a lower bound based
on the largest initial reservation.
"clamp-mem-limit-query-option" => if false, the mem_limit defined in
the query options is used directly and the max/min query mem limits
are not enforced on it.
Testing:
Added e2e test cases.
Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6
---
M be/src/benchmarks/process-wide-locks-benchmark.cc
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-exec-mgr.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/test-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M common/thrift/ImpalaInternalService.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/llama-site-test.xml
A fe/src/test/resources/mem-limit-test-fair-scheduler.xml
A fe/src/test/resources/mem-limit-test-llama-site.xml
A
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M
testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
A tests/common/resource_pool_config.py
M tests/custom_cluster/test_admission_controller.py
30 files changed, 1,072 insertions(+), 265 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11157/16
--
To view, visit http://gerrit.cloudera.org:8080/11157
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6
Gerrit-Change-Number: 11157
Gerrit-PatchSet: 16
Gerrit-Owner: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Dan Hecht <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Philip Zeyliger <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
