Qifan Chen has uploaded a new patch set (#9). (
http://gerrit.cloudera.org:8080/16220 )
Change subject: IMPALA-9989 Improve admission control pool stats logging
......................................................................
IMPALA-9989 Improve admission control pool stats logging
This work addresses the current limitation in admission controller by
appending the last known memory consumption statistics about a pool or
a host to the existing host memory exhaustion message. The message is
logged in impalad.INFO when a query is queued or timed out due to
memory pressure on the pool or on the host.
This new memory consumption statistics covers the following content:
topN_query_stats::=
queries: a list of query Ids for up to 5 queries with top memory
consumptions
total_mem_consumed: total memory consumed by these topN queries
percentage_mem_consumed_per_pool: total memory consumed divided
by pool memory usage (if
feasible to report)
all_query_stats::=
min: the minimal memory consumption of all running queries
max: the maximal memory consumption of all running queries
total: the total memory consumption of all running queries
average: the average memory consumption of all running queries
(if feasible to report)
pool_stats_per_host::=
<pool_name>: <topN_query_stats> <all_query_stats>
pool_stats::=
List of <pool_stats_per_host>
host_stats_per_pool::=
<host_name>: <topN_query_stats>
host_stats::=
List of <host_stats_per_pool>
pool_stats will be logged when the memory demand exceeds the host
memory limit. host_stats will be logged when the memory demand exceeds
the pool memory limit.
Testing:
1. Added one new test TopNQueryCheck in admission-controller-test.cc to
simulate queries running in 4 pools in 3 hosts. This new test will test
the following:
a. Identifying top 5 queries among 4 pools in local host;
b. Identifying top 5 queries among 3 hosts for a pool.
Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
---
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/util/container-util.h
M common/thrift/StatestoreService.thrift
7 files changed, 751 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/9
--
To view, visit http://gerrit.cloudera.org:8080/16220
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
Gerrit-Change-Number: 16220
Gerrit-PatchSet: 9
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Sahil Takiar <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
