Qifan Chen has uploaded a new patch set (#26). (
http://gerrit.cloudera.org:8080/16220 )
Change subject: IMPALA-9989 Improve admission control pool stats logging
......................................................................
IMPALA-9989 Improve admission control pool stats logging
This work addresses the current limitation in admission controller by
appending the last known memory consumption statistics about a pool or
a host to the existing memory exhaustion message. The statistics is
logged in impalad.INFO when a query is queued or timed out due to
memory pressure on the pool or on the host. The statistics can also be
part of the query profile.
The BNF of the new memory consumption statistics is as follows.
topN_query_stats ::=
queries: a list of query Ids for up to 5 queries with top memory
consumptions
total_mem_consumed: total memory consumed by these topN queries
percentage_mem_consumed_per_pool: total memory consumed divided
by pool memory usage (if
feasible to report)
all_query_stats ::=
min: the minimal memory consumption of all running queries
max: the maximal memory consumption of all running queries
total: the total memory consumption of all running queries
average: the average memory consumption of all running queries
(if feasible to report)
pool_stats_per_host ::= <pool_name> ':' <topN_query_stats> <all_query_stats>
pool_stats ::= List of <pool_stats_per_host>
host_stats_per_pool ::= <host_name> ':' <topN_query_stats>
host_stats ::= List of <host_stats_per_pool>
memory_consumption_statistics ::= <pool_stats> | <host_stats>
The pool_stats describes memory consumption in all pools in a host and is
useful in analyzing memory exhaustion in that host. The host_stats describes
the memory consumption in all hosts for a pool and is useful in analyzing
memory exhaustion in that pool.
Example of pool_stats_per_host:
pool_name=root.queueD:
topN_query_stats:
queries=[
0000000000000003:0000000000000012,
0000000000000003:0000000000000011
],
total_mem_consumed=18.00 MB
fraction_of_pool_total_mem=0.19
all_query_stats:
num_running=20,
min=1.00 MB,
max=9.00 MB,
total_mem_consumed=95.00 MB,
average=4.75 MB
Example of host_stats_per_pool:
host_name=host2:25000:
topN_query_stats:
queries=[
0000000200000002:0000000000000001,
0000000200000002:0000000000000002,
0000000200000002:0000000000000000,
0000000200000002:0000000000000004
],
total_mem_consumed=55.00 MB
When a query request is queued due to memory exhaustion, the above
memory_consumption_statistics is loggerd when the logging is set
at level 2 or higher.
When a query request is timed out due to memory exhaustion, the above
memory_consumption_statistics is reported when the logging is set
at level 1 or higher.
Testing:
1. Added a new test TopNQueryCheck in admission-controller-test.cc to
verify that the topN query memory consumption details are reported
correctly.
2. Add two new tests in test_admission_controller.py to simulate
queries being queued and then timed out due to pool or host memory
pressure.
3. Core tests.
Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
---
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/util/container-util.h
M common/thrift/StatestoreService.thrift
M tests/custom_cluster/test_admission_controller.py
8 files changed, 885 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/26
--
To view, visit http://gerrit.cloudera.org:8080/16220
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
Gerrit-Change-Number: 16220
Gerrit-PatchSet: 26
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Sahil Takiar <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
