Qifan Chen has uploaded a new patch set (#25). (
http://gerrit.cloudera.org:8080/16220 )
Change subject: IMPALA-9989 Improve admission control pool stats logging
......................................................................
IMPALA-9989 Improve admission control pool stats logging
This work addresses the current limitation in admission controller by
appending the last known memory consumption statistics about a pool or
a host to the existing memory exhaustion message. The message is
logged in impalad.INFO when a query is queued or timed out due to
memory pressure on the pool or on the host.
This new memory consumption statistics covers the following content:
topN_query_stats ::=
queries: a list of query Ids for up to 5 queries with top memory
consumptions
total_mem_consumed: total memory consumed by these topN queries
percentage_mem_consumed_per_pool: total memory consumed divided
by pool memory usage (if
feasible to report)
all_query_stats ::=
min: the minimal memory consumption of all running queries
max: the maximal memory consumption of all running queries
total: the total memory consumption of all running queries
average: the average memory consumption of all running queries
(if feasible to report)
pool_stats_per_host ::=
<pool_name>: <topN_query_stats> <all_query_stats>
pool_stats::=
List of <pool_stats_per_host>
host_stats_per_pool ::=
<host_name>: <topN_query_stats>
host_stats::=
List of <host_stats_per_pool>
memory_consumption_statistics ::=
<host_stats_per_pool> | <host_stats>
pool_stats describes memory consumption in all pools in a host
and is useful in analyzing memory exhaustion in that host.
host_stats describes the memory consumption for all hosts in a pool
and is useful in analyzing memory exhaustion in that pool.
Example of pool_stats_per_host:
pool_name=root.queueD:
topN_query_stats:
queries=[
0000000000000003:0000000000000012,
0000000000000003:0000000000000011
],
total_mem_consumed=18.00 MB
fraction_of_pool_total_mem=0.19
all_query_stats:
num_running=20,
min=1.00 MB,
max=9.00 MB,
total_mem_consumed=95.00 MB,
average=4.75 MB
Example of host_stats_per_pool:
host_name=host2:25000:
topN_query_stats:
queries=[
0000000200000002:0000000000000001,
0000000200000002:0000000000000002,
0000000200000002:0000000000000000,
0000000200000002:0000000000000004
],
total_mem_consumed=55.00 MB
When a query request is queued due to memory exhaustion, the above
memory_consumption_statistics is loggerd when the logging is set
at level 2 or higher.
When a query request is timed out due to memory exhaustion, the above
memory_consumption_statistics is reported when the logging is set
at level 1 or higher.
Testing:
1. Added a new test TopNQueryCheck in admission-controller-test.cc to
verify that the topN query memory consumption details are reported
correctly.
2. Add two new tests in test_admission_controller.py to simulate
queries being queued and then timed out due to pool or host memory
pressure.
3. Core tests.
Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
---
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/util/container-util.h
M common/thrift/StatestoreService.thrift
M tests/custom_cluster/test_admission_controller.py
8 files changed, 885 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/25
--
To view, visit http://gerrit.cloudera.org:8080/16220
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781
Gerrit-Change-Number: 16220
Gerrit-PatchSet: 25
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Sahil Takiar <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
