Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20039
to look at the new patch set (#4).
Change subject: IMPALA-11887: Support auto adjust mt_dop when CPU utilization
is high
......................................................................
IMPALA-11887: Support auto adjust mt_dop when CPU utilization is high
MT_DOP can greatly improve query performance in some cases. When MT_DOP is
turned on, the number of fragment instances may increase exponentially, and the
CPU utilization of the backends node will also increase accordingly. At this
time, the backends node may be due to excessive Slow response due to high node
CPU load. Therefore, we propose to take the node load into consideration when
setting MT_DOP. When the ratio of the number of high-load nodes to the total
number of nodes in the cluster reaches the specified threshold (controlled by
high_cpu_node_threshold), the value of MT_DOP is automatically divided by a
reduction factor, the result is a new MT_DOP (the minimum is 0, that is, MT_DOP
is turned off), and the high load environment lasts for a period of time
(controlled by high_load_keep_time_threshold) or a specific number of queries
(controlled by high_load_query_count_threshold) are executed during high load.
After that, the reduction factor will be doubled to a maximum of 16. When the
cluster load drops, the reduction factor will be reset. Realize the effect of
automatically adjusting MT_DOP according to the cluster load.
This patch consists of two parts. First, based on the heartbeat mechanism
between impalad and statestored, the statestored node collects the CPU
indicators of impalad nodes, and sends the CPU data of all impalads in the
cluster to the coordinator through heartbeat. Then the coordinator side
realizes whether to start reducing the value of MT_DOP according to the preset
threshold.
Change-Id: I5e52fbf0373d2bbffcb8832d821a0f227bb5a4c7
---
M be/src/common/global-flags.cc
M be/src/runtime/exec-env.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M be/src/util/backend-gflag-util.cc
M be/src/util/system-state-info.cc
M be/src/util/system-state-info.h
M common/thrift/BackendGflags.thrift
M common/thrift/StatestoreService.thrift
13 files changed, 245 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/20039/4
--
To view, visit http://gerrit.cloudera.org:8080/20039
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5e52fbf0373d2bbffcb8832d821a0f227bb5a4c7
Gerrit-Change-Number: 20039
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
