[email protected] has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20039
Change subject: IMPALA-11887: Support auto adjust mt_dop when CPU utilization is high ...................................................................... IMPALA-11887: Support auto adjust mt_dop when CPU utilization is high MT_DOP can greatly improve query performance in some cases. When MT_DOP is turned on, the number of fragment instances may increase exponentially, and the CPU utilization of the backends node will also increase accordingly. At this time, the backends node may be due to excessive Slow response due to high node CPU load. Therefore, we propose to take the node load into consideration when setting MT_DOP. When the ratio of the number of high-load nodes to the total number of nodes in the cluster reaches the specified threshold (controlled by high_cpu_node_threshold), the value of MT_DOP is automatically divided by a reduction factor, the result is a new MT_DOP (the minimum is 0, that is, MT_DOP is turned off), and the high load environment lasts for a period of time (controlled by high_load_keep_time_threshold) or a specific number of queries (controlled by high_load_query_count_threshold) are executed during high load. After that, the reduction factor will be doubled to a maximum of 16. When the cluster load drops, the reduction factor will be reset. Realize the effect of automatically adjusting MT_DOP according to the cluster load. This patch consists of two parts. First, based on the heartbeat mechanism between impalad and statestored, the statestored node collects the CPU indicators of impalad nodes, and sends the CPU data of all impalads in the cluster to the coordinator through heartbeat. Then the coordinator side realizes whether to start reducing the value of MT_DOP according to the preset threshold. Change-Id: I5e52fbf0373d2bbffcb8832d821a0f227bb5a4c7 --- M be/src/common/global-flags.cc M be/src/runtime/exec-env.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/util/backend-gflag-util.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h M common/thrift/BackendGflags.thrift M common/thrift/StatestoreService.thrift 13 files changed, 237 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/20039/1 -- To view, visit http://gerrit.cloudera.org:8080/20039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5e52fbf0373d2bbffcb8832d821a0f227bb5a4c7 Gerrit-Change-Number: 20039 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <[email protected]>
