Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21257
Change subject: IMPALA-12980: Translate CpuAsk into admission control slot to use ...................................................................... IMPALA-12980: Translate CpuAsk into admission control slot to use Admission control slot accounting is described in IMPALA-8998. On each backend, number of. It compute 'slots_to_use' for each backend based on the max number of instances of any fragment on that backend. This is simplistic, because multiple fragments with same number of instance count, say 4 non-blocking fragments each with 12 instances, only request the max instance (12) admission slots rather than sum of it (48). When COMPUTE_PROCESSING_COST is enabled, Planner will generate a CpuAsk number that represent the cpu requirement of that query over a particular executor group set. This number is an estimation of what is the largest number of query fragments that can run in-parallel given the blocking operator analysis. Therefore, the fragment trace that sums into that CpuAsk number can be translated into 'slots_to_use' as well, which will be a closer resemblance of maximum parallel execution of fragment instances. This patch add new query option called SLOT_COUNT_STRATEGY to control which admission control slot accounting to use. There are two possible values: - LARGEST_FRAGMENT, which do the default algorithm from IMPALA-8998. This still the default value for SLOT_COUNT_STRATEGY option. - PLANNER_EDOP, which will follow fragment trace that contributes towards CpuAsk number. To do PLANNER_EDOP strategy, Planner will mark fragments that contribute into CpuAsk as dominant fargment. It also pass max_slot_per_executor information that it knows about the executor group set to scheduler. AvgAdmissionSlotsPerExecutor counter is added to describe what Planner think the average 'slots_to_use' per backend will be. Actual 'slots_to_use' in each backend may differ than AvgAdmissionSlotsPerExecutor, depending on what actually scheduled on that backend. Testing: - Update test_executors.py with AvgAdmissionSlotsPerExecutor assertion. - Pass test_tpcds_queries.py::TestTpcdsQueryWithProcessingCost. Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/scheduler.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaService.thrift M common/thrift/Planner.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/common/Id.java M fe/src/main/java/org/apache/impala/planner/CoreCount.java M fe/src/main/java/org/apache/impala/planner/CostingSegment.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_tpcds_queries.py 16 files changed, 299 insertions(+), 86 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/21257/1 -- To view, visit http://gerrit.cloudera.org:8080/21257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2 Gerrit-Change-Number: 21257 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto <[email protected]>
