Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21257
to look at the new patch set (#2).
Change subject: IMPALA-12980: Translate CpuAsk into admission control slot to
use
......................................................................
IMPALA-12980: Translate CpuAsk into admission control slot to use
Admission control slot accounting is described in IMPALA-8998. On each
backend, number of. It compute 'slots_to_use' for each backend based on
the max number of instances of any fragment on that backend. This is
simplistic, because multiple fragments with same number of instance
count, say 4 non-blocking fragments each with 12 instances, only request
the max instance (12) admission slots rather than sum of it (48).
When COMPUTE_PROCESSING_COST is enabled, Planner will generate a CpuAsk
number that represent the cpu requirement of that query over a
particular executor group set. This number is an estimation of what is
the largest number of query fragments that can run in-parallel given the
blocking operator analysis. Therefore, the fragment trace that sums into
that CpuAsk number can be translated into 'slots_to_use' as well, which
will be a closer resemblance of maximum parallel execution of fragment
instances.
This patch add new query option called SLOT_COUNT_STRATEGY to control
which admission control slot accounting to use. There are two possible
values:
- LARGEST_FRAGMENT, which do the default algorithm from IMPALA-8998.
This still the default value for SLOT_COUNT_STRATEGY option.
- PLANNER_EDOP, which will follow fragment trace that contributes
towards CpuAsk number.
To do PLANNER_EDOP strategy, Planner will mark fragments that contribute
into CpuAsk as dominant fargment. It also pass max_slot_per_executor
information that it knows about the executor group set to scheduler.
AvgAdmissionSlotsPerExecutor counter is added to describe what Planner
think the average 'slots_to_use' per backend will be. Actual
'slots_to_use' in each backend may differ than
AvgAdmissionSlotsPerExecutor, depending on what actually scheduled on
that backend.
Testing:
- Update test_executors.py with AvgAdmissionSlotsPerExecutor assertion.
- Pass test_tpcds_queries.py::TestTpcdsQueryWithProcessingCost.
Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2
---
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/scheduler.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Planner.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/common/Id.java
M fe/src/main/java/org/apache/impala/planner/CoreCount.java
M fe/src/main/java/org/apache/impala/planner/CostingSegment.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M tests/custom_cluster/test_executor_groups.py
M tests/query_test/test_tpcds_queries.py
16 files changed, 311 insertions(+), 82 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/21257/2
--
To view, visit http://gerrit.cloudera.org:8080/21257
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I338ca96555bfe8d07afce0320b3688a0861663f2
Gerrit-Change-Number: 21257
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
