Qifan Chen has uploaded a new patch set (#29). (
http://gerrit.cloudera.org:8080/18143 )
Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1
......................................................................
IMPALA-10992 Planner changes for estimate peak memory - v1
This patch provides replan support in planner for a set of executor
groups. Each executor group in the set is associated with a distinct
number of nodes and a threshold for estimated memory per host in bytes
that can be denoted as [<group_name_prefix>:<#nodes>, <threshold>].
In the patch, a query of type EXPLAIN, QUERY or DML can be compiled
more than once. In each attempt, per host memory is estimated and
compared with the threshold of an executor group. If the estimated
memory is no more than the threshold, the iteration process terminates
and the final plan is determined. The executor group with the threshold
is selected to run against the query.
A new query option 'enable_replan', default to 1 (enabled), is added.
It can be set to 0 to disable this patch and to generate the
distributed plan for the default executor group.
To facilitate testing, the patch imposes an artificial two executor
group setup in FE. This setup is enabled when 'enable_replan' is set
to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most
frontend tests. The artificial two executor groups are configured as
follows.
1. [regular:3, 64MB]
2. [large:3, 8PB]
To avoid long compilation time, the following enhancement is enabled.
Note 1) and 2) can be disabled when some meta-data change is detected.
1. Authorization is performed only for the 1st compilation;
2. The relevant meta-data is fetched into a StmtTableCache in 1st
compilation and reused in subsequent compilations;
3. openTransaction() is called for transactional queries in 1st
compilation and the saved transactional info is used in
subsequent compilations. Similar logic is applied to Kudu
transactional queries.
Testing:
1. Ran core tests
Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/debug-util.cc
M be/src/util/debug-util.h
M common/thrift/Frontend.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/ClassUtil.java
M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/common/QueryFixture.java
M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java
M
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_coordinators.py
M tests/custom_cluster/test_executor_groups.py
M tests/query_test/test_observability.py
21 files changed, 499 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/29
--
To view, visit http://gerrit.cloudera.org:8080/18143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24
Gerrit-Change-Number: 18143
Gerrit-PatchSet: 29
Gerrit-Owner: Qifan Chen <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
