Hello Riza Suminto, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21653
to look at the new patch set (#4).
Change subject: IMPALA-12737: Limit workload management initialization to one
coordinator.
......................................................................
IMPALA-12737: Limit workload management initialization to one coordinator.
The workload management initialization process creates the two tables
"sys.impala_query_log" and "sys.impala_query_live" during coordinator
startup.
The current design for this init process is to create both tables on
each coordinator at startup by running create table if not exists
DDLs. This design causes unnecessary DDLs to execute which delays
coordinator startup and introduces the potential for unnecessary
startup failures should the DDLs fail. Additionally, this design will
not work once table schema modifications are needed since multiple
coordinators running the same create and alter table statements
while the catalog is warming results in conflicts.
This change introduces using the statestore to determine which
coordinator will run the workload management initialization process.
The first coordinator that puts a message on the new
impala-workload-management statestore topic will be the coordinator
that runs all the DDLs to create the workload management tables. This
coordinator places a follow-up message on the same statestore topic
to indicate it has completed workload management initialization.
At that point, all the coordinators start writing completed queries
to the completed queries table.
Additionally, this patch adds functionality to first check if the two
workload management tables already exist and skip running the create
table DDL if they already exist.
A new coordinator startup flag "workload_mgmt_schema_version" has
been added to prepare for changing the workload management tables
schemas.
Testing is accomplished by adding new ctests and custom cluster tests.
The workload management init process is tested by starting a cluster,
adding more coordinators after the cluster completes startup, and
forcing delays in the startup process to simulate delays in forming
a large cluster.
Change-Id: Id645f94c8da73b91c13a23d7ac0ea026425f0f96
---
M be/src/service/CMakeLists.txt
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/workload-management-fields.cc
M be/src/service/workload-management-flags.cc
A be/src/service/workload-management-init.cc
A be/src/service/workload-management-init.h
M be/src/service/workload-management.cc
M be/src/service/workload-management.h
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M be/src/util/CMakeLists.txt
A be/src/util/version-util-test.cc
A be/src/util/version-util.cc
A be/src/util/version-util.h
M bin/start-impala-cluster.py
M tests/common/impala_cluster.py
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_query_log.py
A tests/custom_cluster/test_workload_mgmt_init.py
21 files changed, 1,227 insertions(+), 227 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/21653/4
--
To view, visit http://gerrit.cloudera.org:8080/21653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id645f94c8da73b91c13a23d7ac0ea026425f0f96
Gerrit-Change-Number: 21653
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
