Hello Andrew Sherman, Riza Suminto, Jason Fehr, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20762
to look at the new patch set (#47).
Change subject: IMPALA-12540: Query Live Table
......................................................................
IMPALA-12540: Query Live Table
Defines SystemTable which are in-memory tables that can provide access
to Impala state. Adds the 'impala_query_live' to the database 'sys',
which already exists for 'sys.impala_query_log'.
Implements the 'impala_query_live' table to view active queries across
all coordinators sharing the same statestore. SystemTables create new
SystemTableScanNodes for their scan node implementation. When computing
scan range locations, SystemTableScanNodes creates a scan range for each
in the cluster (identified via ClusterMembershipMgr). This produces a
plan that looks like:
Query: explain select * from sys.impala_query_live
+------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=4.00MB Threads=2 |
| Per-Host Resource Estimates: Memory=11MB |
| WARNING: The following tables are missing relevant table |
| and/or column statistics. |
| sys.impala_query_live |
| |
| PLAN-ROOT SINK |
| | |
| 01:EXCHANGE [UNPARTITIONED] |
| | |
| 00:SCAN SYSTEM_TABLE [sys.impala_query_live] |
| row-size=72B cardinality=20 |
+------------------------------------------------------------+
Impala's scheduler checks for whether the query contains fragments that
can be scheduled on coordinators, and if present includes an
ExecutorGroup containing all coordinators. These are used to schedule
scan ranges that are flagged as 'use_coordinator', allowing
SystemTableScanNodes to be scheduled on dedicated coordinators and
outside the selected executor group.
Execution will pull data from ImpalaServer on the backend via a
SystemTableScanner implementation based on table name.
In the query profile, SYSTEM_TABLE_SCAN_NODE includes
ActiveQueryCollectionTime and PendingQueryCollectionTime to track time
spent collecting QueryState from ImpalaServer.
Grants QueryScanner private access to ImpalaServer, identical to how
ImpalaHttpHandler access internal server state.
Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
M be/src/exec/scan-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/cluster-membership-mgr-test.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler-test-util.h
M be/src/scheduling/scheduler-test.cc
M be/src/scheduling/scheduler.cc
M be/src/scheduling/scheduler.h
M be/src/service/fe-support.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/query-state-record.h
M be/src/service/workload-management-fields.cc
M be/src/service/workload-management.cc
M be/src/util/sharded-query-map-util.h
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
A common/thrift/SystemTables.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
A fe/src/main/java/org/apache/impala/catalog/SystemTable.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
A fe/src/main/java/org/apache/impala/planner/SystemTableScanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java
A fe/src/test/java/org/apache/impala/catalog/SystemTableTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A
testdata/workloads/functional-planner/queries/PlannerTest/impala-query-live.test
A tests/custom_cluster/test_query_live.py
M tests/util/workload_management.py
52 files changed, 1,727 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/20762/47
--
To view, visit http://gerrit.cloudera.org:8080/20762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie2f9a449f0e5502078931e7f1c5df6e0b762c743
Gerrit-Change-Number: 20762
Gerrit-PatchSet: 47
Gerrit-Owner: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
