Jason Fehr has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21142 )
Change subject: IMPALA-12737: Query columns in workload management tables. ...................................................................... IMPALA-12737: Query columns in workload management tables. Adds "Select Columns", "Where Columns", "Join Columns", "Aggregate Columns", and "OrderBy Columns" to the query profile and the workload management active/completed queries tables. These fields are presented as comma separate strings containing the fully qualified column name in the format database.table_name.column_name. Aggregate columns include all columns in the order by and having clauses. Since new columns are being added, the workload management init process is also being modified to allow for one-way upgrades of the table schemas if necessary. Additionally, workload management can be set up to run under a schema version that is not the latest. This ability will be useful during troubleshooting. To enable these upgrades, the workload management initialization that manages the structure of the tables has been moved to the catalogd. The changes in this patch must be backwards compatible so that Impala clusters running previous workload management code can co-exist with Impala clusters running this workload management code. To enable that backwards compatibility, a new table property named 'wm_schema_version' is now used to track the schema version of the workload management tables. Thus, the old property 'schema_version' will always be set to '1.0.0' since modifying that property value causes Impala running previous workload management code to error at startup. Testing accomplished by * Adding/updating workload and custom cluster tests to assert the new columns and the workload management upgrade process. * JUnit tests added to verify the new workload management columns are being correctly parsed. * GTests added to ensure the workload management columns are correctly defined and in the correct order. Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Reviewed-on: http://gerrit.cloudera.org:8080/21142 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Riza Suminto <[email protected]> --- M be/CMakeLists.txt M be/src/catalog/CMakeLists.txt M be/src/catalog/catalog-server.h M be/src/catalog/catalogd-main.cc A be/src/catalog/workload-management-init.cc M be/src/exec/system-table-scanner.cc M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-state-record.cc M be/src/service/query-state-record.h D be/src/service/workload-management-fields.cc D be/src/service/workload-management-init.cc D be/src/service/workload-management-test.cc A be/src/service/workload-management-worker-test.cc A be/src/service/workload-management-worker.cc A be/src/service/workload-management-worker.h D be/src/service/workload-management.cc D be/src/service/workload-management.h M be/src/util/version-util-test.cc M be/src/util/version-util.cc M be/src/util/version-util.h A be/src/workload_mgmt/CMakeLists.txt A be/src/workload_mgmt/workload-management-fields-test.cc R be/src/workload_mgmt/workload-management-flags.cc A be/src/workload_mgmt/workload-management-test.cc A be/src/workload_mgmt/workload-management.cc A be/src/workload_mgmt/workload-management.h M common/thrift/Frontend.thrift M common/thrift/SystemTables.thrift M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ArithmeticExpr.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/test/java/org/apache/impala/planner/ColumnsTest.java A testdata/workload_mgmt/create_impala_query_live_table.sql A testdata/workload_mgmt/create_impala_query_log_table.sql R testdata/workloads/functional-query/queries/QueryTest/workload-management-live-v1.0.0.test A testdata/workloads/functional-query/queries/QueryTest/workload-management-live-v1.1.0.test R testdata/workloads/functional-query/queries/QueryTest/workload-management-log-v1.0.0.test A testdata/workloads/functional-query/queries/QueryTest/workload-management-log-v1.1.0.test M tests/common/custom_cluster_test_suite.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_query_live.py M tests/custom_cluster/test_query_log.py A tests/custom_cluster/test_workload_mgmt_init.py A tests/custom_cluster/test_workload_mgmt_sql_details.py M tests/query_test/test_observability.py M tests/util/workload_management.py 60 files changed, 4,657 insertions(+), 1,556 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Impala Public Jenkins: Verified Riza Suminto: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I78f3670b067c0c192ee8a212fba95466fbcb51d7 Gerrit-Change-Number: 21142 Gerrit-PatchSet: 68 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
