[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 05:56:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. IMPALA-8125: Add query option to limit number of hdfs writer instances This patch adds a new query option MAX_FS_WRITERS that limits the number of HDFS writer instances. Highlights: - Depending on the plan, it either restricts the num of instances of the root fragment or adds an exchange and then limits the num of instances of that. - Assigns instances evenly across available backends. - "no-shuffle" query hint is ignored when using query option. - Change in behavior of plans is only when this query option is used. - The only exception to the previous point is that the optimization logic that decides to add an exchange now looks at the num of instances instead of the number of nodes. Limitation: A mismatch of cluster state during query planning and scheduling can result in more or less fragment instances to be scheduled than expected. Eg. If max_fs_writers in 2 and the planner sees only 2 executors then it might not add an exchange between a scan node and the table sink, but during scheduling if there are 3 nodes then that scan+tablesink instance will be scheduled on 3 backends. Testing: - Added planner tests to cover all cases where this enforcement kicks in and to highlight the behavior. - Added e2e tests to confirm that the scheduler is enforcing the limit and distributing the instance evenly across backends for different plan shapes. Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Reviewed-on: http://gerrit.cloudera.org:8080/16204 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/insert-hdfs-writer-limit.test M tests/query_test/test_insert.py 16 files changed, 903 insertions(+), 34 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 10 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. IMPALA-10010: Add option to configure metrics webserver Currently, when security is turned on for the webui, eg. with --webserver_require_ldap, authentication is applied to all webui endpoints. However, there are some endpoints that expose low-sensitivity info and which are scraped by other systems that it may be difficult to get credentials to in order to be able to authenticate. This patch adds a flag, --metrics_webserver_port, which if specified turns on an unsecured webserver that exposes only the /metrics, /jsonmetrics, /metrics_prometheus for all Impala daemons. Impalads also have the /healthz endpoint exposed. Testing: - Added a test that turns on the metrics server and verifies its reachable and unsecured. Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Reviewed-on: http://gerrit.cloudera.org:8080/16270 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalogd-main.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/statestore/statestored-main.cc M be/src/util/metrics.cc M be/src/util/metrics.h M be/src/util/webserver.cc M be/src/util/webserver.h M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java M fe/src/test/java/org/apache/impala/util/Metrics.java 14 files changed, 161 insertions(+), 43 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 05:40:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16280 ) Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6776/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad Gerrit-Change-Number: 16280 Gerrit-PatchSet: 1 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 02:40:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6217/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 04 Aug 2020 02:35:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16204 to look at the new patch set (#8). Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. IMPALA-8125: Add query option to limit number of hdfs writer instances This patch adds a new query option MAX_FS_WRITERS that limits the number of HDFS writer instances. Highlights: - Depending on the plan, it either restricts the num of instances of the root fragment or adds an exchange and then limits the num of instances of that. - Assigns instances evenly across available backends. - "no-shuffle" query hint is ignored when using query option. - Change in behavior of plans is only when this query option is used. - The only exception to the previous point is that the optimization logic that decides to add an exchange now looks at the num of instances instead of the number of nodes. Limitation: A mismatch of cluster state during query planning and scheduling can result in more or less fragment instances to be scheduled than expected. Eg. If max_fs_writers in 2 and the planner sees only 2 executors then it might not add an exchange between a scan node and the table sink, but during scheduling if there are 3 nodes then that scan+tablesink instance will be scheduled on 3 backends. Testing: - Added planner tests to cover all cases where this enforcement kicks in and to highlight the behavior. - Added e2e tests to confirm that the scheduler is enforcing the limit and distributing the instance evenly across backends for different plan shapes. Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 --- M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/insert-hdfs-writer-limit.test M tests/query_test/test_insert.py 16 files changed, 903 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/16204/8 -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16280 ) Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload. .. Patch Set 1: Adding the missing tpc-ds queries. Easy spot to compare the query against https://github.com/cwida/tpcds-result-reproduction Assuming decimalv2 will be the default soon, so only added templates for those queries. Also there are slight variances from the answer sets due to decimal rounding, I assume this is expected behavior but wouldn't mind a second set of eyes. -- To view, visit http://gerrit.cloudera.org:8080/16280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad Gerrit-Change-Number: 16280 Gerrit-PatchSet: 1 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 02:23:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16280 to look at the new patch set (#2). Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload. .. IMPALA-10034: Add remaining TPC-DS queries to workload. Include remaining TPC-DS queries to the testdata workload definition. Q8 and Q38 were using non standard variants, those have been replaced by the official query versions. Q35 is using an official variant. Had to escape a table alias in Q90 as we treat 'AT' as a reserved keyword. Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad --- A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-2.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-2.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q28.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q35.test D testdata/workloads/tpcds/queries/tpcds-decimal_v2-q38-rewrite.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q38.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q44.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q49.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q66.test M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q8.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q87.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q90.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q93.test M tests/query_test/test_tpcds_queries.py M tests/util/parse_util.py 17 files changed, 1,249 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/16280/2 -- To view, visit http://gerrit.cloudera.org:8080/16280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad Gerrit-Change-Number: 16280 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10034: Add remaining TPC-DS queries to workload.
Shant Hovsepian has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16280 Change subject: IMPALA-10034: Add remaining TPC-DS queries to workload. .. IMPALA-10034: Add remaining TPC-DS queries to workload. Include remaining TPC-DS queries to the testdata workload definition. Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad --- A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-1.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q23-2.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-1.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q24-2.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q28.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q35.test D testdata/workloads/tpcds/queries/tpcds-decimal_v2-q38-rewrite.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q38.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q44.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q49.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q66.test M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q8.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q87.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q90.test A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q93.test M tests/query_test/test_tpcds_queries.py M tests/util/parse_util.py 17 files changed, 1,249 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/16280/1 -- To view, visit http://gerrit.cloudera.org:8080/16280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id5436689390f149694f14e6da1df624de4f5f7ad Gerrit-Change-Number: 16280 Gerrit-PatchSet: 1 Gerrit-Owner: Shant Hovsepian
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6775/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 01:10:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6216/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 00:51:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 00:51:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8125: Add query option to limit number of hdfs writer instances
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16204 ) Change subject: IMPALA-8125: Add query option to limit number of hdfs writer instances .. Patch Set 8: Code-Review+2 a test was failing in dockerised tests. The issue was that while checking for whether writer limits need to be applied, i was only checking whether the target table was a HdfsTable or not. I missed checking for its counterpart, LocalFsTable, when local catalog mode is turned on. Since in dockerised testing mode the impala process is started in local catalog mode, this test failed. Carrying forward Tim's +2 -- To view, visit http://gerrit.cloudera.org:8080/16204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17c8e61b9a32d908eec82c83618ff9caa41078a5 Gerrit-Change-Number: 16204 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 00:51:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6215/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 00:30:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 04 Aug 2020 00:30:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 23:37:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16270 ) Change subject: IMPALA-10010: Add option to configure metrics webserver .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6774/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 22:23:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10010: Add option to configure metrics webserver
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16270 to look at the new patch set (#2). Change subject: IMPALA-10010: Add option to configure metrics webserver .. IMPALA-10010: Add option to configure metrics webserver Currently, when security is turned on for the webui, eg. with --webserver_require_ldap, authentication is applied to all webui endpoints. However, there are some endpoints that expose low-sensitivity info and which are scraped by other systems that it may be difficult to get credentials to in order to be able to authenticate. This patch adds a flag, --metrics_webserver_port, which if specified turns on an unsecured webserver that exposes only the /metrics, /jsonmetrics, /metrics_prometheus for all Impala daemons. Impalads also have the /healthz endpoint exposed. Testing: - Added a test that turns on the metrics server and verifies its reachable and unsecured. Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d --- M be/src/catalog/catalogd-main.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/statestore/statestored-main.cc M be/src/util/metrics.cc M be/src/util/metrics.h M be/src/util/webserver.cc M be/src/util/webserver.h M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java M fe/src/test/java/org/apache/impala/util/Metrics.java 14 files changed, 161 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/16270/2 -- To view, visit http://gerrit.cloudera.org:8080/16270 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibcf297d798a1a5c9cd59d4d82706d2d945e10d3d Gerrit-Change-Number: 16270 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16273/1/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/16273/1/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@551 PS1, Line 551: alterKuduTable(tbl, alterTableOptions, errMsg); This may be more a question for Kudu, but in what versions of Kudu is native ownership not supported? In those versions, what does this do? From the commit message here https://github.com/apache/kudu/commit/f0446b73630d75f6bf9c11b3fcce8953c557b578 it looks like the owner is synchronized between Kudu and the HMS. Attila, can you chime in with what should the behavior here be when the HMS synchronization is disabled vs enabled? Also, should there any difference in behavior when the Impala table is external/internal? -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 20:27:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16098 ) Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6773/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576 Gerrit-Change-Number: 16098 Gerrit-PatchSet: 26 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 18:33:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6772/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 21 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 18:27:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans
Qifan Chen has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/16098 ) Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans .. IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans This work addresses the current limitation in computing the total row count for a Hive table in a scan. The row count can be incorrectly computed as 0, even though there exists data in the Hive table. This is the stats corruption at table level. Similar stats corruption exists for a partition. The row count of a table or a partition sometime can also be -1 which indicates a missing stats situation. In the fix, as long as no partition in a Hive table exhibits any missing or corrupt stats, the total row count for the table is computed from the row counts in all partitions. Otherwise, Impala looks at the table level stats particularly the table row count. In addition, if the table stats is missing or corrupted, Impala estimates a row count for the table, if feasible. This row count is the sum of the row count from the partitions with good stats, and an estimation of the number of rows in the partitions with missing or corrupt stats. Such estimation also applies when some partition has missing or corrupt stats. One way to observe the fix is through the explain of queries scanning Hive tables with missing or corrupted stats. The cardinality for any full scan should be a positive value (i.e. the estimated row count), instead of 'unavailable'. At the beginning of the explain output, that table is still listed in the WARNING section for potentially corrupt table statistics. Testing: 1. Ran unit tests with queries documented in the case against Hive tables with the following configrations: a. No stats corruption in any partitions b. Stats corruption in some partitions c. Stats corruption in all partitions 2. Added two new tests in test_compute_stats.py: a. test_corrupted_stats_in_partitioned_Hive_tables b. test_corrupted_stats_in_unpartitioned_Hive_tables 3. Fixed failures in corrupt-stats.test 4. Ran "core" test Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576 --- M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-query/queries/QueryTest/corrupt-stats.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/metadata/test_compute_stats.py M tests/metadata/test_explain.py 5 files changed, 193 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/16098/26 -- To view, visit http://gerrit.cloudera.org:8080/16098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576 Gerrit-Change-Number: 16098 Gerrit-PatchSet: 26 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Qifan Chen has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. IMPALA-9989 Improve admission control pool stats logging This work addresses the current limitation in admission controller by appending the last known memory consumption statistics about a pool or a host to the existing memory exhaustion message. The message is logged in impalad.INFO when a query is queued or timed out due to memory pressure on the pool or on the host. This new memory consumption statistics covers the following content: topN_query_stats ::= queries: a list of query Ids for up to 5 queries with top memory consumptions total_mem_consumed: total memory consumed by these topN queries percentage_mem_consumed_per_pool: total memory consumed divided by pool memory usage (if feasible to report) all_query_stats ::= min: the minimal memory consumption of all running queries max: the maximal memory consumption of all running queries total: the total memory consumption of all running queries average: the average memory consumption of all running queries (if feasible to report) pool_stats_per_host ::= : pool_stats::= List of host_stats_per_pool ::= : host_stats::= List of memory_consumption_statistics ::= | pool_stats describes memory consumption in all pools in a host and is useful in analyzing memory exhaustion in that host. host_stats describes the memory consumption for all hosts in a pool and is useful in analyzing memory exhaustion in that pool. Example of pool_stats_per_host: pool_name=root.queueD: topN_query_stats: queries=[ 0003:0012, 0003:0011 ], total_mem_consumed=18.00 MB fraction_of_pool_total_mem=0.19 all_query_stats: num_running=20, min=1.00 MB, max=9.00 MB, total_mem_consumed=95.00 MB, average=4.75 MB Example of host_stats_per_pool: host_name=host2:25000: topN_query_stats: queries=[ 00020002:0001, 00020002:0002, 00020002:, 00020002:0004 ], total_mem_consumed=55.00 MB When a query request is queued due to memory exhaustion, the above memory_consumption_statistics is loggerd when the logging is set at level 2 or higher. When a query request is timed out due to memory exhaustion, the above memory_consumption_statistics is reported when the logging is set at level 1 or higher. Testing: 1. Added a new test TopNQueryCheck in admission-controller-test.cc to simulate queries running in 4 pools in 3 hosts. This new test identifies the following: a. Top 5 queries among 4 pools in host 0; a. Top 5 queries among 4 pools in host 1; c. Top 5 queries among 3 hosts for a pool. 2. Core tests. Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 --- M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/util/container-util.h M common/thrift/StatestoreService.thrift 7 files changed, 827 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16220/21 -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 21 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9909: Print body of http error code in Impala Shell.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16269 ) Change subject: IMPALA-9909: Print body of http error code in Impala Shell. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6771/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff Gerrit-Change-Number: 16269 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Aug 2020 17:35:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9909: Print body of http error code in Impala Shell.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16269 ) Change subject: IMPALA-9909: Print body of http error code in Impala Shell. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6770/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff Gerrit-Change-Number: 16269 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Aug 2020 17:31:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9909: Print body of http error code in Impala Shell.
Andrew Sherman has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16269 ) Change subject: IMPALA-9909: Print body of http error code in Impala Shell. .. IMPALA-9909: Print body of http error code in Impala Shell. Make Impala Shell closer to Impyla by printing the body of any http error code message received when using hs2-over-http. The common case is that there is nothing in the body, in which case the behavior is unchanged. TESTING Added a test for the new functionality. Ran all end-to-end tests. Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff --- M shell/ImpalaHttpClient.py M tests/shell/test_shell_interactive.py 2 files changed, 90 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/16269/4 -- To view, visit http://gerrit.cloudera.org:8080/16269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff Gerrit-Change-Number: 16269 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9909: Print body of http error code in Impala Shell.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16269 ) Change subject: IMPALA-9909: Print body of http error code in Impala Shell. .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/16269/3/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/16269/3/tests/shell/test_shell_interactive.py@84 PS3, Line 84: e flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16269/3/tests/shell/test_shell_interactive.py@1080 PS3, Line 1080: T flake8: E501 line too long (93 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff Gerrit-Change-Number: 16269 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Aug 2020 17:02:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9909: Print body of http error code in Impala Shell.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16269 Change subject: IMPALA-9909: Print body of http error code in Impala Shell. .. IMPALA-9909: Print body of http error code in Impala Shell. Make Impala Shell closer to Impyla by printing the body of any http error code message received when using hs2-over-http. The common case is that there is nothing in the body, in which case the behavior is unchanged. TESTING Added a test for the new functionality. Ran all end-to-end tests. Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff --- M shell/ImpalaHttpClient.py M tests/shell/test_shell_interactive.py 2 files changed, 88 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/16269/3 -- To view, visit http://gerrit.cloudera.org:8080/16269 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iabc45eda0b87ca694b8359148cda6a7c1d5a8fff Gerrit-Change-Number: 16269 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman
[Impala-ASF-CR] IMPALA-9741: Support querying Iceberg table by impala
Anonymous Coward (606) has posted comments on this change. (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
..
Patch Set 16:
(4 comments)
The code looks fine. All of my comments are optional. Thank you for the
opportunity to review.
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16143/16//COMMIT_MSG@29
PS16, Line 29: We achieved this function by treating the iceberg table as normal
Are there plans to support read of Iceberg table as a partitioned table? This
will help with collocated joins.
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java@80
PS16, Line 80: "SHOW FILES not applicable to a non hdfs table and non
iceberg table: %s",
The double negative is pretty hard to parse.
How about:
"SHOW FILES is applicable only to a HDFS table"?
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
File fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java@139
PS16, Line 139: if (!(predicate.getChild(0) instanceof SlotRef)) return
false;
Can the predicate be of the form: '10 = p1'? In that case, should there be
symmetric code to get slotRef and literalExpr?
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/16143/16/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@114
PS16, Line 114: if ("PARQUET".equalsIgnoreCase(format)) return
TIcebergFileFormat.PARQUET;
Rest of the code seems to support Iceberg ORC file format. This code does not
seem to support it?
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Anonymous Coward (606)
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Mon, 03 Aug 2020 16:46:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9984: Implement codegen for TupleIsNullPredicate
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16227 ) Change subject: IMPALA-9984: Implement codegen for TupleIsNullPredicate .. Patch Set 3: I looped those end-to-end tests over the weekend and couldn't reproduce. I can rerun the merge after your cleanup. -- To view, visit http://gerrit.cloudera.org:8080/16227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I410aa7ec762ca16f455bd7da1dce763c1a7b156e Gerrit-Change-Number: 16227 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 16:23:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6213/ -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Mon, 03 Aug 2020 15:32:56 + Gerrit-HasComments: No
[native-toolchain-CR] Fix Debian 8 Dockerfile to handle expired repos
Laszlo Gaal has abandoned this change. ( http://gerrit.cloudera.org:8080/15843 ) Change subject: Fix Debian 8 Dockerfile to handle expired repos .. Abandoned Debian 8 reached end-of-support on June 30, 2020, so Impala is retiring support for it too. -- To view, visit http://gerrit.cloudera.org:8080/15843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ic1fe3a43cc1ef753d63d8fd76a029cf9ef657219 Gerrit-Change-Number: 15843 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Mon, 03 Aug 2020 10:32:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6213/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Mon, 03 Aug 2020 10:26:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6768/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Mon, 03 Aug 2020 10:26:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3316 PS3, Line 3316:* condition on the inner table that filters out non‑matching rows. In a left outer > line too long (93 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Mon, 03 Aug 2020 10:07:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Xianqing He has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. IMPALA-5022: Outer join simplification As a general rule, an outer join can be converted to an inner join if there is a condition on the inner table that filters out non‑matching rows. In a left outer join, the right table is the inner table, while it is the left table in a right outer join. In a full outer join, both tables are inner tables. Conditions that are FALSE for nulls are referred to as null filtering conditions, and these are the conditions that enable the outer‑to‑inner join conversion to be made. An outer join can be converted to an inner join if the WHERE clause contains at least one null rejecting condition on the inner table. For example, 1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE B.v > 10 2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10 = A INNER JOIN B ON A.id = B.id WHERE B.v > 10 3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 = A LEFT JOIN B ON A.id = B.id WHERE A.v > 10 4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10 = A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10 5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 Tests: * Update the baseline plan Tests * Add some plan tests in outer-joins.test * Ran the full set of verifications in Impala Public Jenkins Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test 21 files changed, 1,545 insertions(+), 968 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/4 -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/16266/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3316 PS3, Line 3316:* condition on the inner table that filters out non‑matching rows. In a left outer join, line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 03 Aug 2020 09:59:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Xianqing He has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16266 Change subject: IMPALA-5022: Outer join simplification .. IMPALA-5022: Outer join simplification As a general rule, an outer join can be converted to an inner join if there is a condition on the inner table that filters out non‑matching rows. In a left outer join, the right table is the inner table, while it is the left table in a right outer join. In a full outer join, both tables are inner tables. Conditions that are FALSE for nulls are referred to as null filtering conditions, and these are the conditions that enable the outer‑to‑inner join conversion to be made. An outer join can be converted to an inner join if the WHERE clause contains at least one null rejecting condition on the inner table. For example, 1. A LEFT JOIN B ON A.id = B.id WHERE B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE B.v > 10 2. A RIGHT JOIN B ON A.id = B.id WHERE A.v > 10 = A INNER JOIN B ON A.id = B.id WHERE B.v > 10 3. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 = A LEFT JOIN B ON A.id = B.id WHERE A.v > 10 4. A FULL JOIN B ON A.id = B.id WHERE B.v > 10 = A RIGHT JOIN B ON A.id = B.id WHERE B.v > 10 5. A FULL JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 = A INNER JOIN B ON A.id = B.id WHERE A.v > 10 AND B.v > 10 Tests: * Update the baseline plan Tests * Add some plan tests * Ran the full set of verifications in Impala Public Jenkins Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test M testdata/workloads/functional-planner/queries/PlannerTest/card-outer-join.test M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/implicit-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/joins.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test M testdata/workloads/functional-planner/queries/PlannerTest/nested-loop-join.test M testdata/workloads/functional-planner/queries/PlannerTest/outer-joins.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test 21 files changed, 1,544 insertions(+), 968 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/16266/3 -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 3 Gerrit-Owner: Xianqing He
[Impala-ASF-CR] IMPALA-9990: Support SET OWNER for Kudu tables
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/16273 ) Change subject: IMPALA-9990: Support SET OWNER for Kudu tables .. Patch Set 1: (2 comments) > Patch Set 1: > > Hi all, please review my patch for supporting SET OWNER for Kudu tables in > Impala. > > This patch does not consider the cases in which a user is creating a new Kudu > table from scratch or creating a new Kudu table based on another table > (CTAS), since according to my investigation at IMPALA-9990, Impala indeed > passes the name of the logged in user instead of "impala" to Kudu when a Kudu > table is created. > > Let me know if you have any comment or suggestion. Thanks! http://gerrit.cloudera.org:8080/#/c/16273/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test File testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test: http://gerrit.cloudera.org:8080/#/c/16273/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test@51 PS1, Line 51: alter table simple set owner user non_owner is it possible to check if the owner of "simple" is actually "non_owner"? If yes, could you also add such tests for "create table"? http://gerrit.cloudera.org:8080/#/c/16273/1/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test File testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test: http://gerrit.cloudera.org:8080/#/c/16273/1/testdata/workloads/functional-query/queries/QueryTest/kudu_hms_alter.test@53 PS1, Line 53: 'Owner has been altere.' nit: typo in altered. Is it normal that this test didn't fail? -- To view, visit http://gerrit.cloudera.org:8080/16273 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29d641efc8db314964bc5ee9828a86d4a44ae95c Gerrit-Change-Number: 16273 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 03 Aug 2020 09:42:21 + Gerrit-HasComments: Yes
