[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5877/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Sat, 25 Apr 2020 05:07:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu Defined the BloomFilter class as the wrapper of Kudu BlockBloomFilter, which build runtime bloom filter in Kudu BlockBloomFilter APIs with FastHash as default hash algorithm. Removed the duplicated functions from BloomFillter class. Pushed down bloom filter to Kudu through Kudu clinet API. Added a new query option to set enabled runtime filter types. By default, both bloom filter and min-max filter will be enabled for Kudu. Added new test cases in PlannerTest and end-end runtime_filters test for pushing down bloom filter to Kudu. Testing: Passed end-end runtime filter tests. Passed frontend Planner tests. Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 --- M be/CMakeLists.txt M be/src/codegen/gen_ir_descriptions.py M be/src/exec/filter-context.cc M be/src/exec/kudu-scanner.cc M be/src/runtime/raw-value-ir.cc M be/src/runtime/raw-value.h M be/src/runtime/raw-value.inline.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter-ir.cc M be/src/runtime/runtime-filter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h A be/src/util/bloom-filter.inline.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/hash-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/kudu-update.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test A testdata/workloads/functional-query/queries/QueryTest/all_runtime_filters.test M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test M tests/query_test/test_runtime_filters.py 33 files changed, 1,445 insertions(+), 522 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/15683/7 -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 25 Apr 2020 04:33:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download netty 3.10.5 and netty-all 4.1.44 (and earlier) have known security issues. Tested by including netty and netty-all in banned dependencies and then ensuring tests still pass on jenkins.impala.io. Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Reviewed-on: http://gerrit.cloudera.org:8080/15761 Reviewed-by: David Knupp Tested-by: Impala Public Jenkins --- M fe/pom.xml 1 file changed, 38 insertions(+), 0 deletions(-) Approvals: David Knupp: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 8 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Filter out "Checksum validation failed" messages during the maven build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15775 ) Change subject: Filter out "Checksum validation failed" messages during the maven build .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15775 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19afbd157533e52ef3157730c7ec5159241749bc Gerrit-Change-Number: 15775 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 25 Apr 2020 02:39:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9539: Enable CNF rewrites by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15807 ) Change subject: IMPALA-9539: Enable CNF rewrites by default .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5709/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 Gerrit-Change-Number: 15807 Gerrit-PatchSet: 2 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 25 Apr 2020 02:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9539: Enable CNF rewrites by default
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15807 ) Change subject: IMPALA-9539: Enable CNF rewrites by default .. Patch Set 2: Code-Review+2 LGTM. Thanks for the work on CNF rewrites! -- To view, visit http://gerrit.cloudera.org:8080/15807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 Gerrit-Change-Number: 15807 Gerrit-PatchSet: 2 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 25 Apr 2020 02:00:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9539: Enable CNF rewrites by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15807 ) Change subject: IMPALA-9539: Enable CNF rewrites by default .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5876/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 Gerrit-Change-Number: 15807 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 25 Apr 2020 01:51:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9539: Enable CNF rewrites by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15807 ) Change subject: IMPALA-9539: Enable CNF rewrites by default .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5708/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 Gerrit-Change-Number: 15807 Gerrit-PatchSet: 2 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 25 Apr 2020 01:29:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9539: Enable CNF rewrites by default
Aman Sinha has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15807 Change subject: IMPALA-9539: Enable CNF rewrites by default .. IMPALA-9539: Enable CNF rewrites by default This patch enables the conjunctive normal form rewrites by default by setting enable_cnf_rewrites to true. Functional and performance testing with this flag set to true did not uncover any regressions and showed significant performance gains for queries with disjunctions in the tpch and tpcds suites. Testing: - Updated the PlannerTest tests with plan changes in various test suites. Removed previously added tpch tests which were explicitly setting this flag to true. - I had previously added a test in convert-to-cnf.test with enable_cnf_rewrites=false, so I did not add any new tests with this flag disabled. Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 --- M common/thrift/ImpalaInternalService.thrift M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test 8 files changed, 499 insertions(+), 660 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/15807/1 -- To view, visit http://gerrit.cloudera.org:8080/15807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4dde86e092c61d71ddf9081f768072ced470b589 Gerrit-Change-Number: 15807 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5875/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 25 Apr 2020 00:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5707/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 23:25:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. Patch Set 7: Code-Review+2 Rebased and updated commit msg. Carrying Tim's +2. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 23:25:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download
Hello Tamas Mate, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15761 to look at the new patch set (#7). Change subject: IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download .. IMPALA-9648: Exclude netty and netty-all from hadoop-hdfs mvn download netty 3.10.5 and netty-all 4.1.44 (and earlier) have known security issues. Tested by including netty and netty-all in banned dependencies and then ensuring tests still pass on jenkins.impala.io. Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 --- M fe/pom.xml 1 file changed, 38 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/15761/7 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 7 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. IMPALA-9642: Set USE_CDP_HIVE to true by default Sets the default value for USE_CDP_HIVE to true instead of false. This means that by default all asf builds and tests will run with USE_CDP_HIVE=true. This also modifies build-all-flag-combinations.sh to use USE_CDP_HIVE=true for all builds except one sanity check for USE_CDP_HIVE=false. This is the reverse of the current behavior. The USE_CDP_HIVE=false build can be removed in a subsequent change. Testing: * Ran exhaustive tests Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Reviewed-on: http://gerrit.cloudera.org:8080/15710 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh 2 files changed, 5 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 22:48:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 22:15:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Filter out "Checksum validation failed" messages during the maven build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15775 ) Change subject: Filter out "Checksum validation failed" messages during the maven build .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5706/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15775 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19afbd157533e52ef3157730c7ec5159241749bc Gerrit-Change-Number: 15775 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 22:12:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 22:06:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 21:43:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. IMPALA-9609: Minimize Frontend activity in executor only impalads Until now the Frontend started fully regardless of flag is_coordinator, e.g. created connections to the HMS, which is both error prone and can DoS the metastore. (note that even coordinators started to connect to HMS only in the recent past, related to local catalog mode and ACID transactions) Executor only impalads still need a JVM as queries can contain java calls (HDFS/Hbase API calls, Hive UDFs), but most of the JNI API provided by JniFrontend shouldn't be called by executors. It seems that the whole Frontend object is needed only by coordinators. Testing: - generally executor only mode doesn't seem to be well covered - ran test_coordinators.py which has some tests with executor only impalads - added new test for HBase tables (Hive UDFs and HDFS were already covered) Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Reviewed-on: http://gerrit.cloudera.org:8080/15793 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/exec-env.cc M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M tests/custom_cluster/test_coordinators.py 4 files changed, 63 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 16: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 20:56:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. IMPALA-9176: shared null-aware anti-join build This switches null-aware anti-join (NAAJ) to use shared join builds with mt_dop > 0. To support this, we make all access to the join build data structures from the probe read-only. NAAJ requires iterating over rows from build partitions at various steps in the algorithm and before this patch this was not thread-safe. We avoided that problem by having a separate builder for each join node and duplicating the data. The main challenge was iteration over null_aware_partition()->build_rows() from the probe side, because it uses an embedded iterator in the stream so was not thread-safe (since each thread would be trying to use the same iterator). The solution is to extend BufferedTupleStream to allow multiple read iterators into a pinned, read-only, stream. Each probe thread can then iterate over the stream independently with no thread safety issues. With BufferedTupleStream changes, I partially abstracted ReadIterator more from the rest of BufferedTupleStream, but decided not to completely refactor so that this patchset didn't cause excessive churn. I.e. much BufferedTupleStream code still accesses internal fields of ReadIterator. Fix a pre-existing bug in grouping-aggregator where Spill() hit a DCHECK because the hash table was destroyed unnecessarily when it hit an OOM. This was flushed out by the parameter change in test_spilling. Testing: Add test to buffered-tuple-stream-test for multiple readers to BTS. Tweaked test_spilling_naaj_no_deny_reservation to have a smaller minimum reservation, required to keep the test passing with the new, lower, memory requirement. Updated a TPC-H planner test where resource requirements slightly decreased for the NAAJ. Ran the naaj tests in test_spilling.py with TSAN enabled, confirmed no data races. Ran exhaustive tests, which passed after fixing IMPALA-9611. Ran core tests with ASAN. Ran backend tests with TSAN. Perf: I ran this query that exercises EvaluateNullProbe() heavily. select l_orderkey, l_partkey, l_suppkey, l_linenumber from tpch30_parquet.lineitem where l_suppkey = 4162 and l_shipmode = 'AIR' and l_returnflag = 'A' and l_shipdate > '1993-01-01' and if(l_orderkey > 550, NULL, l_orderkey) not in ( select if(o_orderkey % 2 = 0, NULL, o_orderkey + 1) from orders where l_orderkey = o_orderkey) order by 1,2,3,4; It went from ~13s to ~11s running on a single impalad with this change, because of the inlining of CreateOutputRow() and EvalConjuncts(). I also ran TPC-H SF 30 on Parquet with mt_dop=4, and there was no change in performance. Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Reviewed-on: http://gerrit.cloudera.org:8080/15612 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h A be/src/exec/blocking-join-node.inline.h M be/src/exec/data-source-scan-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h A be/src/exec/exec-node.inline.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hdfs-avro-scanner-ir.cc M be/src/exec/hdfs-columnar-scanner-ir.cc M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-rcfile-scanner.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scanner.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node-ir.cc M be/src/exec/unnest-node.cc M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/buffered-tuple-stream.inline.h M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-query/queries/QueryTest/spilling-naaj-no-deny-reservation.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test M
[Impala-ASF-CR] Filter out "Checksum validation failed" messages during the maven build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15775 ) Change subject: Filter out "Checksum validation failed" messages during the maven build .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5874/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15775 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19afbd157533e52ef3157730c7ec5159241749bc Gerrit-Change-Number: 15775 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 20:36:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 20:02:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml fluent-hc has been flagged in CVE-2014-3577 and CVE-2015-5262, and doesn't seem to be a required component for us. Tested first by confirming that the following line no longer shows up in the output of mvn dependency:tree after the change: +- org.apache.httpcomponents:fluent-hc:jar:4.3.2:compile Then confirmed that upstream tests on jenkins.impala.io pass. Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Reviewed-on: http://gerrit.cloudera.org:8080/15760 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/pom.xml 1 file changed, 7 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Filter out "Checksum validation failed" messages during the maven build
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15775 to look at the new patch set (#2). Change subject: Filter out "Checksum validation failed" messages during the maven build .. Filter out "Checksum validation failed" messages during the maven build Some Impala dependencies come from repositories that don't have checksums available. During the build, this produces a large number of messages like: [WARNING] Checksum validation failed, no checksums available from the repository for ... or: [WARNING] Checksum validation failed, could not read expected checksum ... These messages are not very useful, and they make it harder to search the console output for failed tests. This filters them out of the maven output. Differet versions of maven structure the messsages differently, so this filters all the "Checksum validation failed" messages that happen at WARNING level. Testing: - Ran core tests, verified the messages are gone Change-Id: I19afbd157533e52ef3157730c7ec5159241749bc --- M bin/mvn-quiet.sh 1 file changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/15775/2 -- To view, visit http://gerrit.cloudera.org:8080/15775 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I19afbd157533e52ef3157730c7ec5159241749bc Gerrit-Change-Number: 15775 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:53:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5705/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:53:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5873/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:42:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9097: Don't require minicluster for backend tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15641 ) Change subject: IMPALA-9097: Don't require minicluster for backend tests .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f1b1385853fb23df28d24d38761237e6e5c97a7 Gerrit-Change-Number: 15641 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 24 Apr 2020 17:40:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 5: Code-Review+2 I think I had failed to appreciate the interaction between the enforcer rules and compile/test/runtime dependencies - I think it might've been failing because of a test-time dependency. Anyway I think this will solve our immediate problem. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:35:13 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Fix link text for Impala 3.4
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15803 ) Change subject: Fix link text for Impala 3.4 .. Patch Set 1: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Icebd74cfaff25adcf6da674bfd21f87286713dc1 Gerrit-Change-Number: 15803 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 24 Apr 2020 17:31:38 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Fix link text for Impala 3.4
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15803 ) Change subject: Fix link text for Impala 3.4 .. Fix link text for Impala 3.4 Change-Id: Icebd74cfaff25adcf6da674bfd21f87286713dc1 Reviewed-on: http://gerrit.cloudera.org:8080/15803 Reviewed-by: Joe McDonnell Tested-by: Joe McDonnell --- M downloads.html 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: Icebd74cfaff25adcf6da674bfd21f87286713dc1 Gerrit-Change-Number: 15803 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR](asf-site) Fix link text for Impala 3.4
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15803 Change subject: Fix link text for Impala 3.4 .. Fix link text for Impala 3.4 Change-Id: Icebd74cfaff25adcf6da674bfd21f87286713dc1 --- M downloads.html 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/15803/1 -- To view, visit http://gerrit.cloudera.org:8080/15803 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: Icebd74cfaff25adcf6da674bfd21f87286713dc1 Gerrit-Change-Number: 15803 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.4 docs to be published
Joe McDonnell has removed a vote on this change. Change subject: [DOCS] Impala 3.4 docs to be published .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/15750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: deleteVote Gerrit-Change-Id: I65343187983b1f0f9833055598b4acf256f770da Gerrit-Change-Number: 15750 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kristine Hahn Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](asf-site) Add 3.4.0 change log
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15669 ) Change subject: Add 3.4.0 change log .. Add 3.4.0 change log Change-Id: If0e0697483ce71226191dee3f21aa54492feb6d1 Reviewed-on: http://gerrit.cloudera.org:8080/15669 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- A docs/changelog-3.4.html M impala-docs.html 2 files changed, 698 insertions(+), 5 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/15669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: If0e0697483ce71226191dee3f21aa54492feb6d1 Gerrit-Change-Number: 15669 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.4 docs to be published
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15750 ) Change subject: [DOCS] Impala 3.4 docs to be published .. Patch Set 2: Verified+1 The regular check job does not handle this repo properly, so the -1 is invalid. Going forward with this. -- To view, visit http://gerrit.cloudera.org:8080/15750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I65343187983b1f0f9833055598b4acf256f770da Gerrit-Change-Number: 15750 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kristine Hahn Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:24:57 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Add 3.4.0 change log
Joe McDonnell has removed a vote on this change. Change subject: Add 3.4.0 change log .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/15669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: deleteVote Gerrit-Change-Id: If0e0697483ce71226191dee3f21aa54492feb6d1 Gerrit-Change-Number: 15669 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](asf-site) Update download links for release 3.4.0
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15667 ) Change subject: Update download links for release 3.4.0 .. Update download links for release 3.4.0 The Apache Infra team deprecated the existing download point https://www.apache.org/dist for the new https://downloads.apache.org. This patch uses the new location for the 3.4.0 release. Change-Id: I314662221b50f1594b57817d2ae61a264e7a3866 Reviewed-on: http://gerrit.cloudera.org:8080/15667 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- M downloads.html 1 file changed, 13 insertions(+), 4 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/15667 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: I314662221b50f1594b57817d2ae61a264e7a3866 Gerrit-Change-Number: 15667 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](asf-site) Add 3.4.0 change log
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15669 ) Change subject: Add 3.4.0 change log .. Patch Set 4: Verified+1 The verify job doesn't handle this, removing that vote and verifying. -- To view, visit http://gerrit.cloudera.org:8080/15669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: If0e0697483ce71226191dee3f21aa54492feb6d1 Gerrit-Change-Number: 15669 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:25:26 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.4 docs to be published
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15750 ) Change subject: [DOCS] Impala 3.4 docs to be published .. [DOCS] Impala 3.4 docs to be published - Add docs from Impala branch-3.4.0 - hash 9f1c31c Change-Id: I65343187983b1f0f9833055598b4acf256f770da Reviewed-on: http://gerrit.cloudera.org:8080/15750 Reviewed-by: Tim Armstrong Tested-by: Joe McDonnell --- M docs/build/html/index.html M docs/build/html/topics/impala_abort_on_error.html M docs/build/html/topics/impala_adls.html M docs/build/html/topics/impala_admin.html M docs/build/html/topics/impala_admission.html M docs/build/html/topics/impala_admission_config.html M docs/build/html/topics/impala_aggregate_functions.html M docs/build/html/topics/impala_aliases.html M docs/build/html/topics/impala_allow_erasure_coded_files.html M docs/build/html/topics/impala_allow_unsupported_formats.html M docs/build/html/topics/impala_alter_database.html M docs/build/html/topics/impala_alter_table.html M docs/build/html/topics/impala_alter_view.html M docs/build/html/topics/impala_analytic_functions.html M docs/build/html/topics/impala_appx_count_distinct.html M docs/build/html/topics/impala_appx_median.html M docs/build/html/topics/impala_array.html M docs/build/html/topics/impala_auditing.html M docs/build/html/topics/impala_authentication.html M docs/build/html/topics/impala_authorization.html M docs/build/html/topics/impala_avg.html M docs/build/html/topics/impala_avro.html M docs/build/html/topics/impala_batch_size.html M docs/build/html/topics/impala_bigint.html M docs/build/html/topics/impala_bit_functions.html M docs/build/html/topics/impala_boolean.html M docs/build/html/topics/impala_breakpad.html A docs/build/html/topics/impala_broadcast_bytes_limit.html M docs/build/html/topics/impala_buffer_pool_limit.html M docs/build/html/topics/impala_char.html M docs/build/html/topics/impala_client.html M docs/build/html/topics/impala_comment.html M docs/build/html/topics/impala_comments.html M docs/build/html/topics/impala_complex_types.html M docs/build/html/topics/impala_components.html M docs/build/html/topics/impala_compression_codec.html M docs/build/html/topics/impala_compute_stats.html M docs/build/html/topics/impala_compute_stats_min_sample_size.html M docs/build/html/topics/impala_concepts.html M docs/build/html/topics/impala_conditional_functions.html M docs/build/html/topics/impala_config.html M docs/build/html/topics/impala_config_options.html M docs/build/html/topics/impala_config_performance.html M docs/build/html/topics/impala_connecting.html M docs/build/html/topics/impala_conversion_functions.html M docs/build/html/topics/impala_count.html M docs/build/html/topics/impala_create_database.html M docs/build/html/topics/impala_create_function.html M docs/build/html/topics/impala_create_role.html M docs/build/html/topics/impala_create_table.html M docs/build/html/topics/impala_create_view.html M docs/build/html/topics/impala_custom_timezones.html M docs/build/html/topics/impala_data_cache.html M docs/build/html/topics/impala_databases.html M docs/build/html/topics/impala_datatypes.html M docs/build/html/topics/impala_date.html M docs/build/html/topics/impala_datetime_functions.html M docs/build/html/topics/impala_ddl.html M docs/build/html/topics/impala_debug_action.html M docs/build/html/topics/impala_decimal.html M docs/build/html/topics/impala_decimal_v2.html M docs/build/html/topics/impala_dedicated_coordinator.html M docs/build/html/topics/impala_default_file_format.html M docs/build/html/topics/impala_default_hints_insert_statement.html M docs/build/html/topics/impala_default_join_distribution_mode.html M docs/build/html/topics/impala_default_spillable_buffer_size.html M docs/build/html/topics/impala_default_transactional_type.html M docs/build/html/topics/impala_delegation.html M docs/build/html/topics/impala_delete.html M docs/build/html/topics/impala_describe.html M docs/build/html/topics/impala_development.html M docs/build/html/topics/impala_disable_codegen.html M docs/build/html/topics/impala_disable_codegen_rows_threshold.html A docs/build/html/topics/impala_disable_hbase_num_rows_estimate.html M docs/build/html/topics/impala_disable_row_runtime_filtering.html M docs/build/html/topics/impala_disable_streaming_preaggregations.html M docs/build/html/topics/impala_disable_unsafe_spills.html M docs/build/html/topics/impala_disk_space.html M docs/build/html/topics/impala_distinct.html M docs/build/html/topics/impala_dml.html M docs/build/html/topics/impala_double.html M docs/build/html/topics/impala_drop_database.html M docs/build/html/topics/impala_drop_function.html M docs/build/html/topics/impala_drop_role.html M docs/build/html/topics/impala_drop_stats.html M docs/build/html/topics/impala_drop_table.html M docs/build/html/topics/impala_drop_view.html M docs/build/html/topics/impala_enable_expr_rewrites.html M
[Impala-ASF-CR](asf-site) Update download links for release 3.4.0
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15667 ) Change subject: Update download links for release 3.4.0 .. Patch Set 1: Verified+1 Verified that the download links are accessible -- To view, visit http://gerrit.cloudera.org:8080/15667 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I314662221b50f1594b57817d2ae61a264e7a3866 Gerrit-Change-Number: 15667 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:22:45 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.4 docs to be published
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15750 ) Change subject: [DOCS] Impala 3.4 docs to be published .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I65343187983b1f0f9833055598b4acf256f770da Gerrit-Change-Number: 15750 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kristine Hahn Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:18:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5872/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:18:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 17:17:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5703/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:57:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9642: Set USE CDP HIVE to true by default
Joe McDonnell has uploaded a new patch set (#3) to the change originally created by Sahil Takiar. ( http://gerrit.cloudera.org:8080/15710 ) Change subject: IMPALA-9642: Set USE_CDP_HIVE to true by default .. IMPALA-9642: Set USE_CDP_HIVE to true by default Sets the default value for USE_CDP_HIVE to true instead of false. This means that by default all asf builds and tests will run with USE_CDP_HIVE=true. This also modifies build-all-flag-combinations.sh to use USE_CDP_HIVE=true for all builds except one sanity check for USE_CDP_HIVE=false. This is the reverse of the current behavior. The USE_CDP_HIVE=false build can be removed in a subsequent change. Testing: * Ran exhaustive tests Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 --- M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh 2 files changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/15710/3 -- To view, visit http://gerrit.cloudera.org:8080/15710 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8549c7065de7eecf74f7437fe9d1ff2bf1bfe9f3 Gerrit-Change-Number: 15710 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 3: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/15793/3/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/15793/3/be/src/runtime/exec-env.cc@287 PS3, Line 287: // TODO: Consider removing AdmissionController from executor only impalads. We do actually need this on executors at the moment because they add the memory used locally to the admission control updates - see UpdatePoolStats() -> AddPoolUpdates() -> UpdateMemTrackerStats(). It could be refactored but for now it's harmless. -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:52:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) Add 3.4.0 change log
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15669 ) Change subject: Add 3.4.0 change log .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: If0e0697483ce71226191dee3f21aa54492feb6d1 Gerrit-Change-Number: 15669 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:46:01 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) Update download links for release 3.4.0
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15667 ) Change subject: Update download links for release 3.4.0 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15667 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I314662221b50f1594b57817d2ae61a264e7a3866 Gerrit-Change-Number: 15667 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:43:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15799 ) Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15799/1/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py File testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py: http://gerrit.cloudera.org:8080/#/c/15799/1/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py@114 PS1, Line 114: use_cdp_components: > Does setting USE_CDP_HIVE=true also change Hadoop version? Yes, it ended up controlling all the components, not just Hive. We need to rename it or get rid of that environment variable. -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:44:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15799 ) Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. IMPALA-9677: Fix frontend tests using a non-existent S3 bucket With HADOOP-16711, Hadoop added extra validation during the initialization of S3AFileSystem that verified that the caller had permissions on the S3 bucket specified. Some frontend tests use non-existent S3 buckets in URIs to check analysis behavior. These started to fail with the new validation. This changes the core-site.xml configuration to disable the new validation by setting fs.s3a.bucket.probe=1. This is equivalent to the old behavior, and it can now run the frontend tests without AWS credentials. Testing: - Hand tested the failing tests (AnalyzeDDLTest, ExplainTest, PlannerTest) - Ran core job on USE_CDP_HIVE=true and USE_CDP_HIVE=false Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Reviewed-on: http://gerrit.cloudera.org:8080/15799 Tested-by: Impala Public Jenkins Reviewed-by: Anurag Mantripragada Reviewed-by: Tim Armstrong --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 1 file changed, 8 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Anurag Mantripragada: Looks good to me, but someone else must approve Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15799 ) Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. Patch Set 1: Code-Review+1 (1 comment) Looks good to me. http://gerrit.cloudera.org:8080/#/c/15799/1/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py File testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py: http://gerrit.cloudera.org:8080/#/c/15799/1/testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py@114 PS1, Line 114: use_cdp_components: Does setting USE_CDP_HIVE=true also change Hadoop version? -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 16:42:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15799 ) Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:43:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5701/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:36:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 15: A different test started failing because of the parameter change. I reduced the scope of my original test change so that max_row_size=64k only applied to the specific query that needed it. Carrying +2 -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:35:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:36:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5702/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:36:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15612 to look at the new patch set (#15). Change subject: IMPALA-9176: shared null-aware anti-join build .. IMPALA-9176: shared null-aware anti-join build This switches null-aware anti-join (NAAJ) to use shared join builds with mt_dop > 0. To support this, we make all access to the join build data structures from the probe read-only. NAAJ requires iterating over rows from build partitions at various steps in the algorithm and before this patch this was not thread-safe. We avoided that problem by having a separate builder for each join node and duplicating the data. The main challenge was iteration over null_aware_partition()->build_rows() from the probe side, because it uses an embedded iterator in the stream so was not thread-safe (since each thread would be trying to use the same iterator). The solution is to extend BufferedTupleStream to allow multiple read iterators into a pinned, read-only, stream. Each probe thread can then iterate over the stream independently with no thread safety issues. With BufferedTupleStream changes, I partially abstracted ReadIterator more from the rest of BufferedTupleStream, but decided not to completely refactor so that this patchset didn't cause excessive churn. I.e. much BufferedTupleStream code still accesses internal fields of ReadIterator. Fix a pre-existing bug in grouping-aggregator where Spill() hit a DCHECK because the hash table was destroyed unnecessarily when it hit an OOM. This was flushed out by the parameter change in test_spilling. Testing: Add test to buffered-tuple-stream-test for multiple readers to BTS. Tweaked test_spilling_naaj_no_deny_reservation to have a smaller minimum reservation, required to keep the test passing with the new, lower, memory requirement. Updated a TPC-H planner test where resource requirements slightly decreased for the NAAJ. Ran the naaj tests in test_spilling.py with TSAN enabled, confirmed no data races. Ran exhaustive tests, which passed after fixing IMPALA-9611. Ran core tests with ASAN. Ran backend tests with TSAN. Perf: I ran this query that exercises EvaluateNullProbe() heavily. select l_orderkey, l_partkey, l_suppkey, l_linenumber from tpch30_parquet.lineitem where l_suppkey = 4162 and l_shipmode = 'AIR' and l_returnflag = 'A' and l_shipdate > '1993-01-01' and if(l_orderkey > 550, NULL, l_orderkey) not in ( select if(o_orderkey % 2 = 0, NULL, o_orderkey + 1) from orders where l_orderkey = o_orderkey) order by 1,2,3,4; It went from ~13s to ~11s running on a single impalad with this change, because of the inlining of CreateOutputRow() and EvalConjuncts(). I also ran TPC-H SF 30 on Parquet with mt_dop=4, and there was no change in performance. Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h A be/src/exec/blocking-join-node.inline.h M be/src/exec/data-source-scan-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h A be/src/exec/exec-node.inline.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hdfs-avro-scanner-ir.cc M be/src/exec/hdfs-columnar-scanner-ir.cc M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-rcfile-scanner.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scanner.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node-ir.cc M be/src/exec/unnest-node.cc M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/buffered-tuple-stream.inline.h M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-query/queries/QueryTest/spilling-naaj-no-deny-reservation.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test M tests/query_test/test_spilling.py 46 files changed,
[Impala-ASF-CR] IMPALA-9679: Remove some jars from Docker images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15753 ) Change subject: IMPALA-9679: Remove some jars from Docker images .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15753/5/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/15753/5/fe/pom.xml@1309 PS5, Line 1309: : : org.apache.knox : gateway-shell : I looked around, and I think gateway-cloud-bindings needs some classes from gateway-shell. Maybe skip this exclusion? We can circle back if we can prove we don't need it. -- To view, visit http://gerrit.cloudera.org:8080/15753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 Gerrit-Change-Number: 15753 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 24 Apr 2020 16:25:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 16:26:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. IMPALA-9631: Import HLL functionality from DataSketches This patch imports the functionality needed for HLL approximate algorithm from Apache DataSketches. I decided to copy the necessary files into be/src/thirdparty/datasketches. Note, that the original structure of files was changed during this process as originally hll/ and common/ libraries were both affected but I copied these into the same directory so that Impala can compile them without rewriting the include paths in the files themselves. Also note, that not the whole common/ directory was copied just the files needed for HLL. The git hash of the snapshot I used as a source for the files: a6265b307a03085abe26c20413fdbf7d7a5eaf29 Browse the source files here: https://github.com/apache/incubator-datasketches-cpp Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Reviewed-on: http://gerrit.cloudera.org:8080/15746 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exprs/CMakeLists.txt A be/src/exprs/datasketches-test.cc A be/src/thirdparty/datasketches/AuxHashMap-internal.hpp A be/src/thirdparty/datasketches/AuxHashMap.hpp A be/src/thirdparty/datasketches/CommonUtil.hpp A be/src/thirdparty/datasketches/CompositeInterpolationXTable-internal.hpp A be/src/thirdparty/datasketches/CompositeInterpolationXTable.hpp A be/src/thirdparty/datasketches/CouponHashSet-internal.hpp A be/src/thirdparty/datasketches/CouponHashSet.hpp A be/src/thirdparty/datasketches/CouponList-internal.hpp A be/src/thirdparty/datasketches/CouponList.hpp A be/src/thirdparty/datasketches/CubicInterpolation-internal.hpp A be/src/thirdparty/datasketches/CubicInterpolation.hpp A be/src/thirdparty/datasketches/HarmonicNumbers-internal.hpp A be/src/thirdparty/datasketches/HarmonicNumbers.hpp A be/src/thirdparty/datasketches/Hll4Array-internal.hpp A be/src/thirdparty/datasketches/Hll4Array.hpp A be/src/thirdparty/datasketches/Hll6Array-internal.hpp A be/src/thirdparty/datasketches/Hll6Array.hpp A be/src/thirdparty/datasketches/Hll8Array-internal.hpp A be/src/thirdparty/datasketches/Hll8Array.hpp A be/src/thirdparty/datasketches/HllArray-internal.hpp A be/src/thirdparty/datasketches/HllArray.hpp A be/src/thirdparty/datasketches/HllSketch-internal.hpp A be/src/thirdparty/datasketches/HllSketchImpl-internal.hpp A be/src/thirdparty/datasketches/HllSketchImpl.hpp A be/src/thirdparty/datasketches/HllSketchImplFactory.hpp A be/src/thirdparty/datasketches/HllUnion-internal.hpp A be/src/thirdparty/datasketches/HllUtil.hpp A be/src/thirdparty/datasketches/LICENSE A be/src/thirdparty/datasketches/MurmurHash3.h A be/src/thirdparty/datasketches/README.md A be/src/thirdparty/datasketches/RelativeErrorTables-internal.hpp A be/src/thirdparty/datasketches/RelativeErrorTables.hpp A be/src/thirdparty/datasketches/coupon_iterator-internal.hpp A be/src/thirdparty/datasketches/coupon_iterator.hpp A be/src/thirdparty/datasketches/hll.hpp A be/src/thirdparty/datasketches/hll.private.hpp A be/src/thirdparty/datasketches/inv_pow2_table.hpp M bin/rat_exclude_files.txt M bin/run_clang_tidy.sh 41 files changed, 7,381 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9679: Remove some jars from Docker images
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15753 ) Change subject: IMPALA-9679: Remove some jars from Docker images .. Patch Set 5: Code-Review+1 This all looks good to me. The only reason why I'm not giving a +2 is that my docker-fu is a little weak. -- To view, visit http://gerrit.cloudera.org:8080/15753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 Gerrit-Change-Number: 15753 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 24 Apr 2020 16:16:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5871/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 16:15:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15799 ) Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5870/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 16:10:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 3: Code-Review+1 Changes LGTM. -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:46:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5700/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:43:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:42:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5699/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:37:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Hello Tamas Mate, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15761 to look at the new patch set (#4). Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. IMPALA-9648: Exclude netty from hadoop-hdfs mvn download Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 --- M fe/pom.xml 1 file changed, 38 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/15761/4 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 4 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15761/3/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/15761/3/fe/pom.xml@755 PS3, Line 755: io.netty:netty-all:4.1.42.Final > Can we tighten this up to exclude everything from io.netty? I.e. io.netty:* But note that your suggestion is exactly what I had in patch #2, and it failed. - https://jenkins.impala.io/job/gerrit-verify-dryrun/5692/console - https://jenkins.impala.io/job/all-build-options-ub1604/5596/ Also, see the top comment on the internal mirror of IMPALA-9648. Is there something I need to do to toggle USE_CDP_HIVE when I run the verify job on upstream jenkins? Note that Abhishek also provided some additional details about the CVE that caused this issue. I might try something based on that new info. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:29:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9677: Fix frontend tests using a non-existent S3 bucket
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15799 Change subject: IMPALA-9677: Fix frontend tests using a non-existent S3 bucket .. IMPALA-9677: Fix frontend tests using a non-existent S3 bucket With HADOOP-16711, Hadoop added extra validation during the initialization of S3AFileSystem that verified that the caller had permissions on the S3 bucket specified. Some frontend tests use non-existent S3 buckets in URIs to check analysis behavior. These started to fail with the new validation. This changes the core-site.xml configuration to disable the new validation by setting fs.s3a.bucket.probe=1. This is equivalent to the old behavior, and it can now run the frontend tests without AWS credentials. Testing: - Hand tested the failing tests (AnalyzeDDLTest, ExplainTest, PlannerTest) - Ran core job on USE_CDP_HIVE=true and USE_CDP_HIVE=false Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/15799/1 -- To view, visit http://gerrit.cloudera.org:8080/15799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id61ffbf686f8b7827e7fbf13167cfc1dfc06a325 Gerrit-Change-Number: 15799 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 15:26:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/15793/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java File fe/src/main/java/org/apache/impala/service/JniFrontend.java: http://gerrit.cloudera.org:8080/#/c/15793/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@146 PS2, Line 146: frontend_ = null; > With this set to null, to be safe it would be good to add Preconditions not Added the Preconditions. My idea for making this class nicer was to make all functions needed by executors static and create a JniFrontend in backend only for coordinators. I didn't do this change because I wanted to minimize the size and risks of this patch to make it easily backportable. http://gerrit.cloudera.org:8080/#/c/15793/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@191 PS2, Line 191: ExecutorMembershipSnapshot.update(req); > Removing the indirection makes sense. Can we go further and just not do the Done -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 14:49:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9609: Minimize Frontend activity in executor only impalads
Csaba Ringhofer has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15793 ) Change subject: IMPALA-9609: Minimize Frontend activity in executor only impalads .. IMPALA-9609: Minimize Frontend activity in executor only impalads Until now the Frontend started fully regardless of flag is_coordinator, e.g. created connections to the HMS, which is both error prone and can DoS the metastore. (note that even coordinators started to connect to HMS only in the recent past, related to local catalog mode and ACID transactions) Executor only impalads still need a JVM as queries can contain java calls (HDFS/Hbase API calls, Hive UDFs), but most of the JNI API provided by JniFrontend shouldn't be called by executors. It seems that the whole Frontend object is needed only by coordinators. Testing: - generally executor only mode doesn't seem to be well covered - ran test_coordinators.py which has some tests with executor only impalads - added new test for HBase tables (Hive UDFs and HDFS were already covered) Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 --- M be/src/runtime/exec-env.cc M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M tests/custom_cluster/test_coordinators.py 4 files changed, 63 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/15793/3 -- To view, visit http://gerrit.cloudera.org:8080/15793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4627e5e3520175153cb49e24fd480815dfefdae1 Gerrit-Change-Number: 15793 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15797 ) Change subject: IMPALA-9688: Support create iceberg table by impala .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5868/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 14:11:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15797 to look at the new patch set (#5). Change subject: IMPALA-9688: Support create iceberg table by impala .. IMPALA-9688: Support create iceberg table by impala This patch mainly realizes the creation of iceberg table through impala, we can use the following sql to create a new iceberg table: create table iceberg_test( level string, event_time string, message string) partition by spec( level identity, event_time identity ) stored as iceberg; The 'identity' mean the partition type in iceberg, we can alse use 'show create table iceberg_test' to dispaly table schema, and use 'show partition iceberg_test' to display partition column info. By the way, parititon column must be the source column. Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 --- M bin/impala-config.sh M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/JniCatalog.thrift M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDataLayout.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java A fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/jflex/sql-scanner.flex M impala-parent/pom.xml 26 files changed, 1,247 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/15797/5 -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9097: Don't require minicluster for backend tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15641 ) Change subject: IMPALA-9097: Don't require minicluster for backend tests .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5698/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f1b1385853fb23df28d24d38761237e6e5c97a7 Gerrit-Change-Number: 15641 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 24 Apr 2020 13:18:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15797 ) Change subject: IMPALA-9688: Support create iceberg table by impala .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5867/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 13:17:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
wangsheng has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/15797 ) Change subject: IMPALA-9688: Support create iceberg table by impala .. IMPALA-9688: Support create iceberg table by impala This patch mainly realizes the creation of iceberg table through impala, we can use the following sql to create a new iceberg table: create table iceberg_test( level string, event_time string, message string) partition by spec( level identity, event_time identity ) stored as iceberg; The 'identity' mean the partition type in iceberg, we can alse use 'show create table iceberg_test' to dispaly table schema, and use 'show partition iceberg_test' to display partition column info. By the way, parititon column must be the source column. Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 --- M bin/impala-config.sh M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/JniCatalog.thrift M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDataLayout.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java A fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/jflex/sql-scanner.flex M impala-parent/pom.xml 26 files changed, 1,246 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/15797/4 -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15797 ) Change subject: IMPALA-9688: Support create iceberg table by impala .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5866/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 3 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 12:14:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5697/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 12:06:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 24 Apr 2020 12:06:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9688: Support create iceberg table by impala
wangsheng has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15797 ) Change subject: IMPALA-9688: Support create iceberg table by impala .. IMPALA-9688: Support create iceberg table by impala This patch mainly realizes the creation of iceberg table through impala, we can use the following sql to create a new iceberg table: create table iceberg_test( level string, event_time string, message string) partition by spec( level identity, event_time identity ) stored as iceberg; The 'identity' mean the partition type in iceberg, we can alse use 'show create table iceberg_test' to dispaly table schema, and use 'show partition iceberg_test' to display partition column info. By the way, parititon column must be the source column. Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 --- M bin/impala-config.sh M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/JniCatalog.thrift M fe/pom.xml M fe/src/main/cup/sql-parser.cup D fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java A fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDataLayout.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java A fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java A fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/jflex/sql-scanner.flex M impala-parent/pom.xml 27 files changed, 1,246 insertions(+), 99 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/15797/3 -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 3 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9649: Add shiro* to the banned dependency maven plugin
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15796 ) Change subject: IMPALA-9649: Add shiro* to the banned dependency maven plugin .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5865/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15796 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f9994bf81c1d2e025a03925e8eccb147c34d66e Gerrit-Change-Number: 15796 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 07:32:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 3: (1 comment) LGTM basically, one q about making it stricter, but I'm OK with this as-is. http://gerrit.cloudera.org:8080/#/c/15761/3/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/15761/3/fe/pom.xml@755 PS3, Line 755: io.netty:netty-all:4.1.42.Final Can we tighten this up to exclude everything from io.netty? I.e. io.netty:* . It seems like we're excluding it all anyway (at least I don't see it when I run (cd fe && mvn dependency:tree -Dscope=runtime) with USE_CDP_HIVE=true -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 24 Apr 2020 06:57:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9649: Add shiro* to the banned dependency maven plugin
Hello Tim Armstrong, Joe McDonnell, Dinesh Garg, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15796 to look at the new patch set (#3). Change subject: IMPALA-9649: Add shiro* to the banned dependency maven plugin .. IMPALA-9649: Add shiro* to the banned dependency maven plugin The earlier attempt to exclude the shiro-core and shiro-crypto-cipher jars from fe/pom.xml failed to find all instances, and security scans picked them up again. This patch also moves the shiro exclusions from the sentry node to the dependencyManagement node, and excludes the jar from the following: - sentry-core-common - sentry-provider-db - sentry-provider-file - sentry-provider-cache Furthermore, to avoid compilation errors related to the absense of shiro, it was necessary to remove delete the TestSentryResourceAuthorizationProvider class, and any tests that referenced it. Since Sentry is not being used any longer, this shouldn't be an issue. Tested by running build, which didn't fail from banned dependency plugin, as well running the standard set of tests on jenkins.impala.io. Change-Id: I9f9994bf81c1d2e025a03925e8eccb147c34d66e --- M fe/pom.xml M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java D fe/src/test/java/org/apache/impala/testutil/TestSentryResourceAuthorizationProvider.java 3 files changed, 62 insertions(+), 163 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/15796/3 -- To view, visit http://gerrit.cloudera.org:8080/15796 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9f9994bf81c1d2e025a03925e8eccb147c34d66e Gerrit-Change-Number: 15796 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Dinesh Garg (430) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9649: Add shiro* to the banned dependency maven plugin
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15796 Change subject: IMPALA-9649: Add shiro* to the banned dependency maven plugin .. IMPALA-9649: Add shiro* to the banned dependency maven plugin The earlier attempt to excludethe shiro-core and shiro-crypto-cipher jars from fe/pom.xml failed to find all instances, and security scans picked them up again. This patch also moves the shiro exclusions from the sentry node to the dependencyManagement node, and excludes the jar from the following: - sentry-core-common - sentry-provider-db - sentry-provider-file - sentry-provider-cache Furthermore, to avoid compilation errors related to the absense of shiro, it was necessary to remove delete the TestSentryResourceAuthorizationProvider class, and any tests that referenced it. Since Sentry is not being used any longer, this shouldn't be an issue. Tested by running build, which didn't fail from banned dependency plugin, as well running the standard set of tests on jenkins.impala.io. Change-Id: I9f9994bf81c1d2e025a03925e8eccb147c34d66e --- M fe/pom.xml M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java D fe/src/test/java/org/apache/impala/testutil/TestSentryResourceAuthorizationProvider.java 3 files changed, 62 insertions(+), 163 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/15796/2 -- To view, visit http://gerrit.cloudera.org:8080/15796 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9f9994bf81c1d2e025a03925e8eccb147c34d66e Gerrit-Change-Number: 15796 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins
