[Impala-ASF-CR] IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15676 ) Change subject: IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5747/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 Gerrit-Change-Number: 15676 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 08 Apr 2020 05:30:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5746/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Wed, 08 Apr 2020 05:27:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. Patch Set 5: Code-Review+1 Thanks for addressing my comments. I will let Vihang +2 the change. -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Wed, 08 Apr 2020 05:17:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5748/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 08 Apr 2020 05:08:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/15683/2/be/src/exec/kudu-scanner.cc File be/src/exec/kudu-scanner.cc: http://gerrit.cloudera.org:8080/#/c/15683/2/be/src/exec/kudu-scanner.cc@269 PS2, Line 269: const ColumnType& col_type = ColumnType::FromThrift(target_desc.kudu_col_type); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/15683/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/15683/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@713 PS2, Line 713: // Kudu only supports targeting a single column, not general exprs, so the target line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/15683/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@729 PS2, Line 729: // Kudu only supports targeting a single column, not general exprs, so the target line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/15683/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@732 PS2, Line 732: // Kudu also cannot currently return nulls if a filter is applied, so it does not line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 08 Apr 2020 04:50:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15683 Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu Defined the BloomFilter class as the wrapper of Kudu BlockBloomFilter, which build runtime bloom filter in Kudu BlockBloomFilter APIs with FastHash as default hash algorithm. Removed the duplicated functions from BloomFillter class. Added a query option to set runtime filter scheme for Kudu. Pushed down runtime filters to Kudu through Kudu client APIs. Added new test cases in PlannerTest and end-end runtime_filters test for pushing down bloom filter to Kudu. Testing: Passed end-end runtime filter tests with codegen disabled. Passed frontend Planner tests. Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 --- M be/CMakeLists.txt M be/src/exec/filter-context.cc M be/src/exec/kudu-scanner.cc M be/src/runtime/raw-value.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter-ir.cc M be/src/runtime/runtime-filter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h A be/src/util/bloom-filter.inline.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/hash-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-query-options.test M tests/query_test/test_runtime_filters.py 24 files changed, 623 insertions(+), 360 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/15683/2 -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15676 Change subject: IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter .. IMPALA-3741 [part 1]: Upgraded kudu/util for BloomFilter Ported BlockBloomFilter related source files from Kudu upstream to Impala be/src/kudu/util. Testing: Passed core tests. Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 --- M be/src/kudu/util/CMakeLists.txt M be/src/kudu/util/block_bloom_filter-test.cc M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/block_bloom_filter.h A be/src/kudu/util/block_bloom_filter.proto M be/src/kudu/util/block_bloom_filter_avx2.cc M be/src/kudu/util/hash.proto M be/src/kudu/util/hash_util-test.cc M be/src/kudu/util/hash_util.h M be/src/kudu/util/random.h M be/src/kudu/util/random_util-test.cc M be/src/kudu/util/random_util.h 12 files changed, 732 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/15676/1 -- To view, visit http://gerrit.cloudera.org:8080/15676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifac41ffb3e1742ffb6a969cb1c368d6d93c23357 Gerrit-Change-Number: 15676 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Xiaomeng Zhang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. IMPALA-8632: Add support for self-event detection for insert events In case of INSERT_EVENTS if Impala inserts into a table it causes a refresh to the underlying table/partition. This could be unnecessary when there is only one Impala cluster in the system. We can detect a self-event in such cases when the HMS API to fire a listener event returns the event id. This is used by EventProcessor to ignore the event when it is fetched later in the next polling cycle. Testing: Add testInsertFromImpala() in MetastoreEventsProcessorTest.java to test insert event self-event detection when insert into table and partition. Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M bin/impala-config.sh M common/thrift/BackendGflags.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M tests/custom_cluster/test_event_processing.py 18 files changed, 487 insertions(+), 171 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/15648/5 -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Alice Fan has posted comments on this change. ( http://gerrit.cloudera.org:8080/15674 ) Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 04:42:11 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5745/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 26 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Apr 2020 03:18:29 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5744/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 25 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 08 Apr 2020 03:07:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5743/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 03:02:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15674 ) Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 08 Apr 2020 02:51:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 2: > Patch Set 2: > > I should've asked first time, but were you able to reproduce this at all? > It'd be good to confirm that it doesn't reproduce after the fix. > > Ideally we would add a regression test, but I understand that the regression > test may not be feasible with reasonable effort - I didn't see an obvious way > to do it without special tooling to force the timing, but maybe you have some > ideas. My first thought is running model checking, but that might be overkill. We can do simpler by running two threads. Thread A call RuntimeFilter::WaitForArrival(X). Thread B call RuntimeFilter::Cancel(), runs slightly after thread A. We force the race by injecting small delay within WaitForArrival() loop so that Cancel() signal arrival_cv_ before WaitForArrival() arrived at line arrival_cv_.WaitFor(). Racy execution will see test duration >= X ms. I can reproduce the long waiting bug this way. Question is, how to correctly inject the time delay so that it only happen in test and not in production? -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 02:48:39 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A tests/query_test/test_async_codegen.py M tests/query_test/test_queries.py M tests/query_test/test_query_mem_limit.py 54 files changed, 911 insertions(+), 432 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/26 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 26 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 26: (21 comments) http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py File tests/query_test/test_async_codegen.py: http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@19 PS26, Line 19: from copy import copy flake8: F401 'copy.copy' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@21 PS26, Line 21: from tests.beeswax.impala_beeswax import ImpalaBeeswaxException flake8: F401 'tests.beeswax.impala_beeswax.ImpalaBeeswaxException' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@22 PS26, Line 22: from tests.common.test_dimensions import ( flake8: F401 'tests.common.test_dimensions.create_parquet_dimension' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@22 PS26, Line 22: from tests.common.test_dimensions import ( flake8: F401 'tests.common.test_dimensions.create_avro_snappy_dimension' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@26 PS26, Line 26: from tests.common.impala_cluster import ImpalaCluster flake8: F401 'tests.common.impala_cluster.ImpalaCluster' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@28 PS26, Line 28: from tests.common.skip import SkipIfNotHdfsMinicluster flake8: F401 'tests.common.skip.SkipIfNotHdfsMinicluster' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@29 PS26, Line 29: from tests.common.test_dimensions import ( flake8: F401 'tests.common.test_dimensions.extend_exec_option_dimension' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@32 PS26, Line 32: from tests.common.test_vector import ImpalaTestDimension flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@33 PS26, Line 33: from tests.verifiers.metric_verifier import MetricVerifier flake8: F401 'tests.verifiers.metric_verifier.MetricVerifier' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@36 PS26, Line 36: from RuntimeProfile.ttypes import TRuntimeProfileFormat flake8: F401 'RuntimeProfile.ttypes.TRuntimeProfileFormat' imported but unused http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@38 PS26, Line 38: class TestAsyncCodegen(ImpalaTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@79 PS26, Line 79: ) flake8: E123 closing bracket does not match indentation of opening bracket's line http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@87 PS26, Line 87: = flake8: E712 comparison to False should be 'if cond is False:' or 'if not cond:' http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@93 PS26, Line 93: ) flake8: E123 closing bracket does not match indentation of opening bracket's line http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@106 PS26, Line 106: flake8: E261 at least two spaces before inline comment http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@106 PS26, Line 106: n flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@108 PS26, Line 108: c flake8: F841 local variable 'codegen_start' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@120 PS26, Line 120: a flake8: F631 assertion is always true, perhaps remove parentheses? http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@155 PS26, Line 155: flake8: E261 at least two spaces before inline comment http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@157 PS26, Line 157: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/15105/26/tests/query_test/test_async_codegen.py@158 PS26, Line 158: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 26 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5623/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 02:36:03 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/query_test/test_queries.py M tests/query_test/test_query_mem_limit.py 53 files changed, 745 insertions(+), 429 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/25 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 25 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 8: Hi, Tim, this has been modified as your request. -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 02:23:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
zhaoren...@hotmail.com has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. IMPALA-9590 Resolve error when build tsan and ubsan on arm64 Tsan build will fail on atomicops-internals-x86.cc build, so if on arm64, just don't build it. And Ubsan build should link to libclang_rt.ubsan_standalone's aarch 64 version, not x86 version. Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d --- M be/src/gutil/CMakeLists.txt M be/src/service/CMakeLists.txt 2 files changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/15630/8 -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5742/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 Apr 2020 02:17:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 ) Change subject: IMPALA-9602: Fix case-sensitivity for local catalog .. Patch Set 3: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21 PS3, Line 21:start Impala with local catalog enabled : start-impala-cluster.py : --catalogd_args="--catalog_topic_mode=minimal" : --impalad_args="--use_local_catalog=true" :Create database in lower-case: "CREATE DATABASE db1;" :Run the following a few times (this errors without the patch): :impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE; : CREATE TABLE DB1.ddl_test1 (val string) > That's good to know about the python script which seems great for end-to-en sure. Since the existing unit-test will mostly be doing the same I think its okay for now. http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625 PS3, Line 625: DbCacheKey > I had thought of that but looking at existing code in other places (e.g Cat yeah, I am okay with that. -- To view, visit http://gerrit.cloudera.org:8080/15653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26 Gerrit-Change-Number: 15653 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 Apr 2020 01:56:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15654 ) Change subject: IMPALA-9529: Fix multi-tuple predicates not assigned in column masking .. Patch Set 5: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5616/ FWIW, this failure is flaky and is not related: IMPALA-9453. Thanks for Csaba and Aman's review! -- To view, visit http://gerrit.cloudera.org:8080/15654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3 Gerrit-Change-Number: 15654 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 Apr 2020 01:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/15654 )
Change subject: IMPALA-9529: Fix multi-tuple predicates not assigned in column
masking
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15654/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/15654/3/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1493
PS3, Line 1493: if (node instanceof JoinNode) {
> I'm afraid that doesn't work since JoinNode.init() is calling this method t
Ok, that makes sense. Thanks.
--
To view, visit http://gerrit.cloudera.org:8080/15654
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3
Gerrit-Change-Number: 15654
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 08 Apr 2020 01:46:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5622/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 Apr 2020 01:40:07 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 Apr 2020 01:40:06 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 Apr 2020 01:38:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Bump up CDP_BUILD_NUMBER to 2523282 This change bumps up the CDP_BUILD_NUMBER to 2523282. The new CDP build includes a fix on Hive side (HIVE-23018) which is needed for improving the INSERT event support in Catalog. Note: This CDP build introduces some HMS schema change and hence the metastore in the existing local HMS will need to be recreated using bin/create-test-configuration.sh -create_metastore Testing: 1. Ran core tests for with USE_CDP_HIVE=true. Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 --- M bin/impala-config.sh 1 file changed, 8 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15668/3 -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 ) Change subject: IMPALA-9602: Fix case-sensitivity for local catalog .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21 PS3, Line 21:start Impala with local catalog enabled : start-impala-cluster.py : --catalogd_args="--catalog_topic_mode=minimal" : --impalad_args="--use_local_catalog=true" :Create database in lower-case: "CREATE DATABASE db1;" :Run the following a few times (this errors without the patch): :impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE; : CREATE TABLE DB1.ddl_test1 (val string) > We have a test_local_catalog.py or test_ddl.py in case you want to add this That's good to know about the python script which seems great for end-to-end test and testing race conditions in cache updates. I will keep this in mind for future testing. In this case though, since the problem is simpler, I am thinking the unit test is good enough. Let me know if you think otherwise. http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625 PS3, Line 625: DbCacheKey > May be its better to move toLowerCase() call inside DbCacheKey constructor. I had thought of that but looking at existing code in other places (e.g Catalog.java, CatalogObjectCache.java), the caller explicitly lowercases instead of the constructor doing it. One motivation for it is there's a caseInsensitiveKeys_ flag in CatalogObjectCache to do this conditionally. I am thinking it would be good to maintain similar logic in the local catalog in case in the future we want to do this conditionally for any reason. What do you think ? -- To view, visit http://gerrit.cloudera.org:8080/15653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26 Gerrit-Change-Number: 15653 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 08 Apr 2020 01:31:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15642 ) Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. Patch Set 6: Agree it would be good to sanity check at least one large query. Otherwise this looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/15642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 Gerrit-Change-Number: 15642 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 00:36:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 2: I should've asked first time, but were you able to reproduce this at all? It'd be good to confirm that it doesn't reproduce after the fix. Ideally we would add a regression test, but I understand that the regression test may not be feasible with reasonable effort - I didn't see an obvious way to do it without special tooling to force the timing, but maybe you have some ideas. -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 00:15:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. IMPALA-9565 Remove unused included file mm_malloc.h on ARM Remove unused included file mm_malloc.h on ARM. ARM version gcc don't have mm_malloc.h file. Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Reviewed-on: http://gerrit.cloudera.org:8080/15586 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M be/src/kudu/util/memory/memory.cc M be/src/kudu/util/striped64.cc 2 files changed, 7 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 9: Thanks for the contribution! -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 00:09:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 8: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 08 Apr 2020 00:09:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. Patch Set 4: (15 comments) Thanks for the change, I have added some comments, most of them are style-related and some nits. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@54 PS4, Line 54: import org.apache.impala.catalog.events.NoOpEventProcessor; Nit: Duplicate import, could you remove this? http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@934 PS4, Line 934: Nit: Incorrect indentation. Remove the empty spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@937 PS4, Line 937: Incorrect indentation. Remove the empty spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/Table.java@807 PS4, Line 807: Nit: Incorrect indentation. Remove the empty spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/Table.java@826 PS4, Line 826: Nit: Incorrect indentation. Remove the empty spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java File fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java: http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java@65 PS4, Line 65: eventIds_ Here and everywhere else, if this means "only" insert events, may I suggest using insertEventIds_? http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java@69 PS4, Line 69: @param isInsertEvent if true, return list of versions for in-flight Insert events :* if false, return list of eventIds for in-flight DDL events Should this be the other way around? i.e., return eventIds for in-flight insert events if it is insert events and versions if it is not. Also, indentation is incorrect, please remove the extra spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java@87 PS4, Line 87: Nit: Indentation incorrect, remove extra spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java@107 PS4, Line 107: Nit: Indentation incorrect, remove extra spaces. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@773 PS4, Line 773: // long eventId_test = catalog_.public_eventId.consumeId(); Could you remove this comment? http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@787 PS4, Line 787: /** : * Check self-event for in flight Insert event using eventId : */ Nit: Not sure this is needed since this is an overridden method and its purpose is the same as described in L318. http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@75 PS4, Line 75: *; We should be careful with *. Are you sure we are importing everything from org.apache.impala.catalog? http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4424 PS4, Line 4424: List of all catalog object that was insert into Did you mean "List of all partitions we insert into"? http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4487 PS4, Line 4487: // Firing insert events by making calls to HMS APIs can be slow for tables with : // large number of partitions. > this comment should be updated with the bulk API it should not be a problem I see that MetastoreShim.fireInsertEvent() for hive-2 does not insert in bulk and is still making one RPC per partition. We should keep the earlier asynchronous fireInsertEvent() for hive-2.
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. IMPALA-9611: fix hang when cancelling join builder The error could occur in the following scenario, where thread A is executing a join build fragment and thread B is cancelling the fragment instance. 1. Thread A is in HandoffToProbesAndWait(), reads is_cancelled_ and sees false. 2. Thread B in RuntimeState::Cancel() sets is_cancelled_ = true, acquires cancellation_cvs_lock_, then calls NotifyAll() on the condition variable 3. Thread A calls Wait() on the condition variable, blocks forever because cancellation already happened. The fix is for thread B to acquire the lock that thread A is holding. That prevents the race because #1 and #3 above are in the same critical section and thread B won't be able to signal the condition variable until thread A has released it. Testing: Added metric check to test_failpoints to make it easier to detect hangs caused by those tests in future. Looped test_failpoints.py overnight, which was previously enough to reproduce the failure within a couple of hours. Ran exhaustive tests. Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Reviewed-on: http://gerrit.cloudera.org:8080/15672 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/join-builder.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M tests/failure/test_failpoints.py 4 files changed, 40 insertions(+), 12 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 23:26:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5741/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Tue, 07 Apr 2020 22:23:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15648 )
Change subject: IMPALA-8632: Add support for self-event detection for insert
events
..
Patch Set 4:
(9 comments)
mostly looks good to me. Should be okay once we add the suggestions.
http://gerrit.cloudera.org:8080/#/c/15648/3/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/15648/3/be/src/common/global-flags.cc@254
PS3, Line 254:
: DEFINE_int32(hms_event_polling_interval_s, 0,
> I think this is implementation detail and can be skipped. Also the summary
Are you planning to update this in a subsequent patch?
http://gerrit.cloudera.org:8080/#/c/15648/4/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15648/4/bin/impala-config.sh@177
PS4, Line 177: export CDP_BUILD_NUMBER=2506706
The changes in this file should be made in a separate change. Also, this build
number has a problem which causes a test failure. I uploaded a different patch
to bump it up a better build. https://gerrit.cloudera.org/#/c/15668/. May be
revert the changes to this file and rebase on top of my patch linked above?
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java@555
PS4, Line 555: null
May be change this to Collections.emptyList() so that the callers don't need to
check for null.
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@997
PS4, Line 997: public static class InsertEventInfo {
this duplicates a lot of code between the two version of the MetastoreShim. I
think we can keep the existing fireInsertEvent method in the MetastoreUtils
method. Here we can just make use of the return types. For instance, in hive-2
shim we do
public static List fireInsertEvent(IMetaStoreClient msClient,
List insertEventInfos) throws TException {
MetastoreUtil.fireInsertEvent(msClient, insertEventInfos);
return Collections.emptyList();
}
in hive-3 shim we do
public static List fireInsertEvent(IMetaStoreClient msClient,
List insertEventInfos) throws TException {
FireEventResponse resp = MetastoreUtil.fireInsertEvent(msClient,
insertEventInfos);
Preconditions.checkState(resp.getEventIds() != null &&
!resp.getEventIds().isEmpty());
return resp.getEventIds();
}
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java
File fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java:
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java@35
PS4, Line 35: private final long idFromEvent_;
rename to insertEventId_?
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java@69
PS4, Line 69: idFromEvent_ = eventId;
can you add a preconditions check that this eventId>0
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4487
PS4, Line 4487: // Firing insert events by making calls to HMS APIs can be
slow for tables with
: // large number of partitions.
this comment should be updated with the bulk API it should not be a problem
anymore. May be add a TODO to evaluate the performance of bulk API
http://gerrit.cloudera.org:8080/#/c/15648/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4506
PS4, Line 4506: e
can be moved to earlier line.
http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@725
PS3, Line 725: Assume.assumeTrue("Skipping this test because it only works
with Hive-3 or greater",
> Can you also enable the which is commented out here: https://github.com/apa
ping
--
To view, visit http://gerrit.cloudera.org:8080/15648
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id:
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15674 ) Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5621/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 22:16:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 21:50:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Xiaomeng Zhang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. IMPALA-8632: Add support for self-event detection for insert events In case of INSERT_EVENTS if Impala inserts into a table it causes a refresh to the underlying table/partition. This could be unnecessary when there is only one Impala cluster in the system. We can detect a self-event in such cases when the HMS API to fire a listener event returns the event id. This is used by EventProcessor to ignore the event when it is fetched later in the next polling cycle. Testing: Add testInsertFromImpala() in MetastoreEventsProcessorTest.java to test insert event self-event detection when insert into table and partition. Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M bin/impala-config.sh M common/thrift/BackendGflags.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 17 files changed, 506 insertions(+), 205 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/15648/4 -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Xiaomeng Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15648 ) Change subject: IMPALA-8632: Add support for self-event detection for insert events .. Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@286 PS3, Line 286:* > guess this will be removed once we bump up the CDP_BUILD right? Yes. It will be removed after I have hive jar updated. http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@849 PS3, Line 849: versionNumber == -1 > Why would this be a case? As we check -1 for version number. I am checking -1 for eventId as well. Is it guaranteed that eventId will not be less than 0? http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@897 PS3, Line 897: Preconditions.checkState(inFlightEvents_.size(false) == 0); > do we need a similar check for inFlightEvents_.size(false)? Done http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4411 PS3, Line 4411: ram isInsertOverwrite indicates if the operation was an inse > I think this check should be ((!catalog_.isEventProcessingActive() && isIns Done http://gerrit.cloudera.org:8080/#/c/15648/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4466 PS3, Line 4466: sMapBeforeInsert.entrySet().iterator().next(); > I think we need to do this via MetastoreShim since otherwise the response w Done -- To view, visit http://gerrit.cloudera.org:8080/15648 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf Gerrit-Change-Number: 15648 Gerrit-PatchSet: 4 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Tue, 07 Apr 2020 21:41:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5740/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 20:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3343, IMPALA-9489: Make impala-shell compatible with python 3.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15524 )
Change subject: IMPALA-3343, IMPALA-9489: Make impala-shell compatible with
python 3.
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15524/6/bin/impala-shell.sh
File bin/impala-shell.sh:
http://gerrit.cloudera.org:8080/#/c/15524/6/bin/impala-shell.sh@27
PS6, Line 27: USE_THRIFT11_GEN_PY
> who sets this? the user?
Presumably. I just wanted to leave a safety valve for reverting to the previous
version of thrift in the dev environment, but to be honest, I'm wondering if
it's even necessary.
http://gerrit.cloudera.org:8080/#/c/15524/6/shell/packaging/requirements.txt
File shell/packaging/requirements.txt:
http://gerrit.cloudera.org:8080/#/c/15524/6/shell/packaging/requirements.txt@8
PS6, Line 8: thrift==0.11.0
> a bit confused about the version of Thrift being used. after this patch, wi
This requirements file only pertains to the pip-installable python package that
a user might independently install outside of a cluster -- e.g., on their
laptop or whatever. In this case, it's always thrift 0.11.0. Honestly, for the
pip-installed shell, we could easily use a later version than 0.11.0, but I
wanted it to match the version that comes with CDH/CDP.
http://gerrit.cloudera.org:8080/#/c/15524/6/shell/shell_output.py
File shell/shell_output.py:
http://gerrit.cloudera.org:8080/#/c/15524/6/shell/shell_output.py@101
PS6, Line 101: with open(self.filename, 'ab') as out_file:
> this seems to change the logic so that it opens and closed the file during
When I had to update the string/unicode handling here, I took the opportunity
to refactor the class. I'm not wholly sure the result is behaving any
differently. Every time the impala-shell calls self.output_stream.write(), it
seems to always be preceded by an instantiation of a new OutputStream instance,
so I think the object is used once and destroyed. It doesn't look we ever cache
the object long term. This may reflect that fact that, presumably, one can
change the name of the output_file in an already running shell session.
Am I wrong about this though?
http://gerrit.cloudera.org:8080/#/c/15524/6/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/15524/6/tests/shell/test_shell_commandline.py@472
PS6, Line 472: # args = ['-q', "select '{0}'".format(RUSSIAN_CHARS)]
> delete?
Ack
--
To view, visit http://gerrit.cloudera.org:8080/15524
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb004d352fe230a890a6b6356496ba76c2fab615
Gerrit-Change-Number: 15524
Gerrit-PatchSet: 6
Gerrit-Owner: David Knupp
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 07 Apr 2020 20:31:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9199: Add support for single query retries on cluster membership changes
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14824 ) Change subject: IMPALA-9199: Add support for single query retries on cluster membership changes .. Patch Set 9: (8 comments) Still going through this, but some high level thoughts (and a few nit-picks I happened to notice already) http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/runtime/coordinator.cc@922 PS9, Line 922: parent_query_driver_->TryQueryRetry(parent_request_state_, _status)); So obviously the circular dependency here between QueryDriver, ClientRequestState, and Coordinator is unfortunate. It probably works as is, and it might be difficult to get rid of without significant code restructuring, but I'm concerned its confusing/brittle (eg. this call results in us taking ClientRequestState::lock_ on the ReportExecStatus rpc thread) and so I've been thinking through the options. Would it be possible to instead have the QueryDriver wait on the coordinator to finish and then check its status and decide whether to retry then? One problem is the QueryDriver needs to know not just if the query hit an error but if the error was something retryable, but we could do something like have the coordinator remember any nodes it blacklists and expose that info to the QueryDriver. Another issue is that it means we won't start the retry quite as quickly, since we have to wait for the QueryDriver to notice the error, but that might be fine - its already the case that Coordinator::Wait() will return immediately after an error is reported, without waiting for other backends to be cancelled, see HandleExecStateTransition()->CancelBackends()->backend_exec_complete_barrier_->NotifyRemaining(). The worse case is when the blacklisting info and the error status don't arrive in the same report, eg. the call to TryQueryRetry below, but that should be rare since the error status will have been generated at the same time as the AuxErrorInfo, so you just have to get pretty unlucky with the timing of when the report is generated. Maybe the bigger issue is I'm not sure its easy to do with the way the code is set up. I was trying to trace through the various Wait()/WaitAsync()/BlockOnWait() whatever calls but I'm not sure how that all works now. http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/runtime/coordinator.cc@1060 PS9, Line 1060: ExecEnv::GetInstance()->cluster_membership_mgr()->BlacklistExecutor( This of course doesn't actually guarantee that the retried query won't be scheduled on the executor that gets blacklisted, eg. if it takes longer for the query to get through admission control again than the blacklist timeout. We might want to consider doing something like passing through a list of executors to reschedule on, to avoid having queries repeatedly fail in the same way. On the other hand, the blacklist timeout was designed to be longer than it should take the statestore to notice the executor is down, so in theory if the executor really is down maybe it doesn't matter. Probably not necessary to do anything different for this patch, though, just wanted to point this out. http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/runtime/query-driver.h File be/src/runtime/query-driver.h: http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/runtime/query-driver.h@113 PS9, Line 113: (2) the : /// query has already been retried not sure what this is supposed to mean http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/client-request-state.h@84 PS9, Line 84: UNKNOWN I don't think this is used anywhere? http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/impala-hs2-server.cc@148 PS9, Line 148: unique_ptr exec_request = make_unique(); I don't think this is used anywhere? http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/impala-hs2-server.cc@763 PS9, Line 763: ClientRequestState* request_state = nullptr; So I'm wondering if its safe to no longer have any shared_ptr here and in the other places in this file - what's to stop the QueryDriver from getting deleted by Unregister while this function is executing, which would make this pointer no longer valid? http://gerrit.cloudera.org:8080/#/c/14824/9/be/src/service/impala-hs2-server.cc@1039 PS9, Line 1039: // If the query was retried, fetch the profile for the most recent attempt of the query I think its definitely necessary that we provide a way for all relevant profiles to be accessed through HS2, not just the webserver. This is one case where the
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15673/1/be/src/runtime/runtime-filter.cc File be/src/runtime/runtime-filter.cc: http://gerrit.cloudera.org:8080/#/c/15673/1/be/src/runtime/runtime-filter.cc@31 PS1, Line 31: DCHECK(!HasFilter()) << "SetFilter() should not be called multiple times."; > nit: we generally use braces to denote the critical section instead of call Done Move the DCHECKs within critical section as well, just to be safe. -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 19:55:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15673 to look at the new patch set (#2). Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival In function RuntimeFilter::WaitForArrival, there is a race condition where condition variable arrival_cv_ may be signaled right after thread get into the loop and before it call arrival_cv_.WaitFor(). This can cause runtime filter to wait the entire RUNTIME_FILTER_WAIT_TIME_MS even though the filter has arrived or canceled earlier than that. This commit avoid the race condition by making RuntimeFilter::SetFilter and RuntimeFilter::Cancel acquire arrival_mutex_ first before checking the value of arrival_time_ and release arrival_mutex_ before signaling arrival_cv_. Testing: - Pass core tests. Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a --- M be/src/runtime/runtime-filter.cc 1 file changed, 18 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/15673/2 -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5739/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 19:34:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8857: Fix flaky Kudu tests with external inserts
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15633 ) Change subject: IMPALA-8857: Fix flaky Kudu tests with external inserts .. Patch Set 1: > Change looks good to me. Just curious as to what happens during > reads from Impala client after setting the timestamp, do the reads > happen at the snapshot that has been set? Right so all reads of a Kudu table within the session that the timestamp was set for will be at that timestamp until the option is changed or unset. -- To view, visit http://gerrit.cloudera.org:8080/15633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b787f6542dc31dcd846f19576a060a89aec891d Gerrit-Change-Number: 15633 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 07 Apr 2020 19:20:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8857: Fix flaky Kudu tests with external inserts
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15633 ) Change subject: IMPALA-8857: Fix flaky Kudu tests with external inserts .. Patch Set 1: Change looks good to me. Just curious as to what happens during reads from Impala client after setting the timestamp, do the reads happen at the snapshot that has been set? -- To view, visit http://gerrit.cloudera.org:8080/15633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b787f6542dc31dcd846f19576a060a89aec891d Gerrit-Change-Number: 15633 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 19:14:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8857: Fix flaky Kudu tests with external inserts
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15633 ) Change subject: IMPALA-8857: Fix flaky Kudu tests with external inserts .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/15633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b787f6542dc31dcd846f19576a060a89aec891d Gerrit-Change-Number: 15633 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 19:14:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Alice Fan has posted comments on this change. ( http://gerrit.cloudera.org:8080/15674 ) Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. Patch Set 1: Code-Review+1 Thanks Grant for the quick fix. -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 19:13:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15673/1/be/src/runtime/runtime-filter.cc File be/src/runtime/runtime-filter.cc: http://gerrit.cloudera.org:8080/#/c/15673/1/be/src/runtime/runtime-filter.cc@31 PS1, Line 31: unique_lock l(arrival_mutex_); nit: we generally use braces to denote the critical section instead of calling unlock() explicitly. I.e. lines 30-40 should be enclosed in a new set of braces and the unlock() call can be removed. The current code is readable enough, just the consistency is preferable. Also below. -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 19:06:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9398: Fix shell history duplication when cmdloop breaks
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15345 ) Change subject: IMPALA-9398: Fix shell history duplication when cmdloop breaks .. Patch Set 4: Code-Review+1 > Patch Set 4: > > > Patch Set 4: Code-Review+1 > > > > > Patch Set 4: > > > > > > Thank you for the detailed review. > > > > > > While I was thinking on the bookkeeping idea I could not find any other > > > occurrence of calling the preloop() other than the ImpalaShell's base > > > class Cmd, which has the cmdloop() method that is: > > > Repeatedly issue a prompt, accept input, parse an initial prefix off the > > > received input, and dispatch to action methods, passing them the > > > remainder of the line as argument. > > > > > > While ImpalaShell is alive Cmd.cmdloop() is re-started by ImpalaShell > > > here: > > > https://github.com/apache/impala/blob/master/shell/impala_shell.py#L1882 > > > Every time the cmdloop() is called it calls preloop() as well, this could > > > be due to different exceptions in the cmdloop(): > > > > > > postloop() saves the content of the history however it is not called when > > > an exception occurs. > > > > > > Given these conditions, I decided to refactor the history reading logic > > > and moved outside of the preloop() method, so it will only be called once > > > per ImpalaShell object. Let me know your thoughts. > > > > Thanks Tamas for your detailed analysis! > > > > I think I understand the root cause of this issue more after your > > explanation. > > > > When there is no exception of KeyboardInterrupt, the function of postloop() > > overridden in impala_shell.py will write those items currently in the > > history to the history file and thus get_current_history_length() (the > > number of items currently in the history) will become 0 after this. > > Therefore, it is okay for preloop() overridden in impala_shell.py to load > > the history from file the next time shell.cmdloop() is called since there > > is no item currently in the history and hence there will be no duplicate > > item in this case. > > > > But when there is an exception of KeyboardInterrupt caught, postloop() is > > not called so that the number of items currently in the history is not 0, > > i.e., get_current_history_length() is not equal to 0. Therefore, the next > > time when shell.cmdloop() is called, the function preloop() will be called, > > resulting in those items in the history file being loaded to the currently > > non-empty history maintained by self.readline(). > > > > Is my understanding correct? If so, then I do not have any other comment. > > It seems that your new solution is more elegant than your previous > > approach. :-) > > I just realized my understanding is not completely correct. I found in my dev > environment that after readline.write_history_file() in postloop(), the > number of items currently in the history will not become 0. But if this is > the case, then I do not understand why in the case when there is no exception > of KeyboardInterrupt, there will be no duplicate item. Maybe I miss something. I think in the case when there is no exception of KeyboardInterrupt, the function preloop() overridden in implala_shell.py will only be executed once and hence there will be no duplicate item in the current history. And I think it may be a good idea to move the loading of history to the constructor of ImpalaShell so that we could guarantee that read_history_file() is only done once. -- To view, visit http://gerrit.cloudera.org:8080/15345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4faf46134f44d91e56748642f47d448707db53c Gerrit-Change-Number: 15345 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Tue, 07 Apr 2020 19:03:07 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5738/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 19:02:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9398: Fix shell history duplication when cmdloop breaks
Fang-Yu Rao has removed a vote on this change. Change subject: IMPALA-9398: Fix shell history duplication when cmdloop breaks .. Removed Code-Review+1 by Fang-Yu Rao -- To view, visit http://gerrit.cloudera.org:8080/15345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: If4faf46134f44d91e56748642f47d448707db53c Gerrit-Change-Number: 15345 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vincent Tran
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5620/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 18:53:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.h File be/src/runtime/runtime-state.h: http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.h@202 PS1, Line 202: /// The condition variable must have query lifetime. > nit: Note: Make sure the caller thread checks for cancellation while holdin Done http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.cc File be/src/runtime/runtime-state.cc: http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.cc@234 PS1, Line 234: // writing thread (e.g. IMPALA-9611). Drop it before signalling the CV so that a > nit: since is_cancelled_ is read while holding the lock Done -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 18:53:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9398: Fix shell history duplication when cmdloop breaks
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15345 ) Change subject: IMPALA-9398: Fix shell history duplication when cmdloop breaks .. Patch Set 4: > Patch Set 4: Code-Review+1 > > > Patch Set 4: > > > > Thank you for the detailed review. > > > > While I was thinking on the bookkeeping idea I could not find any other > > occurrence of calling the preloop() other than the ImpalaShell's base class > > Cmd, which has the cmdloop() method that is: > > Repeatedly issue a prompt, accept input, parse an initial prefix off the > > received input, and dispatch to action methods, passing them the remainder > > of the line as argument. > > > > While ImpalaShell is alive Cmd.cmdloop() is re-started by ImpalaShell here: > > https://github.com/apache/impala/blob/master/shell/impala_shell.py#L1882 > > Every time the cmdloop() is called it calls preloop() as well, this could > > be due to different exceptions in the cmdloop(): > > > > postloop() saves the content of the history however it is not called when > > an exception occurs. > > > > Given these conditions, I decided to refactor the history reading logic and > > moved outside of the preloop() method, so it will only be called once per > > ImpalaShell object. Let me know your thoughts. > > Thanks Tamas for your detailed analysis! > > I think I understand the root cause of this issue more after your explanation. > > When there is no exception of KeyboardInterrupt, the function of postloop() > overridden in impala_shell.py will write those items currently in the history > to the history file and thus get_current_history_length() (the number of > items currently in the history) will become 0 after this. Therefore, it is > okay for preloop() overridden in impala_shell.py to load the history from > file the next time shell.cmdloop() is called since there is no item currently > in the history and hence there will be no duplicate item in this case. > > But when there is an exception of KeyboardInterrupt caught, postloop() is not > called so that the number of items currently in the history is not 0, i.e., > get_current_history_length() is not equal to 0. Therefore, the next time when > shell.cmdloop() is called, the function preloop() will be called, resulting > in those items in the history file being loaded to the currently non-empty > history maintained by self.readline(). > > Is my understanding correct? If so, then I do not have any other comment. It > seems that your new solution is more elegant than your previous approach. :-) I just realized my understanding is not completely correct. I found in my dev environment that after readline.write_history_file() in postloop(), the number of items currently in the history will not become 0. But if this is the case, then I do not understand why in the case when there is no exception of KeyboardInterrupt, there will be no duplicate item. Maybe I miss something. -- To view, visit http://gerrit.cloudera.org:8080/15345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4faf46134f44d91e56748642f47d448707db53c Gerrit-Change-Number: 15345 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Tue, 07 Apr 2020 18:53:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 18:53:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 18:53:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15672 to look at the new patch set (#2). Change subject: IMPALA-9611: fix hang when cancelling join builder .. IMPALA-9611: fix hang when cancelling join builder The error could occur in the following scenario, where thread A is executing a join build fragment and thread B is cancelling the fragment instance. 1. Thread A is in HandoffToProbesAndWait(), reads is_cancelled_ and sees false. 2. Thread B in RuntimeState::Cancel() sets is_cancelled_ = true, acquires cancellation_cvs_lock_, then calls NotifyAll() on the condition variable 3. Thread A calls Wait() on the condition variable, blocks forever because cancellation already happened. The fix is for thread B to acquire the lock that thread A is holding. That prevents the race because #1 and #3 above are in the same critical section and thread B won't be able to signal the condition variable until thread A has released it. Testing: Added metric check to test_failpoints to make it easier to detect hangs caused by those tests in future. Looped test_failpoints.py overnight, which was previously enough to reproduce the failure within a couple of hours. Ran exhaustive tests. Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 --- M be/src/exec/join-builder.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M tests/failure/test_failpoints.py 4 files changed, 40 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/15672/2 -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 1: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.h File be/src/runtime/runtime-state.h: http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.h@202 PS1, Line 202: /// The condition variable must have query lifetime. nit: Note: Make sure the caller thread checks for cancellation while holding the 'mutex' to avoid race condition. http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.cc File be/src/runtime/runtime-state.cc: http://gerrit.cloudera.org:8080/#/c/15672/1/be/src/runtime/runtime-state.cc@234 PS1, Line 234: // writing thread (e.g. IMPALA-9611). Drop it before signalling the CV so that a nit: since is_cancelled_ is read while holding the lock -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 18:39:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump up CDP BUILD NUMBER to 2523282
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15668 ) Change subject: Bump up CDP_BUILD_NUMBER to 2523282 .. Bump up CDP_BUILD_NUMBER to 2523282 This change bumps up the CDP_BUILD_NUMBER to 2523282. The new CDP build includes a fix on Hive side (HIVE-23018) which is needed for improving the INSERT event support in Catalog. Testing [WIP]: 1. Running core tests for with USE_CDP_HIVE=true. Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 --- M bin/impala-config.sh 1 file changed, 8 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/15668/2 -- To view, visit http://gerrit.cloudera.org:8080/15668 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f3a989ccd076776615605674f046ccdd7d0dfc2 Gerrit-Change-Number: 15668 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 ) Change subject: IMPALA-9602: Fix case-sensitivity for local catalog .. Patch Set 3: (3 comments) Patch looks good to me. Left a few suggestions below. http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21 PS3, Line 21:start Impala with local catalog enabled : start-impala-cluster.py : --catalogd_args="--catalog_topic_mode=minimal" : --impalad_args="--use_local_catalog=true" :Create database in lower-case: "CREATE DATABASE db1;" :Run the following a few times (this errors without the patch): :impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE; : CREATE TABLE DB1.ddl_test1 (val string) We have a test_local_catalog.py or test_ddl.py in case you want to add this test case. http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625 PS3, Line 625: DbCacheKey May be its better to move toLowerCase() call inside DbCacheKey constructor. That way any future usage of dbName will not need a explicit lower-casing. http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@681 PS3, Line 681: TableCacheKey same as earlier, may be moving to TableCacheKey constructor is better. -- To view, visit http://gerrit.cloudera.org:8080/15653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26 Gerrit-Change-Number: 15653 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 07 Apr 2020 18:19:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15642 ) Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5737/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 Gerrit-Change-Number: 15642 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 18:12:34 + Gerrit-HasComments: No
[native-toolchain-CR] Add build script to create and publish Docker containers
Hello Hector Acosta, Joe McDonnell, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15675 to review the following change. Change subject: Add build script to create and publish Docker containers .. Add build script to create and publish Docker containers Adds automation to build and publish the containers used by the Docker-based toolchain builds. These can be used by the toolchain builds to save time by using pre-created containers instead of building the containers on the fly. They can also be used to provide a stable, controlled environment for the toolchain builds. The script can also be invoked from a Jenkins build environment. Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 --- M docker/buildall.py A jenkins/build.sh 2 files changed, 59 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/75/15675/1 -- To view, visit http://gerrit.cloudera.org:8080/15675 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iac9b9b17b14c1aa8d9512437eb6ae01f5e896c55 Gerrit-Change-Number: 15675 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-9398: Fix shell history duplication when cmdloop breaks
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15345 ) Change subject: IMPALA-9398: Fix shell history duplication when cmdloop breaks .. Patch Set 4: Code-Review+1 > Patch Set 4: > > Thank you for the detailed review. > > While I was thinking on the bookkeeping idea I could not find any other > occurrence of calling the preloop() other than the ImpalaShell's base class > Cmd, which has the cmdloop() method that is: > Repeatedly issue a prompt, accept input, parse an initial prefix off the > received input, and dispatch to action methods, passing them the remainder of > the line as argument. > > While ImpalaShell is alive Cmd.cmdloop() is re-started by ImpalaShell here: > https://github.com/apache/impala/blob/master/shell/impala_shell.py#L1882 > Every time the cmdloop() is called it calls preloop() as well, this could be > due to different exceptions in the cmdloop(): > > postloop() saves the content of the history however it is not called when an > exception occurs. > > Given these conditions, I decided to refactor the history reading logic and > moved outside of the preloop() method, so it will only be called once per > ImpalaShell object. Let me know your thoughts. Thanks Tamas for your detailed analysis! I think I understand the root cause of this issue more after your explanation. When there is no exception of KeyboardInterrupt, the function of postloop() overridden in impala_shell.py will write those items currently in the history to the history file and thus get_current_history_length() (the number of items currently in the history) will become 0 after this. Therefore, it is okay for preloop() overridden in impala_shell.py to load the history from file the next time shell.cmdloop() is called since there is no item currently in the history and hence there will be no duplicate item in this case. But when there is an exception of KeyboardInterrupt caught, postloop() is not called so that the number of items currently in the history is not 0, i.e., get_current_history_length() is not equal to 0. Therefore, the next time when shell.cmdloop() is called, the function preloop() will be called, resulting in those items in the history file being loaded to the currently non-empty history maintained by self.readline(). Is my understanding correct? If so, then I do not have any other comment. It seems that your new solution is more elegant than your previous approach. :-) -- To view, visit http://gerrit.cloudera.org:8080/15345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4faf46134f44d91e56748642f47d448707db53c Gerrit-Change-Number: 15345 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Tue, 07 Apr 2020 17:52:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15642 ) Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. Patch Set 6: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5613/ Note that this build only failed b/c of the python 2.6 compatibility check, which we will need to disable. All of the e2e tests passed: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/10130/testReport/ -- To view, visit http://gerrit.cloudera.org:8080/15642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 Gerrit-Change-Number: 15642 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:36:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15642 ) Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. Patch Set 6: (4 comments) Thanks for the comments. With regard to adhoc perf testing, I think it's not a bad idea. I think Thomas actually had a use case that he ran into recently that would be good to use. http://gerrit.cloudera.org:8080/#/c/15642/5/LICENSE.txt File LICENSE.txt: http://gerrit.cloudera.org:8080/#/c/15642/5/LICENSE.txt@616 PS5, Line 616: sqlparse-0.3.1: > version update? Done http://gerrit.cloudera.org:8080/#/c/15642/5/README.md File README.md: http://gerrit.cloudera.org:8080/#/c/15642/5/README.md@a75 PS5, Line 75: > why remove all of this? I actually think it's misleading. For one thing, which PYTHONPATH are we talking about? If you run impala-python, PYTHONPATH is different than when you run impala-shell.sh (once https://gerrit.cloudera.org/c/15524/ goes through). But even before then, when you look at set-pythonpath.sh, there's never a point where shell/ext-py/* is appended to PYTHONPATH. Those libs are solely for inclusion in the tarball deliverable. http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt File infra/python/deps/requirements.txt: http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt@a60 PS5, Line 60: > somewhat related question, why do we need to check in a copy of sqlparse? When we build the self-contained shell tarball that gets deployed to customers, we don't actually pull down any third-party libs from PyPI -- we use the packages checked in here: https://github.com/apache/impala/tree/master/shell/ext-py. sqlparse is just one of them. (I've gradually been updating the others too.) However, when we locally run impala-shell.sh, impala-python, or impala-py.test in the dev environment, we use the libs from infra/python/env/lib/*. (Caveat: when we use impala-py.test to run tests/shell/*, we do in fact test the shell environment from the deliverable that customers receive.) The script hat builds the tarball deliverable is https://github.com/apache/impala/blob/master/shell/make_shell_tarball.sh. Up until now, we have been using a patched version of 0.1.19 that was different than the upstream v0.1.19. That patch (from FredyW) was finally merged into upstream sqlparse in v0.3.0: https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG#L54 http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt@a59 PS5, Line 59: : : > is this no longer true? Technically, this comment applies to any package in any of these *requirements.txt files that also happens to be in shell/ext-py, not just sqlparse. -- To view, visit http://gerrit.cloudera.org:8080/15642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 Gerrit-Change-Number: 15642 Gerrit-PatchSet: 6 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:34:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
Hello Sahil Takiar, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15642 to look at the new patch set (#6). Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 Upgrades the impala-shell's bundled version of sqlparse to 0.3.1. There were some API changes in 0.2.0+ that required a re-write of the StripLeadingCommentFilter in impala_shell.py. As 0.1.19 was the last version of sqlparse to support python 2.6, this patch also breaks Impala's compatibility with python 2.6. No new tests were added, but all existing tests passed without modification. Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 --- M LICENSE.txt M README.md M bin/rat_exclude_files.txt M infra/python/deps/requirements.txt M shell/.gitignore D shell/ext-py/sqlparse-0.1.19/.travis.yml D shell/ext-py/sqlparse-0.1.19/AUTHORS D shell/ext-py/sqlparse-0.1.19/CHANGES D shell/ext-py/sqlparse-0.1.19/README.rst D shell/ext-py/sqlparse-0.1.19/bin/sqlformat D shell/ext-py/sqlparse-0.1.19/docs/source/analyzing.rst D shell/ext-py/sqlparse-0.1.19/docs/source/api.rst D shell/ext-py/sqlparse-0.1.19/docs/source/changes.rst D shell/ext-py/sqlparse-0.1.19/docs/source/conf.py D shell/ext-py/sqlparse-0.1.19/docs/source/index.rst D shell/ext-py/sqlparse-0.1.19/docs/source/indices.rst D shell/ext-py/sqlparse-0.1.19/docs/source/intro.rst D shell/ext-py/sqlparse-0.1.19/docs/source/ui.rst D shell/ext-py/sqlparse-0.1.19/docs/sqlformat.1 D shell/ext-py/sqlparse-0.1.19/pytest.ini D shell/ext-py/sqlparse-0.1.19/setup.py D shell/ext-py/sqlparse-0.1.19/sqlparse/engine/__init__.py D shell/ext-py/sqlparse-0.1.19/sqlparse/engine/filter.py D shell/ext-py/sqlparse-0.1.19/sqlparse/engine/grouping.py D shell/ext-py/sqlparse-0.1.19/sqlparse/exceptions.py D shell/ext-py/sqlparse-0.1.19/sqlparse/filters.py D shell/ext-py/sqlparse-0.1.19/sqlparse/formatter.py D shell/ext-py/sqlparse-0.1.19/sqlparse/functions.py D shell/ext-py/sqlparse-0.1.19/sqlparse/keywords.py D shell/ext-py/sqlparse-0.1.19/sqlparse/lexer.py D shell/ext-py/sqlparse-0.1.19/sqlparse/pipeline.py D shell/ext-py/sqlparse-0.1.19/sqlparse/sql.py D shell/ext-py/sqlparse-0.1.19/sqlparse/tokens.py D shell/ext-py/sqlparse-0.1.19/sqlparse/utils.py D shell/ext-py/sqlparse-0.1.19/tests/test_filters.py D shell/ext-py/sqlparse-0.1.19/tests/test_format.py D shell/ext-py/sqlparse-0.1.19/tests/test_functions.py D shell/ext-py/sqlparse-0.1.19/tests/test_grouping.py D shell/ext-py/sqlparse-0.1.19/tests/test_parse.py D shell/ext-py/sqlparse-0.1.19/tests/test_pipeline.py D shell/ext-py/sqlparse-0.1.19/tests/test_regressions.py D shell/ext-py/sqlparse-0.1.19/tests/test_split.py D shell/ext-py/sqlparse-0.1.19/tests/test_tokenize.py D shell/ext-py/sqlparse-0.1.19/tests/utils.py D shell/ext-py/sqlparse-0.1.19/tox.ini A shell/ext-py/sqlparse-0.3.1/AUTHORS A shell/ext-py/sqlparse-0.3.1/CHANGELOG R shell/ext-py/sqlparse-0.3.1/LICENSE R shell/ext-py/sqlparse-0.3.1/MANIFEST.in A shell/ext-py/sqlparse-0.3.1/README.rst R shell/ext-py/sqlparse-0.3.1/TODO A shell/ext-py/sqlparse-0.3.1/setup.cfg A shell/ext-py/sqlparse-0.3.1/setup.py R shell/ext-py/sqlparse-0.3.1/sqlparse/__init__.py A shell/ext-py/sqlparse-0.3.1/sqlparse/__main__.py A shell/ext-py/sqlparse-0.3.1/sqlparse/cli.py A shell/ext-py/sqlparse-0.3.1/sqlparse/compat.py A shell/ext-py/sqlparse-0.3.1/sqlparse/engine/__init__.py A shell/ext-py/sqlparse-0.3.1/sqlparse/engine/filter_stack.py A shell/ext-py/sqlparse-0.3.1/sqlparse/engine/grouping.py A shell/ext-py/sqlparse-0.3.1/sqlparse/engine/statement_splitter.py A shell/ext-py/sqlparse-0.3.1/sqlparse/exceptions.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/__init__.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/aligned_indent.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/others.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/output.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/reindent.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/right_margin.py A shell/ext-py/sqlparse-0.3.1/sqlparse/filters/tokens.py A shell/ext-py/sqlparse-0.3.1/sqlparse/formatter.py A shell/ext-py/sqlparse-0.3.1/sqlparse/keywords.py A shell/ext-py/sqlparse-0.3.1/sqlparse/lexer.py A shell/ext-py/sqlparse-0.3.1/sqlparse/sql.py A shell/ext-py/sqlparse-0.3.1/sqlparse/tokens.py A shell/ext-py/sqlparse-0.3.1/sqlparse/utils.py R shell/ext-py/sqlparse-0.3.1/tests/__init__.py A shell/ext-py/sqlparse-0.3.1/tests/conftest.py R shell/ext-py/sqlparse-0.3.1/tests/files/_Make_DirEntry.sql R shell/ext-py/sqlparse-0.3.1/tests/files/begintag.sql R shell/ext-py/sqlparse-0.3.1/tests/files/begintag_2.sql R shell/ext-py/sqlparse-0.3.1/tests/files/dashcomment.sql A shell/ext-py/sqlparse-0.3.1/tests/files/encoding_gbk.sql A shell/ext-py/sqlparse-0.3.1/tests/files/encoding_utf8.sql R shell/ext-py/sqlparse-0.3.1/tests/files/function.sql R
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/15630/5/be/src/gutil/CMakeLists.txt File be/src/gutil/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/15630/5/be/src/gutil/CMakeLists.txt@56 PS5, Line 56: MATCHES nit: does STREQUALS work? MATCHES is a regex match but I don't think we need a regex. http://gerrit.cloudera.org:8080/#/c/15630/5/be/src/service/CMakeLists.txt File be/src/service/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/15630/5/be/src/service/CMakeLists.txt@67 PS5, Line 67: libclang_rt.ubsan_standalone_cxx-aarch64.a nit: use a variable for just the architecture, instead of duplicating the whole code block -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:29:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5619/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:23:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 7: Looks like there's an equivalent change in Kudu here: https://gerrit.cloudera.org/#/c/14964/. I think at some point we'll be pulling in Kudu's changes en-masse so that change does need to go into Kudu. -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:20:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15674 ) Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5736/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 17:21:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9565 Remove unused included file mm malloc.h on ARM
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15586 ) Change subject: IMPALA-9565 Remove unused included file mm_malloc.h on ARM .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ea1f654b4da0e658843fad1c1f6de99b784dcff Gerrit-Change-Number: 15586 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 17:20:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5746: Test case for remote fragments releasing memory
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15666 ) Change subject: IMPALA-5746: Test case for remote fragments releasing memory .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15666/1/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/15666/1/tests/custom_cluster/test_restart_services.py@240 PS1, Line 240: status_report_max_retry_s So its true that IMPALA-2990 fixes the issue that fragments will eventually get cancelled after the coordinator goes down, but the default value for this is 10 minutes, which is a long time to be holding on to resources after a coordinator failure. Of course, if the fragment has finished executing, then all we're doing for those 10 minutes is retrying sending the final status report, which shouldn't require holding onto many resources, but I'm not sure if we're smart enough to release everything that's not needed before the final status report is done. It would be good to figure that out, and if we are smart enough then maybe have a test where we leave this at the default 10 minutes and check that the resources are released quickly anyways even if the fragments stick around, and if we're not smart enough then it would be a good thing to fix. Fwiw, I was just working on a case yesterday where we think this was an issue - a coordinator went down and clients were able to fail over to another one, but the autoscaler was triggered, presumably because it detected a spike in load due to the old queries not getting cancelled quickly. -- To view, visit http://gerrit.cloudera.org:8080/15666 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9fe8309f80f797d205b756ba58219f595aba4e5 Gerrit-Change-Number: 15666 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 07 Apr 2020 17:20:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9607: Fix test column storage attributes Kudu test
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15674 Change subject: IMPALA-9607: Fix test_column_storage_attributes Kudu test .. IMPALA-9607: Fix test_column_storage_attributes Kudu test This patch fixes the test_column_storage_attributes test for Kudu tables by removing the type attributes from the VARCHAR column name. I tested this fix in my development environment by running test_kudu.py with `--exploration_strategy exhaustive`. Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 --- M tests/query_test/test_kudu.py 1 file changed, 5 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/15674/1 -- To view, visit http://gerrit.cloudera.org:8080/15674 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1b69a342a5b144beceefd17fb5a84547a1ae0103 Gerrit-Change-Number: 15674 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15673 ) Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5735/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 17:05:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5734/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 Gerrit-Change-Number: 15612 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 17:03:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15672 ) Change subject: IMPALA-9611: fix hang when cancelling join builder .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5733/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 16:48:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15673 Change subject: IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival .. IMPALA-9612: Fix race condition in RuntimeFilter::WaitForArrival In function RuntimeFilter::WaitForArrival, there is a race condition where condition variable arrival_cv_ may be signaled right after thread get into the loop and before it call arrival_cv_.WaitFor(). This can cause runtime filter to wait the entire RUNTIME_FILTER_WAIT_TIME_MS even though the filter has arrived or canceled earlier than that. This commit avoid the race condition by making RuntimeFilter::SetFilter and RuntimeFilter::Cancel acquire arrival_mutex_ first before checking the value of arrival_time_ and release arrival_mutex_ before signaling arrival_cv_. Testing: - Pass core tests. Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a --- M be/src/runtime/runtime-filter.cc 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/15673/1 -- To view, visit http://gerrit.cloudera.org:8080/15673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7dffa626103ef0af06ad1e89231b0d2ee54bb94a Gerrit-Change-Number: 15673 Gerrit-PatchSet: 1 Gerrit-Owner: Riza Suminto
[Impala-ASF-CR] IMPALA-9176: shared null-aware anti-join build
Tim Armstrong has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/15612 ) Change subject: IMPALA-9176: shared null-aware anti-join build .. IMPALA-9176: shared null-aware anti-join build This switches null-aware anti-join (NAAJ) to use shared join builds with mt_dop > 0. To support this, we make all access to the join build data structures from the probe read-only. NAAJ requires iterating over rows from build partitions at various steps in the algorithm and before this patch this was not thread-safe. We avoided that problem by having a separate builder for each join node and duplicating the data. The main challenge was iteration over null_aware_partition()->build_rows() from the probe side, because it uses an embedded iterator in the stream so was not thread-safe (since each thread would be trying to use the same iterator). The solution is to extend BufferedTupleStream to allow multiple read iterators into a pinned, read-only, stream. Each probe thread can then iterate over the stream independently with no thread safety issues. With BufferedTupleStream changes, I partially abstracted ReadIterator more from the rest of BufferedTupleStream, but decided not to completely refactor so that this patchset didn't cause excessive churn. I.e. much BufferedTupleStream code still accesses internal fields of ReadIterator. Fix a pre-existing bug in grouping-aggregator where Spill() hit a DCHECK because the hash table was destroyed unnecessarily when it hit an OOM. This was flushed out by the parameter change in test_spilling. Testing: Add test to buffered-tuple-stream-test for multiple readers to BTS. Tweaked test_spilling_naaj_no_deny_reservation to have a smaller minimum reservation, required to keep the test passing with the new, lower, memory requirement. Updated a TPC-H planner test where resource requirements slightly decreased for the NAAJ. Ran the naaj tests in test_spilling.py with TSAN enabled, confirmed no data races. Ran exhaustive tests, which passed after fixing IMPALA-9611. Ran core tests with ASAN. Ran backend tests with TSAN. Perf: I ran this query that exercises EvaluateNullProbe() heavily. select l_orderkey, l_partkey, l_suppkey, l_linenumber from tpch30_parquet.lineitem where l_suppkey = 4162 and l_shipmode = 'AIR' and l_returnflag = 'A' and l_shipdate > '1993-01-01' and if(l_orderkey > 550, NULL, l_orderkey) not in ( select if(o_orderkey % 2 = 0, NULL, o_orderkey + 1) from orders where l_orderkey = o_orderkey) order by 1,2,3,4; It went from ~13s to ~11s running on a single impalad with this change, because of the inlining of CreateOutputRow() and EvalConjuncts(). I also ran TPC-H SF 30 on Parquet with mt_dop=4, and there was no change in performance. Change-Id: I95ead761430b0aa59a4fb2e7848e47d1bf73c1c9 --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h A be/src/exec/blocking-join-node.inline.h M be/src/exec/data-source-scan-node.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h A be/src/exec/exec-node.inline.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hdfs-avro-scanner-ir.cc M be/src/exec/hdfs-columnar-scanner-ir.cc M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-rcfile-scanner.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scanner.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node-ir.cc M be/src/exec/unnest-node.cc M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/buffered-tuple-stream.inline.h M be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test M tests/query_test/test_spilling.py 45 files changed, 786 insertions(+), 397 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/15612/10 -- To view, visit http://gerrit.cloudera.org:8080/15612 To unsubscribe, visit
[Impala-ASF-CR] IMPALA-9422: Re-visit and improve join node and builder's counters
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15663 ) Change subject: IMPALA-9422: Re-visit and improve join node and builder's counters .. Patch Set 1: (4 comments) The concept makes sense to me. Had some questions, and also a concern about the example profile. http://gerrit.cloudera.org:8080/#/c/15663/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15663/1//COMMIT_MSG@24 PS1, Line 24: non-child: 0.000ns, This non-child time doesn't seem right, we should be counting some time against the join node itself - at least PRobeTime. This non-child time in the join is kinda weird, cause we explicitly calculate local time, rather than just subtracting the child time. http://gerrit.cloudera.org:8080/#/c/15663/1/be/src/exec/partitioned-hash-join-builder.h File be/src/exec/partitioned-hash-join-builder.h: http://gerrit.cloudera.org:8080/#/c/15663/1/be/src/exec/partitioned-hash-join-builder.h@586 PS1, Line 586: Status SendHelper(RowBatch* build_batch); I think a better name is possible based on what it actually does, e.g. AddBatch() http://gerrit.cloudera.org:8080/#/c/15663/1/be/src/exec/partitioned-hash-join-builder.h@590 PS1, Line 590: FlushFinalHelper Maybe FinalizeBuild() http://gerrit.cloudera.org:8080/#/c/15663/1/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/15663/1/be/src/exec/partitioned-hash-join-builder.cc@743 PS1, Line 743: SCOPED_TIMER(profile()->total_time_counter()); Maybe we should be actually starting and stopping the inactive timer and this timer at exactly the same time, e.g. with the SCOPED_TIMER2 macro. Since I think we want all the time to be attributed to one or the other. -- To view, visit http://gerrit.cloudera.org:8080/15663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I604075a2c8efcff26705fb39672f29f309b2ed97 Gerrit-Change-Number: 15663 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 16:09:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9611: fix hang when cancelling join builder
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15672 Change subject: IMPALA-9611: fix hang when cancelling join builder .. IMPALA-9611: fix hang when cancelling join builder The error could occur in the following scenario, where thread A is executing a join build fragment and thread B is cancelling the fragment instance. 1. Thread A is in HandoffToProbesAndWait(), reads is_cancelled_ and sees false. 2. Thread B in RuntimeState::Cancel() sets is_cancelled_ = true, acquires cancellation_cvs_lock_, then calls NotifyAll() on the condition variable 3. Thread A calls Wait() on the condition variable, blocks forever because cancellation already happened. The fix is for thread B to acquire the lock that thread A is holding. That prevents the race because #1 and #3 above are in the same critical section and thread B won't be able to signal the condition variable until thread A has released it. Testing: Added metric check to test_failpoints to make it easier to detect hangs caused by those tests in future. Looped test_failpoints.py overnight, which was previously enough to reproduce the failure within a couple of hours. Ran exhaustive tests. Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 --- M be/src/exec/join-builder.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M tests/failure/test_failpoints.py 4 files changed, 35 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/15672/1 -- To view, visit http://gerrit.cloudera.org:8080/15672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I996ad2055d6542eb57e12c663b89de5f84208f77 Gerrit-Change-Number: 15672 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9545 Decide cacheline size of aarch64
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-9545 Decide cacheline size of aarch64 .. IMPALA-9545 Decide cacheline size of aarch64 ARM64's L3 cacheline size is different according to CPU vendor's architecture. If user defined CACHELINESIZE_AARCH64 in impala-config-local.sh, then we will use that value, if user did not define it, then we will get the value from OS, if fail, then we will use the default value 64. Change-Id: Id56bfa63e4b6cd957c4997f10de78a5f4111f61f Reviewed-on: http://gerrit.cloudera.org:8080/1 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M CMakeLists.txt M be/CMakeLists.txt M be/src/gutil/port.h M buildall.sh 4 files changed, 32 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id56bfa63e4b6cd957c4997f10de78a5f4111f61f Gerrit-Change-Number: 1 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15654 ) Change subject: IMPALA-9529: Fix multi-tuple predicates not assigned in column masking .. IMPALA-9529: Fix multi-tuple predicates not assigned in column masking Column masking is implemented by replacing the masked table with a table masking view which has masked expressions in its SelectList. However, nested columns can't be exposed in the SelectList, so we expose them in the output field of the view in IMPALA-9330. As a result, predicates that reference both primitive and nested columns of the masked table become multi-tuple predicates (referencing tuples of the view and the masked table). Such kinds of predicates are not assigned since they no longer bound to the view's tuple or the masked table's tuple. We need to pick up the masked table's tuple id when getting unassigned predicates for the table masking view. Also need to do this for assigning predicates to the JoinNode which is the only place that introduces multi-tuple predicates. Tests: - Add tests with multi-tuple predicates referencing nested columns. - Run CORE tests. Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3 Reviewed-on: http://gerrit.cloudera.org:8080/15654 Reviewed-by: Csaba Ringhofer Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test 10 files changed, 371 insertions(+), 32 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3 Gerrit-Change-Number: 15654 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15654 ) Change subject: IMPALA-9529: Fix multi-tuple predicates not assigned in column masking .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3 Gerrit-Change-Number: 15654 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 07 Apr 2020 15:10:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8980: Remove functional*.alltypesinsert from EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15529 ) Change subject: IMPALA-8980: Remove functional*.alltypesinsert from EE tests .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5617/ -- To view, visit http://gerrit.cloudera.org:8080/15529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea Gerrit-Change-Number: 15529 Gerrit-PatchSet: 5 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 14:53:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15642 ) Change subject: IMPALA-9362: Upgrade sqlparse 0.1.19 -> 0.3.1 .. Patch Set 5: (4 comments) do we need to do any perf testing, even if it is ad-hoc / sanity checks? http://gerrit.cloudera.org:8080/#/c/15642/5/LICENSE.txt File LICENSE.txt: http://gerrit.cloudera.org:8080/#/c/15642/5/LICENSE.txt@616 PS5, Line 616: sqlparse-0.1.19 version update? http://gerrit.cloudera.org:8080/#/c/15642/5/README.md File README.md: http://gerrit.cloudera.org:8080/#/c/15642/5/README.md@a75 PS5, Line 75: why remove all of this? http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt File infra/python/deps/requirements.txt: http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt@a60 PS5, Line 60: somewhat related question, why do we need to check in a copy of sqlparse? http://gerrit.cloudera.org:8080/#/c/15642/5/infra/python/deps/requirements.txt@a59 PS5, Line 59: : : is this no longer true? -- To view, visit http://gerrit.cloudera.org:8080/15642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77a1fd5ae311634a18ee04b8c389d8a3f3a6e001 Gerrit-Change-Number: 15642 Gerrit-PatchSet: 5 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 13:51:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15654 ) Change subject: IMPALA-9529: Fix multi-tuple predicates not assigned in column masking .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5616/ -- To view, visit http://gerrit.cloudera.org:8080/15654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3 Gerrit-Change-Number: 15654 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 07 Apr 2020 12:21:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5732/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 12:15:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5731/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 12:15:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
zhaoren...@hotmail.com has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's crc32 instructions with ARM's instructions Replace Intel's popcntq instruction with ARM's mechanism Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 --- M be/src/benchmarks/bswap-benchmark.cc M be/src/benchmarks/int-hash-benchmark.cc M be/src/codegen/CMakeLists.txt M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/group_varint-inl.h M be/src/kudu/util/group_varint-test.cc A be/src/kudu/util/sse2neon.h M be/src/util/bit-util-test.cc M be/src/util/bit-util.cc M be/src/util/bloom-filter-ir.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/sse-util.h A be/src/util/sse2neon.h 14 files changed, 3,813 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15531/16 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 16 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
zhaoren...@hotmail.com has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. IMPALA-9590 Resolve error when build tsan and ubsan on arm64 Tsan build will fail on atomicops-internals-x86.cc build, so if on arm64, just don't build it. And Ubsan build should link to libclang_rt.ubsan_standalone's aarch 64 version, not x86 version. Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d --- M be/src/gutil/CMakeLists.txt M be/src/service/CMakeLists.txt 2 files changed, 14 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/15630/5 -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15527 ) Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt Reduce duplicate code in thrift CMakeLists.txt. And if in future, we change hive to version 4 or higher. This can adapt autometically. Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Reviewed-on: http://gerrit.cloudera.org:8080/15527 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M common/thrift/CMakeLists.txt 1 file changed, 2 insertions(+), 9 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15527 ) Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 14 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 07 Apr 2020 11:11:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 15: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5730/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 15 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 11:07:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9590 Resolve error when build tsan and ubsan on arm64
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15630 ) Change subject: IMPALA-9590 Resolve error when build tsan and ubsan on arm64 .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5727/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b5b63e5e52bc818ea4591292ac36da941b2cf2d Gerrit-Change-Number: 15630 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 07 Apr 2020 10:56:26 + Gerrit-HasComments: No
