[Impala-ASF-CR] IMPALA-8990: Fix flakiness in test set request pool
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16749 ) Change subject: IMPALA-8990: Fix flakiness in test_set_request_pool .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7686/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife06509e936443579ca60780013ce01352c8932e Gerrit-Change-Number: 16749 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Nov 2020 06:46:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8990: Fix flakiness in test set request pool
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16749 Change subject: IMPALA-8990: Fix flakiness in test_set_request_pool .. IMPALA-8990: Fix flakiness in test_set_request_pool This patch fixes flakiness in test_set_request_pool where one of the test cases expects a query to already be accounted for in a resource pool but sometimes gets into a state where it runs before the previous query passes admission control. This fixes the issue by waiting for the previous query to clear admission control before attempting to run the aforementioned failing query. Testing: - Ran test locally in a loop for an hour to flush out flakiness. Change-Id: Ife06509e936443579ca60780013ce01352c8932e --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16749/1 -- To view, visit http://gerrit.cloudera.org:8080/16749 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ife06509e936443579ca60780013ce01352c8932e Gerrit-Change-Number: 16749 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: For 'I find the two solutions different, as apt-get can fail for reasons other than another process holding the lock. This solution reduces the number of scenarios in which a build spins indefinitely, which is a result that should be avoided in terms of CI/CD ergonomics and economics.' If another process holding the lock, current solution will also spins indefinitely. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 05:05:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: for 'why should it be done', no reason, just don't output to the console, I already tested, adding it or not don't impact the logic. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 05:03:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: > And I think this solution is same with 'changeing for loop to while > true', just different checking lock solution. Sorry, I missed this part of your message. I find the two solutions different, as apt-get can fail for reasons other than another process holding the lock. This solution reduces the number of scenarios in which a build spins indefinitely, which is a result that should be avoided in terms of CI/CD ergonomics and economics. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 04:47:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: > Don't output to the console. Yes, I understand what it does. My question is "why should it be done?" -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 04:38:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7685/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 04:10:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7684/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 04:02:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 3: Don't output to the console. And I think this solution is same with 'changeing for loop to while true', just different checking lock solution. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 03:58:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. IMPALA-10329 Change apt install retry times to 30 Change apt install retry times to 30 in bootstrap_system.sh, Because this always timeout recently. And add solution for waiting the apt's lock-frontend. Change-Id: I790750da36ad53c87a830dfab6803a1862490daf --- M bin/bootstrap_system.sh 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16725/3 -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-10314: Temporarily set optimize simple limit to true for testing.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16747 ) Change subject: IMPALA-10314: Temporarily set optimize_simple_limit to true for testing. .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6676/ -- To view, visit http://gerrit.cloudera.org:8080/16747 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fda41e3521bdb56696c70a1dbda4c23f3c2a45d Gerrit-Change-Number: 16747 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 19 Nov 2020 03:52:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16725/2/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/16725/2/bin/bootstrap_system.sh@212 PS2, Line 212: >/dev/null 2>&1 Looks good! One question - what is the rationale for redirecting stdout and stderr here? -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 03:51:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. Patch Set 2: Hi, Jim, as your suggest, I add the wait solution. -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Thu, 19 Nov 2020 03:41:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10329 Change apt install retry times to 30
zhaoren...@hotmail.com has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16725 ) Change subject: IMPALA-10329 Change apt install retry times to 30 .. IMPALA-10329 Change apt install retry times to 30 Change apt install retry times to 30 in bootstrap_system.sh, Because this always timeout recently. And add solution for waiting the apt's lock-frontend. Change-Id: I790750da36ad53c87a830dfab6803a1862490daf --- M bin/bootstrap_system.sh 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/16725/2 -- To view, visit http://gerrit.cloudera.org:8080/16725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I790750da36ad53c87a830dfab6803a1862490daf Gerrit-Change-Number: 16725 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 19 Nov 2020 03:19:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. This test for IMPALA-897 is testing that queries run by Impala Shell from a script file are closed correctly. This is tested by an assertion that there is one in-flight query during execution of a script containing several queries. The test then closes the shell and checks that there are no in-flight queries. This is the assertion which failed. Change this assertion to instead wait for the number of in-flight queries to be zero. This avoids whatever race was causing the flakiness. Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Reviewed-on: http://gerrit.cloudera.org:8080/16743 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/shell/test_shell_commandline.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 ) Change subject: IMPALA-10314: Optimize planning time for simple limits .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7683/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Nov 2020 02:23:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 ) Change subject: IMPALA-10314: Optimize planning time for simple limits .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test File testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test: http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test@241 PS4, Line 241: limit 1 > Yeah, that is a tricky issue. I went with a simple approach of skipping the empty files when we are iterating over the file descriptors and only increment the count for the non-empty files. The determination of empty vs non-empty also depends on the file system type..so there's a check for that. PS 5 has the latest changes. I also modified the commit message accordingly. I haven't added new tests for this since I am not quite sure if we have a data set that has such mixed empty and non-empty files per partition. If you guys think it is important to test that, I am open to adding one. -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Nov 2020 02:17:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Hello Qifan Chen, Shant Hovsepian, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16723 to look at the new patch set (#5). Change subject: IMPALA-10314: Optimize planning time for simple limits .. IMPALA-10314: Optimize planning time for simple limits This patch optimizes the planning time for simple limit queries by only considering a minimal set of partitions whose file descriptors add up to N (the specified limit). Each file is conservatively estimated to contain 1 row. This reduces the number of partitions processed by HdfsScanNode.computeScanRangeLocations() which, according to query profiling, has been the main contributor to the planning time especially for large number of partitions. Further, within each partition, we only consider the number of non-empty files that brings the total to N. This is an opt-in optimization. A new planner option OPTIMIZE_SIMPLE_LIMIT enables this optimization. Further, if there's a WHERE clause, it must have an 'always_true' hint in order for the optimization to be considered. For example: set optimize_simple_limit = true; SELECT * FROM T WHERE /* +always_true */ LIMIT 10; If there are too many empty files in the partitions, it is possible that the query may produce fewer rows although those are still valid rows. Testing: - Added planner tests for the optimization - Ran query_test.py tests by enabling the optimize_simple_limit - Added an e2e test. Since result rows are non-deterministic, only simple count(*) query on top of subquery with limit was added. Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test M testdata/workloads/functional-query/queries/QueryTest/range-constant-propagation.test 16 files changed, 500 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/16723/5 -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 5 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9121: try to avoid ASAN error in hdfs-util-test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16748 ) Change subject: IMPALA-9121: try to avoid ASAN error in hdfs-util-test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7682/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7b42be0f8b5d6c6a31095f9d1a278fd82bd500c Gerrit-Change-Number: 16748 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 19 Nov 2020 02:04:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 19 Nov 2020 01:54:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 This patch disables metadata/test_catalogd_debug_actions test on S3 builds due to its flakiness. The root cause of this seems to be that listing time on S3 is variable and the test becomes flaky because it measures the time taken by refresh command after a certain debug action is set. Testing: 1. Ran the test on my local environment to make sure it compiles fine. Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Reviewed-on: http://gerrit.cloudera.org:8080/16745 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/common/skip.py M tests/metadata/test_catalogd_debug_actions.py 2 files changed, 4 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7523: increase hbase assignment timeout
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16740 ) Change subject: IMPALA-7523: increase hbase assignment timeout .. IMPALA-7523: increase hbase assignment timeout Try to avoid flakiness by doubling the timeout. Change-Id: I32cf06ddc03abfb8d5f2fdeb3e153cf353b71fb3 Reviewed-on: http://gerrit.cloudera.org:8080/16740 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I32cf06ddc03abfb8d5f2fdeb3e153cf353b71fb3 Gerrit-Change-Number: 16740 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7523: increase hbase assignment timeout
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16740 ) Change subject: IMPALA-7523: increase hbase assignment timeout .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32cf06ddc03abfb8d5f2fdeb3e153cf353b71fb3 Gerrit-Change-Number: 16740 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 19 Nov 2020 01:48:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 ) Change subject: IMPALA-10314: Optimize planning time for simple limits .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test File testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test: http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test@241 PS4, Line 241: limit 1 > In HdfsScanNode.computeScanRangeLocation(), we skip computing the scan rang Yeah, that is a tricky issue. If we take the optimistic approach, we could add a new step in (2) when empty files are found. That is, in this new step we come up with extra up to non-empty files for a total of empty files found in the pruned list. Another approach would be abandon the pruned list and go with the full list. -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 4 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Nov 2020 01:42:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9121: try to avoid ASAN error in hdfs-util-test
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16748 Change subject: IMPALA-9121: try to avoid ASAN error in hdfs-util-test .. IMPALA-9121: try to avoid ASAN error in hdfs-util-test I couldn't discern the likely root cause of the ASAN error, but have a hunch that it's a background thread accessing some data structure that is being torn down as the process exits. The tests in this file are simple so there shouldn't really be that much that can go wrong, except for the stuff started by ExecEnv::Init(). I modified the test to only initialize the necessary configs in ExecEnv, not start up the whole thing. Hopefully that make the problem go away. Testing: Looped the test locally with ASAN. Change-Id: Ic7b42be0f8b5d6c6a31095f9d1a278fd82bd500c --- M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/util/hdfs-util-test.cc 3 files changed, 18 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/16748/1 -- To view, visit http://gerrit.cloudera.org:8080/16748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic7b42be0f8b5d6c6a31095f9d1a278fd82bd500c Gerrit-Change-Number: 16748 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. IMPALA-10189: addendum: improve comment Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Reviewed-on: http://gerrit.cloudera.org:8080/16557 Reviewed-by: Impala Public Jenkins Tested-by: Tim Armstrong --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 1 insertion(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-10189: addendum: improve comment .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 2: Hit IMPALA-10316. Going to override since it's so trivial. -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 19 Nov 2020 01:29:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 19 Nov 2020 01:29:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16412 ) Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7681/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Nov 2020 00:43:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10276: thread-safe access to RuntimeProfile::counter map
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16739 ) Change subject: IMPALA-10276: thread-safe access to RuntimeProfile::counter_map_ .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic21a13acf9c7c326a27334e61ce3729f1e3cab42 Gerrit-Change-Number: 16739 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 19 Nov 2020 00:38:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10276: thread-safe access to RuntimeProfile::counter map
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16739 ) Change subject: IMPALA-10276: thread-safe access to RuntimeProfile::counter_map_ .. IMPALA-10276: thread-safe access to RuntimeProfile::counter_map_ The bug was that 'counter_map_' can be mutated concurrent with total_time_counter() or inactive_timer() being called. This is fixed by storing a pointer directly to those counters and bypassing 'counter_map_'. This is then thread-safe and also has low overhead (adding lock acquisitions might have some perf impact, since total_time_counter() is called throughout query execution). Change-Id: Ic21a13acf9c7c326a27334e61ce3729f1e3cab42 Reviewed-on: http://gerrit.cloudera.org:8080/16739 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h 2 files changed, 34 insertions(+), 34 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic21a13acf9c7c326a27334e61ce3729f1e3cab42 Gerrit-Change-Number: 16739 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. IMPALA-10340: Cannot set up KDC from scratch IMPALA-9361 added experimental support for setting up a KDC for the mini cluster. However, the experimental-kerberos-setup.sh script fails with the error: KRB5_CONFIG: unbound variable The problem is that impala-config.sh unsets KDC-related variables if the cluster is not kerberized or does not exist. However, kerberized cluster cannot be created without installing KDC first. This change updates impala-config.sh to allow setting the KDC-related variables if IMPALA_KERBERIZE is set to 'true' and the cluster does not exist yet. Testing: * successfully set up KDC via experimental-kerberos-setup.sh after building Impala locally * verified that impala-config.sh sets/unsets KDC-related variables depending on cluster existence and config * started Kerberized and non-Kerberized mini clusters (after regenerating configs) Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Reviewed-on: http://gerrit.cloudera.org:8080/16744 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Hello Sahil Takiar, Joe McDonnell, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16412 to look at the new patch set (#9). Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. IMPALA-9930 (part 2): Introduce new admission control rpc service This patch introduces a new krpc service, AdmissionControlService, which coordinators can use to submit queries for admission. This patch adds some simple configuration flags that make it possible to have coordinators use this service to submit their queries for admission to other coordinators. These flags are only to make this patch testable and will be replaced when the separate admission control daemon is introduced in IMPALA-9975. The interface consists of the following RPCs: - AdmitQuery: takes a TQueryExecRequest and a TQueryOptions (serialized into sidecars), places the request on a queue to be processed by a thread pool and then immediately returns. - GetQueryStatus: takes a query id and returns the current admission status, including the QuerySchedulePB if admission has completed successfully but the query has not been released yet. - ReleaseQueryBackends: called when individual backends complete but the overall query is still running to release resources incrementally. This RPC will be called at most O(log(# backends)) per query due to BackendResourceState, which batches backends to release together. - ReleaseQuery: called when the query has completely finished. Releases all remaining resources. - CancelAdmission: called if a query is cancelled before an admission decision has been made to indicate that it should no longer be considered for admission. The majority of the patch consists of two classes: - AdmissionControlClient: used to abstract whether admission is being performed locally or remotely. In the local case, it is basically just a wrapper around AdmissionController. In the remote case, it handles serializing/deserializing of RPC params, polling GetQueryStatus() until a decision has been made, etc. - AdmissionControlService: exports the RPC interface and acts as a wrapper around AdmissionController. Some notable changes involved: - AdmissionController::SubmitForAdmission() no longer blocks while a query is queued. Instead, a new function WaitOnQueued() can be used to monitor the admission status of a queued query. - Adding events to the query timeline is moved out of AdmissionController and into the AdmissionControlClient classes, so that it always happens on the coordinator. - When a cluster is run in the new admission control service mode, only the impalad that is performing admission control exposes the /admission http endpoint. Observability will be cleaned up in a subsequent patch. Testing: - Modified existing admission control tests to run both with and without the admission control service enabled, including both the functional and stress tests. The 'num_queries' param in the stress test is modified to only use a single value to reduce the number of tests that are run and keep the running time reasonable. - Ran tpch10 on a local minicluster and observed no significant regressions. Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae --- M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-control-client.cc M be/src/scheduling/admission-control-client.h A be/src/scheduling/admission-control-service.cc A be/src/scheduling/admission-control-service.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/local-admission-control-client.cc M be/src/scheduling/local-admission-control-client.h A be/src/scheduling/remote-admission-control-client.cc A be/src/scheduling/remote-admission-control-client.h M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M be/src/util/sharded-query-map-util.cc M common/protobuf/admission_control_service.proto M tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py M tests/hs2/hs2_test_suite.py M tests/util/web_pages_util.py 24 files changed, 1,241 insertions(+), 190 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16412/9 -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in dockerized tests.
Vihang Karajgaonkar has removed Anurag Mantripragada from this change. ( http://gerrit.cloudera.org:8080/14272 ) Change subject: IMPALA-8795 : Enable event polling by default in dockerized tests. .. Removed reviewer Anurag Mantripragada. -- To view, visit http://gerrit.cloudera.org:8080/14272 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I222b64236060b3c4c2d554e2f10e129984ebe972 Gerrit-Change-Number: 14272 Gerrit-PatchSet: 15 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16728 ) Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6669/ -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 23:21:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 23:15:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 ) Change subject: IMPALA-10314: Optimize planning time for simple limits .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test File testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test: http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test@241 PS4, Line 241: limit 1 > This makes me feel we should skip those files with 0 rows during pruning. I In HdfsScanNode.computeScanRangeLocation(), we skip computing the scan range if file is empty: Line 912 on master: // Skips files that have no associated blocks. if (fileDesc.getNumFileBlocks() == 0) continue; However, we populate the totalFilesPerFs_ treemap earlier .. on line 891 and that's the one that gets used to display the EXPLAIN string. So, yeah there's some inconsistency in the display (although it is possible it is intentional to show all files including empty ones in the explain). For my patch, there are 2 steps in which the pruning happens: (1) in HdfsPartitionPruner when I am limiting the number of partitions based only the number of file descriptors per partition - i.e not examining each file descriptor since that would have overhead, and (2) in HdfsScanNode I am limiting the number of files since that code already iterates over the file descriptors. I guess I could skip empty files in step 2 even though it would mess up the calculation that was done in step 1. -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 4 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 23:14:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16412 ) Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7680/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 23:08:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16412 ) Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. Patch Set 8: (4 comments) http://gerrit.cloudera.org:8080/#/c/16412/7/be/src/scheduling/remote-admission-control-client.cc File be/src/scheduling/remote-admission-control-client.cc: http://gerrit.cloudera.org:8080/#/c/16412/7/be/src/scheduling/remote-admission-control-client.cc@121 PS7, Line 121: admit_status = Status(get_status_resp.status()); > Is there a JIRA for this? Might be an obstacle for production-readiness for As discussed, switched to long polling which should mean this only ever comes into play for queries that are queued for a non-trivial amount of time, in which case its somewhat less important. Also made it configurable just for good measure. http://gerrit.cloudera.org:8080/#/c/16412/7/be/src/scheduling/remote-admission-control-client.cc@184 PS7, Line 184: ReleaseQueryBackendsResponsePB resp; > Just thinking, but I think something like retrying the RPC then giving up a Added retries here and for the other similar rpcs here. As mentioned elsewhere, I think the cleanup mechanism will be a KeepAlive rpc that will be added in followup work. http://gerrit.cloudera.org:8080/#/c/16412/7/common/protobuf/admission_control_service.proto File common/protobuf/admission_control_service.proto: http://gerrit.cloudera.org:8080/#/c/16412/7/common/protobuf/admission_control_service.proto@233 PS7, Line 233: /// admission and the query getting released. > Does this return immediately and require the client to throttle polling, or As discussed offline, decided to go with long-polling. For fault tolerance, I think the plan is to add a specific KeepAlive rpc which coordinators will periodically send to the admission controller with a list of active query ids. That should be pretty light weight and cover any issues from the coordinator failing or from small rpc errors that get the coordinator and admission controller out of sync. For failures of the admission controller itself, I think the plan is to store the info needed to reconstruct the admission controller's state in the statestored. http://gerrit.cloudera.org:8080/#/c/16412/7/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/16412/7/tests/custom_cluster/test_admission_controller.py@1303 PS7, Line 1303: class TestAdmissionControllerWithACService(TestAdmissionController): > How much time does this add to exhaustive tests? Because of the reduction in dimensions for the stress test, this patch actually reduces the time to run this test file in exhaustive - from ~78 minutes to ~62 minutes (on my machine). -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 22:46:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Hello Sahil Takiar, Joe McDonnell, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16412 to look at the new patch set (#8). Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. IMPALA-9930 (part 2): Introduce new admission control rpc service This patch introduces a new krpc service, AdmissionControlService, which coordinators can use to submit queries for admission. This patch adds some simple configuration flags that make it possible to have coordinators use this service to submit their queries for admission to other coordinators. These flags are only to make this patch testable will be replaced when the separate admission control daemon is introduced in IMPALA-9975. The interface consists of the following RPCs: - AdmitQuery: takes a TQueryExecRequest and a TQueryOptions (serialized into sidecars), places the request on a queue to be processed by a thread pool and then immediately returns. - GetQueryStatus: takes a query id and returns the current admission status, including the QuerySchedulePB if admission has completed successfully but the query has not been released yet. - ReleaseQueryBackends: called when individual backends complete but the overall query is still running to release resources incrementally. This RPC will be called at most O(log(# backends)) per query due to BackendResourceState, which batches backends to release together. - ReleaseQuery: called when the query has completely finished. Releases all remaining resources. - CancelAdmission: called if a query is cancelled before an admission decision has been made to indicate that it should no longer be considered for admission. The majority of the patch consists of two classes: - AdmissionControlClient: used to abstract whether admission is being performed locally or remotely. In the local case, it is basically just a wrapper around AdmissionController. In the remote case, it handles serializing/deserializing of RPC params, polling GetQueryStatus() until a decision has been made, etc. - AdmissionControlService: exports the RPC interface and acts as a wrapper around AdmissionController. Some notable changes involved: - AdmissionController::SubmitForAdmission() no longer blocks while a query is queued. Instead, a new function CheckQueued() can be used to monitor the admission status of a queued query. - Adding events to the query timeline is moved out of AdmissionController and into the AdmissionControlClient classes, so that it always happens on the coordinator. - When a cluster is run in the new admission control service mode, only the impalad that is performing admission control exposes the /admission http endpoint. Observability will be cleaned up in a subsequent patch. Testing: - Modified existing admission control tests to run both with and without the admission control service enabled, including both the functional and stress tests. The 'num_queries' param in the stress test is modified to only use a single value to reduce the number of tests that are run and keep the running time reasonable. - Ran tpch10 on a local minicluster and observed no significant regressions. Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae --- M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-control-client.cc M be/src/scheduling/admission-control-client.h A be/src/scheduling/admission-control-service.cc A be/src/scheduling/admission-control-service.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/local-admission-control-client.cc M be/src/scheduling/local-admission-control-client.h A be/src/scheduling/remote-admission-control-client.cc A be/src/scheduling/remote-admission-control-client.h M be/src/scheduling/schedule-state.cc M be/src/scheduling/schedule-state.h M be/src/service/client-request-state.cc M be/src/service/impala-http-handler.cc M be/src/util/sharded-query-map-util.cc M common/protobuf/admission_control_service.proto M tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py M tests/hs2/hs2_test_suite.py M tests/util/web_pages_util.py 24 files changed, 1,240 insertions(+), 190 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16412/8 -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenk
[Impala-ASF-CR] IMPALA-10314: Temporarily set optimize simple limit to true for testing.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16747 ) Change subject: IMPALA-10314: Temporarily set optimize_simple_limit to true for testing. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7679/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16747 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fda41e3521bdb56696c70a1dbda4c23f3c2a45d Gerrit-Change-Number: 16747 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 22:43:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10314: Temporarily set optimize simple limit to true for testing.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16747 ) Change subject: IMPALA-10314: Temporarily set optimize_simple_limit to true for testing. .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6676/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16747 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fda41e3521bdb56696c70a1dbda4c23f3c2a45d Gerrit-Change-Number: 16747 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 22:25:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 22:24:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables Query option 'default_transactional_type' shouldn't affect Iceberg tables. Also, Iceberg tables shouldn't allow setting transactional properties. Testing: * Added e2e tests Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Reviewed-on: http://gerrit.cloudera.org:8080/16742 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/TableDef.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 3 files changed, 41 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10314: Temporarily set optimize simple limit to true for testing.
Aman Sinha has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16747 Change subject: IMPALA-10314: Temporarily set optimize_simple_limit to true for testing. .. IMPALA-10314: Temporarily set optimize_simple_limit to true for testing. Change-Id: I6fda41e3521bdb56696c70a1dbda4c23f3c2a45d --- M common/thrift/ImpalaInternalService.thrift 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/16747/1 -- To view, visit http://gerrit.cloudera.org:8080/16747 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6fda41e3521bdb56696c70a1dbda4c23f3c2a45d Gerrit-Change-Number: 16747 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6668/ -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 21:59:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6675/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 21:14:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7678/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 21:13:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7677/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 21:13:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6667/ -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 21:06:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6674/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 19:02:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 19:01:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 2: Code-Review+2 Carrying forward +2 from Tim. -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 18:58:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16745/1/tests/metadata/test_catalogd_debug_actions.py File tests/metadata/test_catalogd_debug_actions.py: http://gerrit.cloudera.org:8080/#/c/16745/1/tests/metadata/test_catalogd_debug_actions.py@22 PS1, Line 22: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 18:58:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16745 to look at the new patch set (#2). Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 This patch disables metadata/test_catalogd_debug_actions test on S3 builds due to its flakiness. The root cause of this seems to be that listing time on S3 is variable and the test becomes flaky because it measures the time taken by refresh command after a certain debug action is set. Testing: 1. Ran the test on my local environment to make sure it compiles fine. Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 --- M tests/common/skip.py M tests/metadata/test_catalogd_debug_actions.py 2 files changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/16745/2 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 18:55:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16745 ) Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16745/1/tests/metadata/test_catalogd_debug_actions.py File tests/metadata/test_catalogd_debug_actions.py: http://gerrit.cloudera.org:8080/#/c/16745/1/tests/metadata/test_catalogd_debug_actions.py@22 PS1, Line 22: @SkipIfS3.variable_listing_times flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 18:38:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10286: Disable metadata.test catalogd debug actions on S3
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16745 Change subject: IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 .. IMPALA-10286: Disable metadata.test_catalogd_debug_actions on S3 This patch disables metadata/test_catalogd_debug_actions test on S3 builds due to its flakiness. The root cause of this seems to be that listing time on S3 is variable and the test becomes flaky because it measures the time taken by refresh command after a certain debug action is set. Testing: 1. Ran the test on my local environment to make sure it compiles fine. Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 --- M tests/common/skip.py M tests/metadata/test_catalogd_debug_actions.py 2 files changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/16745/1 -- To view, visit http://gerrit.cloudera.org:8080/16745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I30bd10de468ad449c4a143a65cdcba97d9f0cd78 Gerrit-Change-Number: 16745 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7523: increase hbase assignment timeout
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16740 ) Change subject: IMPALA-7523: increase hbase assignment timeout .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6673/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32cf06ddc03abfb8d5f2fdeb3e153cf353b71fb3 Gerrit-Change-Number: 16740 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 18 Nov 2020 18:29:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7523: increase hbase assignment timeout
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16740 ) Change subject: IMPALA-7523: increase hbase assignment timeout .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32cf06ddc03abfb8d5f2fdeb3e153cf353b71fb3 Gerrit-Change-Number: 16740 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 18 Nov 2020 18:29:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6672/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 18:29:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Makes sense, thanks for taking the initiative to improve this -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 18:28:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 18:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10276: thread-safe access to RuntimeProfile::counter map
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16739 ) Change subject: IMPALA-10276: thread-safe access to RuntimeProfile::counter_map_ .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6671/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic21a13acf9c7c326a27334e61ce3729f1e3cab42 Gerrit-Change-Number: 16739 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Nov 2020 18:10:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10276: thread-safe access to RuntimeProfile::counter map
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16739 ) Change subject: IMPALA-10276: thread-safe access to RuntimeProfile::counter_map_ .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16739 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic21a13acf9c7c326a27334e61ce3729f1e3cab42 Gerrit-Change-Number: 16739 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 18 Nov 2020 18:10:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6670/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 18:03:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16728 ) Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6669/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:54:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16728 ) Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:54:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16744 ) Change subject: IMPALA-10340: Cannot set up KDC from scratch .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:51:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 17:50:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6668/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 17:51:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10189: addendum: improve comment
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16557 ) Change subject: IMPALA-10189: addendum: improve comment .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16557 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81c77c26fe60812f28c14a93ea26a2a44ac2bba9 Gerrit-Change-Number: 16557 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 17:50:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10283: Fix IllegalStateException in applying incremental partition updates
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16649 ) Change subject: IMPALA-10283: Fix IllegalStateException in applying incremental partition updates .. Patch Set 4: Code-Review+2 Thanks for clarifying the problem. The fix makes sense to me. -- To view, visit http://gerrit.cloudera.org:8080/16649 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1657684f8853b76b1524475a3b3c35fa22a0e36e Gerrit-Change-Number: 16649 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Nov 2020 17:49:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10252: fix invalid runtime filters for outer joins
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16622 ) Change subject: IMPALA-10252: fix invalid runtime filters for outer joins .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I507af1cc8df15bca21e0d8555019997812087261 Gerrit-Change-Number: 16622 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 17:40:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10340: Cannot set up KDC from scratch
Attila Doroszlai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16744 Change subject: IMPALA-10340: Cannot set up KDC from scratch .. IMPALA-10340: Cannot set up KDC from scratch IMPALA-9361 added experimental support for setting up a KDC for the mini cluster. However, the experimental-kerberos-setup.sh script fails with the error: KRB5_CONFIG: unbound variable The problem is that impala-config.sh unsets KDC-related variables if the cluster is not kerberized or does not exist. However, kerberized cluster cannot be created without installing KDC first. This change updates impala-config.sh to allow setting the KDC-related variables if IMPALA_KERBERIZE is set to 'true' and the cluster does not exist yet. Testing: * successfully set up KDC via experimental-kerberos-setup.sh after building Impala locally * verified that impala-config.sh sets/unsets KDC-related variables depending on cluster existence and config * started Kerberized and non-Kerberized mini clusters (after regenerating configs) Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/16744/1 -- To view, visit http://gerrit.cloudera.org:8080/16744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icaed2ad2fb1e1b60951a5a4138a9386588bc6972 Gerrit-Change-Number: 16744 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Doroszlai
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16728 ) Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7675/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:36:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16728 ) Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/16728/4/testdata/workloads/functional-query/queries/QueryTest/explain-level3.test File testdata/workloads/functional-query/queries/QueryTest/explain-level3.test: http://gerrit.cloudera.org:8080/#/c/16728/4/testdata/workloads/functional-query/queries/QueryTest/explain-level3.test@86 PS4, Line 86: Iceberg > You mean capital I as Iceberg? oops, yes -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:25:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10332: Add file formats to HdfsScanNode's thrift representation.
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16728 Change subject: IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. .. IMPALA-10332: Add file formats to HdfsScanNode's thrift representation. List all file formats that a HdfsScanNode needs to process in any fragment instance. It is possible that some file formats will not be needed in all fragment instances. This is a step towards sharing codegen between different impala backends. Using the file formats provided in the thrift file, a backend can codegen code for file formats that are not needed in its own process but are needed in other fragment instances running on other backends, and the resulting binary can be shared between multiple backends. Codegenning for file formats will be done based on the thrift message and not on what is needed for the actual backend. This leads to some extra work in case a file format is not needed for the current backend and codegen sharing is not available (at this point it is not implemented). However, the overall number of such cases is low. Also adding the file formats to the node's explain string at level 3. Testing: - Added tests to verify that the file formats are present in the explain string at level 3. Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d --- M be/src/exec/hdfs-scan-node-base.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test 4 files changed, 44 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16728/5 -- To view, visit http://gerrit.cloudera.org:8080/16728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iad6b8271bd248983f327c07883a3bedf50f25b5d Gerrit-Change-Number: 16728 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7674/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:08:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6667/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:05:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:05:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16743 ) Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 17:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7673/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 16:57:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 16:57:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun// DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 16:57:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16742 ) Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 16:51:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10249: Fix the flaky TestImpalaShell.test queries closed test.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16743 Change subject: IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. .. IMPALA-10249: Fix the flaky TestImpalaShell.test_queries_closed test. This test for IMPALA-897 is testing that queries run by Impala Shell from a script file are closed correctly. This is tested by an assertion that there is one in-flight query during execution of a script containing several queries. The test then closes the shell and checks that there are no in-flight queries. This is the assertion which failed. Change this assertion to instead wait for the number of in-flight queries to be zero. This avoids whatever race was causing the flakiness. Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d --- M tests/shell/test_shell_commandline.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16743/1 -- To view, visit http://gerrit.cloudera.org:8080/16743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib0485097c34282523ed0df6faa143fee6f74676d Gerrit-Change-Number: 16743 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman
[Impala-ASF-CR] IMPALA-10152: Add support for Iceberg HiveCatalog
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16721 ) Change subject: IMPALA-10152: Add support for Iceberg HiveCatalog .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/16721/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/16721/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2012 PS2, Line 2012: existingTbl instanceof IncompleteTable This will "double drop" Kudu tables where existingTbl instanceof IncompleteTable, but msTbl table could be retrieved and it indicates a synchronyzed Kudu table, as we dropped them in line 1998. My guess is that this will result in an exception from HMS dropTable, leading to keeping the table in catalogd. http://gerrit.cloudera.org:8080/#/c/16721/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2015 PS2, Line 2015: needsHmsAlterTable it calls dropTable, so needsHmsDropTable would clearer -- To view, visit http://gerrit.cloudera.org:8080/16721 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie574589a1751aaa9ccbd34a89c6819714d103197 Gerrit-Change-Number: 16721 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 16:48:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10318: default transactional type shouldn't affect Iceberg tables
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16742 Change subject: IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables .. IMPALA-10318: default_transactional_type shouldn't affect Iceberg tables Query option 'default_transactional_type' shouldn't affect Iceberg tables. Also, Iceberg tables shouldn't allow setting transactional properties. Testing: * Added e2e tests Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 --- M fe/src/main/java/org/apache/impala/analysis/TableDef.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 3 files changed, 41 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/16742/1 -- To view, visit http://gerrit.cloudera.org:8080/16742 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I86d1ac82ecd01a7455a0881a9e84aeb193dd5385 Gerrit-Change-Number: 16742 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10066: Fix test cancellation mid command fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16322 ) Change subject: IMPALA-10066: Fix test_cancellation_mid_command fails .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6664/ -- To view, visit http://gerrit.cloudera.org:8080/16322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib80706d52a85d2c19b13fbbe5695934658c0bf7e Gerrit-Change-Number: 16322 Gerrit-PatchSet: 7 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 16:01:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10152: Add support for Iceberg HiveCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16721 ) Change subject: IMPALA-10152: Add support for Iceberg HiveCatalog .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6665/ -- To view, visit http://gerrit.cloudera.org:8080/16721 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie574589a1751aaa9ccbd34a89c6819714d103197 Gerrit-Change-Number: 16721 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 15:57:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: I created this test table by Impala cherry-picking the write support patch from review. Since it got merget I'm wondering if it still makes sense to add this table to the dataload as I could create it easily in the tests with 2 consecutive inserts. I'm considering to remove this test table from the dataload and create it on the fly. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 15:29:15 + Gerrit-HasComments: No
[Impala-ASF-CR] WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16741 ) Change subject: WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7672/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16741 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I485680cf79d96d578dd8cfbfd554bec468fe84bd Gerrit-Change-Number: 16741 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 14:52:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10314: Optimize planning time for simple limits
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16723 ) Change subject: IMPALA-10314: Optimize planning time for simple limits .. Patch Set 4: (1 comment) Looks nice! In addition to the empty file concern, I wonder if in the explain output, one can clearly see the application of this optimization, other than checking out the files scanned vs the total one by one. Such an indicator could be very useful in rule out a problem (if any) in the area quickly. Sorry I was not able to see it in the code. http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test File testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test: http://gerrit.cloudera.org:8080/#/c/16723/4/testdata/workloads/functional-planner/queries/PlannerTest/optimize-simple-limit.test@241 PS4, Line 241: limit 1 This makes me feel we should skip those files with 0 rows during pruning. In my test for a table with textfile format, I can add empty files in the folder for the table and impala will process it. Query: explain select * from table_bar ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=0B Threads=2 | | Per-Host Resource Estimates: Memory=10MB | | WARNING: The following tables are missing relevant table and/or column statistics. | | default.table_bar | | | | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED] | | | | | 00:SCAN HDFS [default.table_bar] | |HDFS partitions=1/1 files=1 size=0B | |row-size=4B cardinality=0 | ++ [09:24:31 qchen@qifan-10229: parquet] sqlci -q "select * from table_bar" Starting Impala Shell with no authentication using Python 2.7.16 Warning: live_progress only applies to interactive shell sessions, and is being skipped for now. Opened TCP connection to localhost:21000 Connected to localhost:21000 Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build ebe72ec25f4c6daabaa27f6daddd03b887806507) Query: select * from table_bar Query submitted at: 2020-11-18 09:24:48 (Coordinator: http://qifan-10229:25000) Query progress can be monitored at: http://qifan-10229:25000/query_plan?query_id=df40c6ecaeeb3a0e:11dd5cb7 Fetched 0 row(s) in 4.64s drop table if exists table_bar purge; create table if not exists table_bar (a int) STORED AS textfile location '/tmp/table_bar_dir'; touch empty.txt hdfs dfs -copyFromLocal empty.txt /tmp/table_bar_dir -- To view, visit http://gerrit.cloudera.org:8080/16723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6a79263bc092e0f3e9a1d72da5618f3cc35574 Gerrit-Change-Number: 16723 Gerrit-PatchSet: 4 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 18 Nov 2020 14:43:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16741 ) Change subject: WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16741/1/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/16741/1/tests/query_test/test_iceberg.py@61 PS1, Line 61: b flake8: E501 line too long (95 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16741 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I485680cf79d96d578dd8cfbfd554bec468fe84bd Gerrit-Change-Number: 16741 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 18 Nov 2020 14:31:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16741 Change subject: WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions .. WiP: IMPALA-10237: Support Bucket and Truncate partition transforms as built-in functions Change-Id: I485680cf79d96d578dd8cfbfd554bec468fe84bd --- M be/src/codegen/impala-ir.cc M be/src/exprs/CMakeLists.txt A be/src/exprs/iceberg-functions-ir.cc A be/src/exprs/iceberg-functions.h M be/src/exprs/scalar-expr-evaluator.cc M common/function-registry/impala_functions.py A testdata/workloads/functional-query/queries/QueryTest/iceberg-transform-functions.test M tests/query_test/test_iceberg.py 8 files changed, 137 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/16741/1 -- To view, visit http://gerrit.cloudera.org:8080/16741 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I485680cf79d96d578dd8cfbfd554bec468fe84bd Gerrit-Change-Number: 16741 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 7:
> (1 comment)
We need to use relative path when generate data files by Spark, here is a
simple demo, hope it helps:
val conf = new Configuration()
val warehouseLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test"
val tblLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test/functional_parquet/hadoop_catalog_test"
val catalog = new HadoopCatalog(conf, warehouseLoc);
val sparkSchema = StructType(List(StructField("id", IntegerType,true),
StructField("user", StringType,false),StructField("action", StringType,false),
StructField("event_time", TimestampType,false)))
val icebergSchema = SparkSchemaUtil.convert(sparkSchema)
val spec =
PartitionSpec.builderFor(icebergSchema).hour("event_time").identity("action").build
val table = catalog.createTable(TableIdentifier.of("functional_parquet",
"hadoop_catalog_test") ,icebergSchema, spec, null)
val action_data = Seq(
Row(1,"Alex","view",Timestamp.valueOf("2020-01-01 08:00:00"))
)
val df = spark.createDataFrame(sc.makeRDD(action_data), sparkSchema)
df.write.format("iceberg").option("write-format",
"parquet").mode("append").save(tblLoc)
spark.read.format("iceberg").load(tblLoc).show
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Wed, 18 Nov 2020 12:28:27 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json File testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json: http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json@4 PS7, Line 4: hdfs://localhost:20500/ The docekerised tests don't like the namenode hardcoded. We can easily remove this from the JSON files, but to remove these from the manifest AVRO files we probably need to re-generate them with some different Iceberg configuration property. WangSheng was able to generate files this way, so he might know how to configure Iceberg. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 12:10:32 + Gerrit-HasComments: Yes
