[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: I have a fix for the flakiness thankfully, so we'll get there eventually :) -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 05:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Hello Abhishek Rawat, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15541 to look at the new patch set (#2). Change subject: IMPALA-9547: retry accept in test_shell_commandline .. IMPALA-9547: retry accept in test_shell_commandline This is a point solution to this particular socket.accept() call failing. The more general problem is described in https://www.python.org/dev/peps/pep-0475/ and fixed in Python 3.5. Change-Id: Icc9cab98b059042855ca9149427d079951471be0 --- M tests/shell/test_shell_commandline.py 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/15541/2 -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: > Patch Set 8: > > Same issue again, I'm not sure what suddenly is triggering it as it seems > unrelated to your patch. Sigh. let's see if third time is a charm. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 05:33:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15541 ) Change subject: IMPALA-9547: retry accept in test_shell_commandline .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 04:52:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Fix test_fuzz_nested_types test_fuzz_nested_types had queries that didn't parse successfully because they had duplicated names in an inline view: select count(*) from ( select ... a.pos, ... b.pos from ... ); We have 'a.pos' and 'b.pos' here but Impala still considers those as duplicated names, hence I added aliases for the duplicated fields, e.g.: a.pos as apos Now that the queries pase successfully they start the scanners on corrupted files which is the goal of this test. Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Reviewed-on: http://gerrit.cloudera.org:8080/15528 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 6 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Mar 2020 04:44:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 24 Mar 2020 04:26:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. IMPALA-8533: Impala daemon crash on sort This crash was caused by an empty sort tuple descriptor that was generated as a result of union substitutions replacing all sort fields with literals that were subsequently removed from the ordering spec. There was no check in place to prevent the empty tuple descriptor from being sent to impalad where it caused a divide-by-zero crash. Fix: This fix avoids inserting a sort node when there are no fields remaining to sort on. Also added a precondition to the SortNode that will prevent similar issues from crashing impalad. Testing: Testcases added to PlannerTest/union.test Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Reviewed-on: http://gerrit.cloudera.org:8080/15473 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M testdata/workloads/functional-planner/queries/PlannerTest/union.test 3 files changed, 279 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 7 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: IMPALA-9107: Add timestamp to maven logging options.
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS=${IMPALA_MAVEN_OPTIONS-}
:
: # If enabled, debug symbols are added to cross-comp
> Maybe it's better to just to this at the same place in mvn-quiet.sh?
That makes sense to me. I think that is the right place for this setting.
http://gerrit.cloudera.org:8080/#/c/15537/2/bin/mvn-quiet.sh
File bin/mvn-quiet.sh:
http://gerrit.cloudera.org:8080/#/c/15537/2/bin/mvn-quiet.sh@34
PS2, Line 34: LOGGING_OPTIONS = -Dorg.slf4j.simpleLogger.showDateTime
: DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
You'll need to get rid of the space between the equals and the value. Might as
well add quotes. (See LOG_FILE above)
As a small style nit, I think I would prefer not to have two separate variables
in the command below. I think of these as a single setting to enable
timestamps, so it'd be great if it was one variable. This is a time when going
beyond 90 characters is not so terrible. You can also wrap the line like this:
LOGGING_OPTIONS="-Dorg.slf4j.simpleLogger.showDateTime \
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss"
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 2
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 24 Mar 2020 04:22:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 04:14:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5535/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 04:14:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Same issue again, I'm not sure what suddenly is triggering it as it seems unrelated to your patch. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 04:14:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15541 Change subject: IMPALA-9547: retry accept in test_shell_commandline .. IMPALA-9547: retry accept in test_shell_commandline This is a point solution to this particular socket.accept() call failing. The more general problem is described in https://www.python.org/dev/peps/pep-0475/ and fixed in Python 3.5. Change-Id: Icc9cab98b059042855ca9149427d079951471be0 --- M tests/shell/test_shell_commandline.py 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/15541/1 -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15541 ) Change subject: IMPALA-9547: retry accept in test_shell_commandline .. Patch Set 1: (23 comments) http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@793 PS1, Line 793: # IMPALA-9547: retry accept(). This is required in Python < 3.5 because some flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@793 PS1, Line 793: # IMPALA-9547: retry accept(). This is required in Python < 3.5 because some flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@793 PS1, Line 793: # IMPALA-9547: retry accept(). This is required in Python < 3.5 because some tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@794 PS1, Line 794: # EINTR return calls from syscalls are not automatically retried. See PEP475. flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@794 PS1, Line 794: # EINTR return calls from syscalls are not automatically retried. See PEP475. tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@795 PS1, Line 795: while True: flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@795 PS1, Line 795: while True: tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@796 PS1, Line 796:try: flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@796 PS1, Line 796: flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@796 PS1, Line 796:try: tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@797 PS1, Line 797: connection, client_address = sock.accept() flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@797 PS1, Line 797: flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@797 PS1, Line 797: connection, client_address = sock.accept() tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@798 PS1, Line 798: break flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@799 PS1, Line 799:except IOError, e: flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@799 PS1, Line 799:except IOError, e: flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@799 PS1, Line 799:except IOError, e: tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@800 PS1, Line 800: if e.errno != errno.EINTR: flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@800 PS1, Line 800: flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@800 PS1, Line 800: if e.errno != errno.EINTR: tab used for whitespace http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@801 PS1, Line 801:raise flake8: W191 indentation contains tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@801 PS1, Line 801: flake8: E101 indentation contains mixed spaces and tabs http://gerrit.cloudera.org:8080/#/c/15541/1/tests/shell/test_shell_commandline.py@801 PS1, Line 801:raise tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 04:12:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15510 ) Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5534/ -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 03:32:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9373: more tactical IWYU fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15248 ) Change subject: IMPALA-9373: more tactical IWYU fixes .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5581/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8de71866bdf3211e53560d9bfe930e7657c4d7f1 Gerrit-Change-Number: 15248 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 24 Mar 2020 03:30:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5580/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 03:25:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15510 ) Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5582/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 03:05:40 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15510 ) Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5534/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 02:51:04 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
zhaoren...@hotmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15510 Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. [WIP]IMPALA-9538 Bump up linux-syscall-support.h Bump up linux-syscall-support.h to newest version which support aarch64 Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 --- M be/src/gutil/linux_syscall_support.h M be/src/gutil/spinlock_linux-inl.h M be/src/kudu/util/debug-util.cc 3 files changed, 1,746 insertions(+), 891 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/15510/8 -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5533/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:48:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Code-Review+2 Thank you for addressing that! -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9373: more tactical IWYU fixes
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15248 to look at the new patch set (#9). Change subject: IMPALA-9373: more tactical IWYU fixes .. IMPALA-9373: more tactical IWYU fixes This is a grab-bag of fixes that I did with a mix of manual inspection. The techniques used were: * Getting preprocessor output for a few files by modifying command lines from compiler_commands.json to include -E. This is revealing because you see all the random unrelated cruft that gets pulled in. A useful one liner to extract an (approximate) list of headers from preprocessor output is: grep '^#.*h' be/src/util/CMakeFiles/Util.dir/os-info.cc.i | \ grep -o '".*"' | sort -u * Looking at the IWYU recommendations for guidance on what headers can be removed (and what need to be added). * Grepping for includes of headers, especially in other headers where they become viral. An example one-liner to find these: git grep -l 'include.*' | grep '\.h$' Non-exhaustive list of changes made: --- Unnest classes from TmpFileMgr so we can forward-declare them. This lets us remove tmp-file-mgr.h from buffer-pool.h and query-state.h, which are both widely included headers in the codebase. Also remove webserver.h from other headers, since it pulls in openssl-util.h and consequently a lot of openssl headers. Avoid including runtime/multi-precision.h in other headers. It pulls in a lot of boost multiprecision headers that are only needed for internal implementations of math and decimal operations. This required replacing some references to int128_t with __int128_t, which I don't think significantly hurts code readability. Also remove references to decimal-util.h where they're not needed, since it transitively pulls in multi-precision.h Reduce includes of boost/date_time modules, which are transitively many places via timestamp-value.h. Remove transitive dependencies of timestamp-value.h to avoid pulling in remaining boost date_time headers where not needed. Dependent headers are: scalar-expr-evaluator.h, expr-value.h Remove references to debug-util.h in other headers, because it pulls in a lot of thread headers. Remove references to llvm-codegen.h where possible, because it pulls in many llvm headers. Other opportunities: * boost/algorithm/string.hpp includes many string algorithms and pulls in a lot of headers. * util/string-parser.h is a giant header with many dependencies. * There's lots of redundancy between boost and standard c++ headers. Both pull in vast numbers of utility headers for C++ metaprogramming and similar things. If we reduced virality of boost headers this would help a lot, and also if we switch to equivalent standard headers where possible (e.g. unordered_map, unordered_set, function, bind, etc). Compile time with clang/ASAN: - Before: real9m6.311s user62m25.006s sys 2m44.798s After: real8m17.073s user55m38.425s sys 2m25.808s Change-Id: I8de71866bdf3211e53560d9bfe930e7657c4d7f1 --- M be/src/benchmarks/atod-benchmark.cc M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/benchmarks/overflow-benchmark.cc M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/llvm-codegen.cc M be/src/common/init.cc M be/src/common/logging.cc M be/src/common/logging.h M be/src/common/status.cc M be/src/common/thread-debug-info-test.cc M be/src/common/thread-debug-info.h M be/src/exec/aggregator.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/catalog-op-executor.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/grouping-aggregator.cc M be/src/exec/hash-table-test.cc M be/src/exec/hdfs-avro-scanner-ir.cc M be/src/exec/hdfs-columnar-scanner-ir.cc M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/join-builder.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-chunk-reader.cc M be/src/exec/parquet/parquet-column-chunk-reader.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-version-test.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5527/ -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:46:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5528/ -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 24 Mar 2020 02:44:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#11). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). TODO: * finalize the observability Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 778 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/11 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5579/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:39:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
zhaoren...@hotmail.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain Download Python dependencies even skipping bootstrap toolchain. Because when you set SKIP_TOOLCHAIN_BOOTSTRAP=true, the python dependencies still need to be downloaded. The toolchain building process will not download the python dependencies autometically Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 --- M bin/impala-config.sh M buildall.sh 2 files changed, 10 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/15297/6 -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9373: more tactical IWYU fixes
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15248 ) Change subject: IMPALA-9373: more tactical IWYU fixes .. Patch Set 8: (3 comments) This makes sense to me. Thanks for putting this together! I have a couple minor nits, but it would be good to get this in. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/benchmarks/overflow-benchmark.cc File be/src/benchmarks/overflow-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/benchmarks/overflow-benchmark.cc@27 PS8, Line 27: #include "util/decimal-util.h" Nit: Not just this file: If we are going true IWYU, then we would also want util/decimal-constants.h wherever we use these constants. It doesn't matter from a compilation perspective, but if we end up doing IWYU as a precommit, then we may end up needing it. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/runtime/sorter.h File be/src/runtime/sorter.h: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/runtime/sorter.h@30 PS8, Line 30: class RuntimeProfile; Nit: I think this is no longer needed. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/util/bit-util-test.cc File be/src/util/bit-util-test.cc: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/util/bit-util-test.cc@39 PS8, Line 39: ArithmeticUtil Nit: From an IWYU perspective, this would imply we need util/arithmetic-util.h. -- To view, visit http://gerrit.cloudera.org:8080/15248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8de71866bdf3211e53560d9bfe930e7657c4d7f1 Gerrit-Change-Number: 15248 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 24 Mar 2020 01:16:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: IMPALA-9107: Add timestamp to maven logging options.
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS=${IMPALA_MAVEN_OPTIONS-}
:
: # If enabled, debug symbols are added to cross-comp
> Maybe it's better to just to this at the same place in mvn-quiet.sh?
OK, there was a typo in this comment, but you get the idea.
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 2
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 24 Mar 2020 00:36:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Mar 2020 00:34:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 10: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5577/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 00:33:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15537 ) Change subject: IMPALA-9107: Add timestamp to maven logging options. .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5578/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 24 Mar 2020 00:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: IMPALA-9107: Add timestamp to maven logging options.
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-9107: Add timestamp to maven log
> I think we can re-use IMPALA-9107 as the jira.
Done
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@9
PS1, Line 9: We found that using awk to add a timestamp to the maven log can
fail
: if gawk is not installed. It seems better to configure maven to add
: the timestamp itself.
> Please add a small snippet of what the maven output looks like with a times
Done
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS=${IMPALA_MAVEN_OPTIONS-}
:
: # If enabled, debug symbols are added to cross-comp
> We set IMPALA_MAVEN_OPTIONS in the environment to allow different settings
Maybe it's better to just to this at the same place in mvn-quiet.sh?
LOGGING_OPTIONS = -Dorg.slf4j.simpleLogger.showDateTime
DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
if ! mvn -B $IMPALA_MAVEN_OPTIONS $LOGGING_OPTIONS $LOGGING_OPTIONS "$@" | \
tee -a "$LOG_FILE" | grep -E -e WARNING -e ERROR -e SUCCESS -e FAILURE -e
Test; then
echo "mvn $IMPALA_MAVEN_OPTIONS $@ exited with code $?"
exit 1
fi
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh
File bin/mvn-quiet.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh@35
PS1, Line 35: DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
> Keep the -B
Doh. Sorry. Fixed.
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 2
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 24 Mar 2020 00:27:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
David Knupp has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15537 ) Change subject: IMPALA-9107: Add timestamp to maven logging options. .. IMPALA-9107: Add timestamp to maven logging options. We found that using awk to add a timestamp to the maven log can fail if gawk is not installed. It seems better to configure maven to add the timestamp itself. Running mvn -U -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss -B install -DskipTests Directory /home/dknupp/Impala/ext-data-source 16:37:16 [INFO] Scanning for projects... 16:37:16 [INFO] 16:37:16 [INFO] Reactor Build Order: 16:37:16 [INFO] 16:37:16 [INFO] Apache Impala External Data Source [pom] 16:37:16 [INFO] Apache Impala External Data Source API [jar] 16:37:16 [INFO] Apache Impala External Data Source Sample [jar] 16:37:16 [INFO] Apache Impala External Data Source Test Library [jar] 16:37:17 [INFO] 16:37:17 [INFO] < org.apache.impala:impala-data-source > 16:37:17 [INFO] Building Apache Impala External Data Source 1.0-SNAPSHOT [1/4] 16:37:17 [INFO] [ pom ]- [etc...] Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e --- M bin/mvn-quiet.sh 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15537/2 -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15454 )
Change subject: IMPALA-3766: optionally compress spilled data
..
Patch Set 10:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/bufferpool/buffer-pool-test.cc
File be/src/runtime/bufferpool/buffer-pool-test.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/bufferpool/buffer-pool-test.cc@1628
PS10, Line 1628: void BufferPoolTest::TestTmpFileAllocateError(const string&
compression, bool punch_holes) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc
File be/src/runtime/tmp-file-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@531
PS10, Line 531: int64_t expected_bytes_allocated = punch_holes ? 0 :
expected_scratch_bytes_allocated;
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@750
PS10, Line 750: ASSERT_OK(tmp_file_mgr.InitCustom(tmp_dir_specs, false, "",
punch_holes, metrics_.get()));
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@998
PS10, Line 998: file_group.Read(uncompressed_handle.get(),
MemRange(big_tmp.data(), big_tmp.size(;
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr.cc@788
PS10, Line 788: VLOG(3) << "Write " << tmp_file->path() << " " << file_offset
<< " " << buffer_to_write.len();
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/15454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659
Gerrit-Change-Number: 15454
Gerrit-PatchSet: 10
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 23 Mar 2020 23:50:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#10). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). TODO: * finalize the observability Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 770 insertions(+), 220 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/10 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 9: (5 comments) http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h File be/src/runtime/bufferpool/buffer-pool-counters.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h@59 PS9, Line 59: RuntimeProfile::Counter* uncompressed_bytes_written; > why add this here vs the TmpFileMgr? That's a good point. This is a per-operator counter versus a per-backend counter. Maybe it is more useful at the backend level, I'm not sure that understanding the compressibility of the data per-operator is all that useful. I'm also realising that it would be useful to have encryption and compression time per operator, because those contribute to the CPU time of that operator. So maybe I should do this: * move uncompress bytes to just being at the backend level * add compression time to both levels * add encryption time to the operator level (it's already at the backend level) http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h File be/src/runtime/tmp-file-mgr-internal.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h@96 PS9, Line 96: int64_t bytes_allocated_ = 0; > would it be clearer to rename this to 'file_write_offset_' I went with allocation_offset_ http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h File be/src/runtime/tmp-file-mgr.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h@157 PS9, Line 157: static constexpr int64_t HOLE_PUNCH_BLOCK_SIZE = 4096; > nit: docs on what this is used for and why it is set to 4K Done http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@61 PS9, Line 61: "most cases this should be used in conjunction with --disk_spill_punch_holes=true " : "to maximize the space saved."); > should we just enforce this? e.g. compression can only be used if hole punc Yeah I think compression + no hole punching is not a particularly useful configuration and a potential way that people could misconfigure it. http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@63 PS9, Line 63: disk_spill_compression_buffer_limit_bytes > what happens if compressing a buffer requires more than this many bytes? Updated the comment to explain (I think that's what you were getting at, right?). I also reduced the limit a bit to be more conservative. -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 23:49:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5530/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 23:45:46 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 23:45:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 6: Failure in precommit looks unrelated (one is IMPALA-9547, another sees the linker getting killed), retrying. -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 23:30:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5529/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 23:29:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 6 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 23:29:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15378 ) Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5576/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 Gerrit-Change-Number: 15378 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 23:21:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Sahil Takiar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. IMPALA-5904: (part 3) Fix more TSAN bugs As of this commit, all backend tests are TSAN clean (or suppressions exist for any races that are considered benign). TSAN Fixes: * Fixes IMPALA-9374: Possible data race in TupleDescriptor::GetLlvmStruct * Replaces the assignment of llvm_field_idx_ with slot_idx_, since they are the same; added a DCHECK to enforce this * Data race in be/src/exprs/scalar-expr-evaluator.cc:285:26 impala::ScalarExprEvaluator::GetValue(...) * This race seems specific to data-stream-test, and does not occur during any other tests * I made some changes to DataStreamTest to fix the sharing of TupleRowComparators between threads Suppresions: * Added a suppresion for IMPALA-9455: Possible data race in kudu::security::InitKerberosForServer * Added suppresions for hs2-http-test since it uses the ThriftServer, which is known to be racey (IMPALA-9314) Testing: * Ran exhaustive tests * Re-ran TSAN tests and confirmed the data races have been fixed Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Reviewed-on: http://gerrit.cloudera.org:8080/15363 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/rpc/hs2-http-test.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M bin/tsan-suppressions.txt 5 files changed, 47 insertions(+), 24 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9537: Add LDAP auth to the webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15538 ) Change subject: IMPALA-9537: Add LDAP auth to the webui .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5575/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e92481929f2f06898b8496233ab4134792c9f10 Gerrit-Change-Number: 15538 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 22:52:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Code-Review+2 There was a change several years ago to *not* send non-materialized slots to the backend. I wonder if this logic was actually a holdover from before that change. -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 22:48:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/15378 ) Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@656 PS11, Line 656: if self.max_tries == 1: : return 0 : ratio = float(num_tries) / self.max_tries : if ratio < 0.3: : return 0.1 : elif ratio < 0.6: : return 0.3 : return 2 > The current logic basically has following (We have 3 tries total including I ended up using a simple function same as what you've suggested: retry_interval = (num_tries - 1) * self.sleep_interval. The behavior can be controlled by tweaking self.sleep_interval. I also changed default self.max_tries to 4 (1 original try and 3 retries). -- To view, visit http://gerrit.cloudera.org:8080/15378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 Gerrit-Change-Number: 15378 Gerrit-PatchSet: 11 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 22:39:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Abhishek Rawat has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/15378 ) Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol .. IMPALA-9466: impala-shell client retry for hs2-http protocol Added retries for idempotent rpcs: OpenSession, PingImpalaHS2Service, GetResultSetMetadata, CloseImpalaOperation (non dmls), CancelOperation, GetOperationStatus, GetRuntimeProfile, GetExecSummary, GetLog Retries were also added to the 'set all' query execution and subsequent result fetch in the ImpalaHS2Client._open_session() The retries are only supported for hs2-http protocol and enabled by default. At most there are 3 tries for a failed rpc with at least 2 second wait duration between tries. Only failed rpcs due to an error in the http transport are retried and if an rpc failed because the server returned an error in the rpc response then such scenarios are not retriable. Improved error diagnostics by dumping stack trace when ImpalaShell. _execute_stmt() gets an 'Unknown Exception'. Testing: - Added a custom_cluster test which injects fault into the http transport and checks expected behavior from the various rpcs. Some of these tests leave the session in an open state and so these tests are not suitable for the e2e test framework which have metric verifiers expecting related metrics to be 0 at the end of the test. - Manually tested real world scenarios with impala-shell client communicating with an impala coordinator via a fault injecting istio mesh. - Manually tested dropping connections on an nginx ingress gateway by sending SIGTERM to all worker processes. Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 --- M shell/impala_client.py M shell/impala_shell.py A tests/custom_cluster/test_hs2_fault_injection.py 3 files changed, 498 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/15378/12 -- To view, visit http://gerrit.cloudera.org:8080/15378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 Gerrit-Change-Number: 15378 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15378 ) Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/15378/12/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/15378/12/shell/impala_client.py@976 PS12, Line 976: N flake8: E501 line too long (93 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/15378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 Gerrit-Change-Number: 15378 Gerrit-PatchSet: 12 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 22:36:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 22:25:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Add timestamp to maven logging options.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: Add timestamp to maven logging options.
..
Patch Set 1:
(4 comments)
Please verify that
https://github.com/apache/impala/blob/master/bin/jenkins/get_maven_statistics.sh
still works on this new output. I think it should. It didn't make any real
assumptions about the timestamps.
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@7
PS1, Line 7: Add timestamp to maven logging options.
I think we can re-use IMPALA-9107 as the jira.
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@9
PS1, Line 9: We found that using awk to add a timestamp to the maven log can
fail
: if gawk is not installed. It seems better to configure maven to add
: the timestamp itself.
Please add a small snippet of what the maven output looks like with a timestamp.
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS="-U \
: -Dorg.slf4j.simpleLogger.showDateTime=true \
: -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss"
We set IMPALA_MAVEN_OPTIONS in the environment to allow different settings for
different automated jobs, so we should respect the environment version. (We can
prepend the logging args so you have "${IMPALA_MAVEN_OPTIONS-} -Dorg.slf4j...
etc etc")
We don't always need -U, so we should avoid it if we can.
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh
File bin/mvn-quiet.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh@35
PS1, Line 35: if ! mvn $IMPALA_MAVEN_OPTIONS "$@" | \
Keep the -B
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 1
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Mon, 23 Mar 2020 22:21:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9537: Add LDAP auth to the webui
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15538 Change subject: IMPALA-9537: Add LDAP auth to the webui .. IMPALA-9537: Add LDAP auth to the webui This patch adds a startup flag --webserver_require_ldap, which if set to true secures the debug webserver with LDAP username/password authentication. Testing: - Added a FE test that runs a custom cluster with ldap webserver auth enabled and verifies it works as expected. Change-Id: I6e92481929f2f06898b8496233ab4134792c9f10 --- M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/util/webserver.cc M be/src/util/webserver.h M common/thrift/metrics.json M fe/src/test/java/org/apache/impala/customcluster/CustomClusterRunner.java A fe/src/test/java/org/apache/impala/customcluster/LdapWebserverTest.java M fe/src/test/java/org/apache/impala/util/Metrics.java M tests/common/impala_service.py 9 files changed, 376 insertions(+), 74 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/15538/1 -- To view, visit http://gerrit.cloudera.org:8080/15538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6e92481929f2f06898b8496233ab4134792c9f10 Gerrit-Change-Number: 15538 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Okay, I think this is in a working state now. @Tim if you could take another look at the changes to descriptors.h and descriptors.cc that would be great. In short, TSAN was reporting a race condition on setting the value of 'llvm_field_idx_' in TupleDescriptor::GetLlvmStruct. After taking a closer look, it looks like 'llvm_field_idx_' is always equal to 'slot_idx_' so I just replaced it with 'slot_idx_' instead. I added a DCHECK as well to ensure this. I ran exhaustive tests with TSAN disabled, and be tests with TSAN enabled. Both test runs pass. Pre-commit passes as well. -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 22:07:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5528/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5527/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 7: Agree this looks like a flaky test. I filed IMPALA-9547 to track. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 7 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Code-Review+2 LGTM For anyone else watching, yes the CDP GBN bump will break other tests, but I've triaged all the failures already. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:47:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 This patch bumps up CDP_BUILD_NUMBER to 2244454 which contains a change introduced by RANGER-2688. Due to this change, we added to ranger-admin-site.xml.template a cookie-related configuration so that the Ranger server could be properly started. Testing: Verified that the data loading passes and that all the Ranger-related FE and E2E tests are successful - when $USE_CDP_HIVE is false, and - when $USE_CDP_HIVE is true. Change-Id: I7750f73834368c7109965e78b147238fc6316f49 --- M bin/impala-config.sh M testdata/cluster/ranger/ranger-admin-site.xml.template 2 files changed, 13 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15533/2 -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] Add timestamp to maven logging options.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15537 ) Change subject: Add timestamp to maven logging options. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 23 Mar 2020 21:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5572/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:30:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:13:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5523/ -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 21:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 9: (5 comments) went through this briefly, few comments http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h File be/src/runtime/bufferpool/buffer-pool-counters.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h@59 PS9, Line 59: RuntimeProfile::Counter* uncompressed_bytes_written; why add this here vs the TmpFileMgr? http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h File be/src/runtime/tmp-file-mgr-internal.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h@96 PS9, Line 96: int64_t bytes_allocated_ = 0; would it be clearer to rename this to 'file_write_offset_' http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h File be/src/runtime/tmp-file-mgr.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h@157 PS9, Line 157: static constexpr int64_t HOLE_PUNCH_BLOCK_SIZE = 4096; nit: docs on what this is used for and why it is set to 4K http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@61 PS9, Line 61: "most cases this should be used in conjunction with --disk_spill_punch_holes=true " : "to maximize the space saved."); should we just enforce this? e.g. compression can only be used if hole punching is support on the underlying fs. if users complain and start asking for spill compression support on a fs without hole punching, we can always remove the restriction later. i think the benefit is that is simplifies the combination of configurations Impala needs to support http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@63 PS9, Line 63: disk_spill_compression_buffer_limit_bytes what happens if compressing a buffer requires more than this many bytes? -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:01:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/15462 )
Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive
normal form
..
Patch Set 7:
> Patch Set 7: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5511/
Looking at the archive.zip for this build failure, the failure I see is the
following in TestImpalaShell which seems unrelated to the code changes. However
it doesn't look like PlannerTest was run. Was it supposed to ?
[gw15] FAILED
shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening[table_format_and_file_extension:
('parquet', '.parq') | protocol: beeswax]
--
To view, visit http://gerrit.cloudera.org:8080/15462
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072
Gerrit-Change-Number: 15462
Gerrit-PatchSet: 7
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 23 Mar 2020 20:59:42 +
Gerrit-HasComments: No
[Impala-ASF-CR] Add timestamp to maven logging options.
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15537 Change subject: Add timestamp to maven logging options. .. Add timestamp to maven logging options. We found that using awk to add a timestamp to the maven log can fail if gawk is not installed. It seems better to configure maven to add the timestamp itself. Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e --- M bin/impala-config.sh M bin/mvn-quiet.sh 2 files changed, 5 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15537/1 -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: I'm not planning to take a look unless you think the extra eyes are needed. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 20:48:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8980: Remove functional*.alltypesinsert from EE tests
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15529 ) Change subject: IMPALA-8980: Remove functional*.alltypesinsert from EE tests .. Patch Set 1: (3 comments) Thanks for taking care about this! http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG@11 PS1, Line 11: -Swapped out the Reset table and Drop partition SETUP tags to Truncate table QUERY statement. Do we still use SETUP anywhere in the tests? If not, then it would be great to remove it from https://github.com/apache/impala/blob/master/tests/common/impala_test_suite.py#L770 http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG@12 PS1, Line 12: -Deleted a test query in ‘insert.test’. The test incorrectly tried to clear out a table which was not working, and should not work either. nit: please wrap commit message lines at 72 chars + I added more comments to the deleted test http://gerrit.cloudera.org:8080/#/c/15529/1/testdata/workloads/functional-query/queries/QueryTest/insert.test File testdata/workloads/functional-query/queries/QueryTest/insert.test: http://gerrit.cloudera.org:8080/#/c/15529/1/testdata/workloads/functional-query/queries/QueryTest/insert.test@a675 PS1, Line 675: : : : : : : : : : : : : : : : Instead of deleting we could also for test the current behavior. Note that it is actually not self evident what to do in this case (INSERT OVERWRITE with dynamic partitioning). Hive doesn't delete all partitions, while Spark does it by default (it has an option to decide: spark.sql.sources.partitionOverwriteMode). -- To view, visit http://gerrit.cloudera.org:8080/15529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea Gerrit-Change-Number: 15529 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 20:39:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5522/ -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 20:28:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Last verification job failed due to IMPALA-9491. -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 20:05:57 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5526/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 19:38:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 19:38:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5571/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 19:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15533/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15533/1//COMMIT_MSG@16 PS1, Line 16: $USE_CDP_HIVE : is false. how about when USE_CDP_HIVE=true? do the ranger tests pass now? I expect there are a bunch of other failures as well, but we should validate that at least data-load passes and the ranger tests have now been fixed. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 18:45:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8690: Add LIRS cache eviction algorithm
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15306 ) Change subject: IMPALA-8690: Add LIRS cache eviction algorithm .. Patch Set 18: (5 comments) http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc File be/src/util/cache/lirs-cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc@83 PS15, Line 83: erreference recency), and it has three different types of e > Yeah, this needed more clarity. I reworked this paragraph. makes a lot more sense, thanks! http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc@122 PS15, Line 122: > Added a description here along with the lifecycle. makes sense, thanks! http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc File be/src/util/cache/lirs-cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@77 PS18, Line 77: // If the key has only been accessed once, its reuse distance is considered infinite. if i'm reading the hdfs-file-reader.cc and data-cache.cc code correctly, it looks like it calls Lookup twice whenever there is a cache miss. one call occurs when trying to read the data: HdfsFileReader::ReadDataCache --> DataCache::Partition::Lookup --> Cache::Lookup if that results in a cache miss, it calls tries to insert the data into the cache resulting in another call to Lookup: HdfsFileReader::WriteDataCache --> DataCache::Partition::Store --> Cache::Lookup so pretty much every entry in the cache will have a non-infinite reuse distance will this cause problems for LIRS? http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@150 PS18, Line 150: ref_count nit: could you add some docs for 'ref_count' its not clear to me when it needs to be incremented http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@411 PS18, Line 411: HandleTable table_; nit: add docs -- To view, visit http://gerrit.cloudera.org:8080/15306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I670fa4b2b7c93998130dc4e8b2546bb93e9a84f8 Gerrit-Change-Number: 15306 Gerrit-PatchSet: 18 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 23 Mar 2020 18:42:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5570/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 18:26:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/15378 )
Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol
..
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@656
PS11, Line 656: if self.max_tries == 1:
: return 0
: ratio = float(num_tries) / self.max_tries
: if ratio < 0.3:
: return 0.1
: elif ratio < 0.6:
: return 0.3
: return 2
> if i'm reading this correctly, the first retry will have num_tries = 1, so
The current logic basically has following (We have 3 tries total including the
first one):
1s try
if fail
2nd try
if fail
3rd try
if fail return error
It's probably not that robust if someone uses 10 tries. But, we do need to be
able to cap off the sleep time to a reasonable duration. I will think about a
more robust function.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@697
PS11, Line 697: self.close_query(set_all_handle)
> isn't this already retried?
Also, this is redundant, since we already close the query in 'finally' block. I
will remove this statement.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@701
PS11, Line 701: except Exception, e:
> wont this retry TApplicationException still?
It does, but we seem to be overusing RPCException for a variety of exceptions.
If we get a TApplicationException, or a HTTP error code, or even if the impala
server returns an error response, we throw RPCException.
We could raise a different type of exception from _do_hs2_rpc, if we get a
TApplicationException? But, that will also require changing handling the new
type of exception in the impala_shell.py.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@935
PS11, Line 935: rpc
> you might want to document that this should be a python function and not a
Done
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py@136
PS10, Line 136:
> flake8: E501 line too long (91 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py@237
PS10, Line 237:
> flake8: E501 line too long (91 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py@302
PS11, Line 302: output = capsys.readouterr()[0].splitlines()
: assert output[0] == ("Caught exception HTTP code 502:
Injected Fault, "
: "type= in GetLog. Num
remaining tries: 2")
> since this pattern is duplicated in several places, i think it would make s
Done
--
To view, visit http://gerrit.cloudera.org:8080/15378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5
Gerrit-Change-Number: 15378
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Mon, 23 Mar 2020 18:16:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5520/ -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 18:08:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. IMPALA-6360: Don't show full query statement on Impala WebUI by default In the WebUI's query list the query statements are trimmed, but the full query statement can be seen in the details page. The default statement length is 250 chars and it can be adjusted by the query_stmt_size flag that can be set when the cluster starts. Example: bin/start-impala-cluster.py -s1 --impalad_args --query_stmt_size=10 Testing: -manual testing in the WebUI. -added 'test_query_stmt()' to test_web_pages.py -added test to the custom cluster webserver tests to check without truncate and with custom length truncate Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Reviewed-on: http://gerrit.cloudera.org:8080/15288 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M tests/custom_cluster/test_web_pages.py M tests/webserver/test_web_pages.py M www/queries.tmpl 5 files changed, 62 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 18 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 17 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 17:59:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15533 Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 This patch bumps up CDP_BUILD_NUMBER to 2244454. Due to a recent change in RANGER-2688, we added to ranger-admin-site.xml.template a cookie-related configuration so that Ranger server could be properly started. Testing: Except for a currently known issue of IMPALA-9513, verified that this patch passes the exhaustive tests in the DEBUG build when $USE_CDP_HIVE is false. Change-Id: I7750f73834368c7109965e78b147238fc6316f49 --- M bin/impala-config.sh M testdata/cluster/ranger/ranger-admin-site.xml.template 2 files changed, 13 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15533/1 -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 17:55:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build In CDP build we use Hive3 which has a bug HIVE-22371 (CTAS puts files in the wrong place). It causes failure of newly added test as CTAS creates empty table. Workaround by explicitly creating an external table when hive version >= 3. Tested: Run this test in newest CDP build using job impala-private-basic-parameterized. Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Reviewed-on: http://gerrit.cloudera.org:8080/15520 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_hive_text_codec_interop.py 1 file changed, 17 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 7 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15378 )
Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol
..
Patch Set 11:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@656
PS11, Line 656: if self.max_tries == 1:
: return 0
: ratio = float(num_tries) / self.max_tries
: if ratio < 0.3:
: return 0.1
: elif ratio < 0.6:
: return 0.3
: return 2
if i'm reading this correctly, the first retry will have num_tries = 1, so the
ratio will be 0., so the method will return 0.3. when num_tries
= 2 it will return 2, and when num_tries = 3 it will return 2. not sure if that
is your intention.
i think a better retry policy would be:
* first retry: don't sleep at all
* second retry: sleep 1 second
* third retry: sleep 2 seconds
furthermore, I think this method should be robust enough so that it still
returns a reasonable retry policy if the value of max_retries is changed.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@697
PS11, Line 697: self.close_query(set_all_handle)
isn't this already retried?
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@701
PS11, Line 701: except Exception, e:
wont this retry TApplicationException still?
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@935
PS11, Line 935: rpc
you might want to document that this should be a python function and not a
lambda because the error message include the rpc.__name__
http://gerrit.cloudera.org:8080/#/c/15378/8/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/8/tests/custom_cluster/test_hs2_fault_injection.py@128
PS8, Line 128: OpenSession and CloseImpalaOperation rpcs fail.
> I think the point of having a frequency is so that we have some faults and
oh yeah duh
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py@302
PS11, Line 302: output = capsys.readouterr()[0].splitlines()
: assert output[0] == ("Caught exception HTTP code 502:
Injected Fault, "
: "type= in GetLog. Num
remaining tries: 2")
since this pattern is duplicated in several places, i think it would make sense
to add a dedicated method for it and just pass directly pass in the strings
that should be matched to the output
--
To view, visit http://gerrit.cloudera.org:8080/15378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5
Gerrit-Change-Number: 15378
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Mon, 23 Mar 2020 17:43:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 17:11:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5568/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5569/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 9: This is read for review. I based it on top of my IWYU patch because it drastically improved compile times when modifying tmp-file-mgr.h -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 16:28:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#9). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 752 insertions(+), 211 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/9 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5525/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 16:16:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5567/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 16:10:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
zhaoren...@hotmail.com has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. IMPALA-9428 Add arm64 atomic ops Atomic ops are implemented by asm. Different arc have diffent implementation. Here add arm64 atomic ops implementation. Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd --- A be/src/gutil/atomicops-internals-arm64.h M be/src/gutil/atomicops.h M be/src/gutil/cpu.cc 3 files changed, 479 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/15300/9 -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5523/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:09:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h File be/src/gutil/atomicops-internals-arm64.h: http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@183 PS7, Line 183: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@198 PS7, Line 198: ); line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@198 PS7, Line 198: ); tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 15:58:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
zhaoren...@hotmail.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. IMPALA-9428 Add arm64 atomic ops Atomic ops are implemented by asm. Different arc have diffent implementation. Here add arm64 atomic ops implementation. Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd --- A be/src/gutil/atomicops-internals-arm64.h M be/src/gutil/atomicops.h M be/src/gutil/cpu.cc 3 files changed, 480 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/15300/7 -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9042: Milestone 1: properly scan files that has full ACID schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15395 ) Change subject: IMPALA-9042: Milestone 1: properly scan files that has full ACID schema .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5566/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2afec00c9a5cf87f1d61b5fe52b0085844bcb Gerrit-Change-Number: 15395 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 15:43:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5522/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:30:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 4 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:29:48 + Gerrit-HasComments: No
