[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12438 )
Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..
Patch Set 6: Code-Review+2
(2 comments)
Feel free to carry +2 after the test run finishes.
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java
File fe/src/main/java/org/apache/impala/common/PrintUtils.java:
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java@66
PS5, Line 66: if (value >= GIGA) return new
DecimalFormat(".00G").format(result / GIGA);
> Created an "exact cardinality" method to format the values as suggested. Th
Thanks
http://gerrit.cloudera.org:8080/#/c/12438/6/fe/src/main/java/org/apache/impala/common/PrintUtils.java
File fe/src/main/java/org/apache/impala/common/PrintUtils.java:
http://gerrit.cloudera.org:8080/#/c/12438/6/fe/src/main/java/org/apache/impala/common/PrintUtils.java@156
PS6, Line 156: ,
this is cool, didn't know about this grouping separator.
--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 6
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Wed, 20 Feb 2019 07:12:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f Gerrit-Change-Number: 12020 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (402) Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 20 Feb 2019 06:43:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3801/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f Gerrit-Change-Number: 12020 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (402) Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 20 Feb 2019 06:43:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 20 Feb 2019 05:46:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. IMPALA-8214: Fix bad plan in load_nested.py The previous plan had the larger input on the build side of the join and did a broadcast join, which is very suboptimal. This speeds up data loading on my minicluster - 18s vs 31s and has a more significant impact on a real cluster, where queries execute much faster, the memory requirement is significantly reduced and the data loading can potentially be broken up into fewer chunks. I also considered computing stats on the table to let Impala generate the same plan, but this achieves the same goal more efficiently. Testing: Run core tests. Resource estimates in planner tests changed slightly because of the different distribution of data. Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Reviewed-on: http://gerrit.cloudera.org:8080/12519 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/bin/load_nested.py M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 3 files changed, 12 insertions(+), 12 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3797/ -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 05:24:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla .. Patch Set 2: (1 comment) We need to bump impyla_version in infra/python/deps/compiled-requirements.txt, right? Otherwise looks good. http://gerrit.cloudera.org:8080/#/c/12530/2/tests/query_test/test_cancellation.py File tests/query_test/test_cancellation.py: http://gerrit.cloudera.org:8080/#/c/12530/2/tests/query_test/test_cancellation.py@223 PS2, Line 223: assert any(client.get_state(handle) == 'RUNNING_STATE' or sleep(1) This is cool -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 05:02:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3799/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 20 Feb 2019 04:26:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 20 Feb 2019 04:06:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. IMPALA-8191: Wait for additional breakpad processes during test The Breakpad signal handler forks off a process to write a minidump. During the breakpad tests we send signals to the Impala daemons and then wait for all processes to go away. Prior to this change we did this by waiting on the PID returned by process.get_pid(). It is determined by iterating over psutil.get_pid_list() which is an ordered list of PIDs running on the system. We return the first process in the list with a matching command line. In cases where the PID space rolled over, this could have been the forked off breakpad process and we'd wait on that one. During the subsequent check that all processes are indeed gone, we could then pick up the original Impala daemon that had forked off to write the minidump and was still in the process of shutting down. To fix this, we wait for every process twice. Processes are identified by their command and iterating through them twice makes sure we catch both the original daemon and it's breakpad child. This change also contains improvements to the logging of processes in our tests. This should make it easier to identify similar issues in the future. Testing: I ran the breakpad tests in exhaustive mode. I didn't try to exercise it around a PID roll-over, but we shouldn't see the issue in IMPALA-8191 again. Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Reviewed-on: http://gerrit.cloudera.org:8080/12501 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M tests/common/impala_cluster.py M tests/custom_cluster/test_breakpad.py 2 files changed, 55 insertions(+), 22 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 02:46:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7450. Set thread name during refresh/load operations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11228 ) Change subject: IMPALA-7450. Set thread name during refresh/load operations .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2167/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic7c850d6bb2eedc375ee567c19eb17add335f60c Gerrit-Change-Number: 11228 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 20 Feb 2019 02:29:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/12530/2/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/12530/2/tests/common/impala_connection.py@352 PS2, Line 352: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/12530/2/tests/common/impala_connection.py@352 PS2, Line 352: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/12530/2/tests/query_test/test_cancellation.py File tests/query_test/test_cancellation.py: http://gerrit.cloudera.org:8080/#/c/12530/2/tests/query_test/test_cancellation.py@226 PS2, Line 226: r flake8: F841 local variable 'resp' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 02:18:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12530 Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla .. IMPALA-8158: Retrieve thrift profiles through Impyla This change uses an updated version of Impyla to retrieve thrift profiles through the HS2 api. The companion change in Impyla is PR #333: https://github.com/cloudera/impyla/pull/333 Unfortunately, some of the current usages of get_thrift_profile rely on the Beeswax query states and the ImpylaHS2Connection does not have the required functionality yet. We will have to update these in a future change, once we unified the query states. This change also adds a self-contained test for IMPALA-2063 Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e --- M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/query_test/test_cancellation.py M tests/query_test/test_observability.py 5 files changed, 72 insertions(+), 71 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/12530/2 -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-7450. Set thread name during refresh/load operations
Paul Rogers has uploaded a new patch set (#3) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/11228 ) Change subject: IMPALA-7450. Set thread name during refresh/load operations .. IMPALA-7450. Set thread name during refresh/load operations This adds a small utility class for annotating the current thread's name during potentially long-running operations such as refresh/load. With this change, jstack output now includes useful thread names like: During startup: "main [invalidating metadata - 128/428 dbs complete]" While loading a fresh table: "pool-4-thread-12 [Loading metadata for: foo_db.foo_table] [Loading metadata for all partition(s) of foo_db.foo_table]" Pool refreshing metadata for a particular path: "pool-23-thread-5 [Refreshing file metadata for path: hdfs://nameservice1/path/to/partdir..." Tests: Verified the patch manually by jstacking a catalogd while performing some workload. Also added a simple unit test to verify the thread renaming behavior. Change-Id: Ic7c850d6bb2eedc375ee567c19eb17add335f60c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java A fe/src/main/java/org/apache/impala/util/ThreadNameAnnotator.java A fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java 5 files changed, 291 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/11228/3 -- To view, visit http://gerrit.cloudera.org:8080/11228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic7c850d6bb2eedc375ee567c19eb17add335f60c Gerrit-Change-Number: 11228 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7450. Set thread name during refresh/load operations
Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/11228 )
Change subject: IMPALA-7450. Set thread name during refresh/load operations
..
Patch Set 2:
(6 comments)
Addressed review comments and rebased.
http://gerrit.cloudera.org:8080/#/c/11228/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11228/2//COMMIT_MSG@24
PS2, Line 24: This patch is tricky to automate tests for, but I verified it
manually
: by jstacking a catalogd while performing some workload. Also
added a
: simple unit test to verify the thread renaming behavior
> Can be removed I guess.
Done
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@301
PS2, Line 301: long end;
> Annotate here? This is one of those common interesting entry points for tab
Done
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2185
PS2, Line 2185: private TGetPartialCatalogObjectResponse
doGetPartialCatalogObject(
> This is another interesting entry point RPC for catalog v2 stuff. Add some
Done
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java
File fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java:
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java@24
PS2, Line 24: public class ThreadNameAnnotatorTest {
> great test :-)
Thanks. Handy little item from the bag-o-tricks...
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java@44
PS2, Line 44: wait();
> nit: I hope nothing hangs here forever due to a faulty test? Add a largish
Done
http://gerrit.cloudera.org:8080/#/c/11228/2/fe/src/test/java/org/apache/impala/util/ThreadNameAnnotatorTest.java@107
PS2, Line 107: public void testExternalRename() throws InterruptedException {
> nit: Add a doc of what it does? Probably difficult to understand without an
Done
--
To view, visit http://gerrit.cloudera.org:8080/11228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7c850d6bb2eedc375ee567c19eb17add335f60c
Gerrit-Change-Number: 11228
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 20 Feb 2019 02:12:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2166/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:49:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3798/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 20 Feb 2019 01:43:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 20 Feb 2019 01:43:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12521 ) Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2165/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af Gerrit-Change-Number: 12521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3797/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:27:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 6: Code-Review+2 Made a cosmetic improvement, carrying Tim's +2 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:27:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:27:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12524 to look at the new patch set (#6). Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. IMPALA-8209: Include fragment instance ID in memz/ breakdown The change for IMPALA-7694 had accidentally removed the fragment instance ID from the memz/ breakdown. This change puts it back and adds a test to make sure it's there. This change also pads query IDs with zeros when printing them in the backend. Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 --- M be/src/runtime/runtime-state.cc M be/src/util/debug-util.cc M tests/webserver/test_web_pages.py 3 files changed, 18 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12524/6 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. Patch Set 3: Code-Review+2 Thanks for taking this on. Looks good. -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 20 Feb 2019 01:22:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2164/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 01:22:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12438 ) Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2163/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12438 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c Gerrit-Change-Number: 12438 Gerrit-PatchSet: 6 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Wed, 20 Feb 2019 01:05:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Hello Thomas Marshall, David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12521 to look at the new patch set (#6). Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. IMPALA-6662: Make stress test resilient to hangs due to client crashes Thanks to Sailesh Mukil for the initial version of this patch. The concurrent_select.py process starts multiple sub processes (called query runners), to run the queries. It also starts 2 threads called the query producer thread and the query consumer thread. The query producer thread adds queries to a query queue and the query consumer thread pulls off the queue and feeds the queries to the query runners. The query runner, once it gets queries, does the following: ... with _submit_query_lock: increment(num_queries_started) run_query()# One runner crashes here. increment(num_queries_finished) ... One of the runners crash inside run_query(), thereby never incrementing num_queries_finished. Another thread that's supposed to check for memory leaks (but actually doesn't), periodically acquires '_submit_query_lock' and waits for the number of running queries to reach 0 before releasing the lock. However, in the above case, the number of running queries will never reach 0 because one of the query runners hasn't incremented 'num_queries_finished' and exited. Therefore, the poll_mem_usage() function will hold the lock indefinitely, causing no new queries to be submitted, nor the stress test to complete running. This patch fixes the problem by changing the global trackers of num_queries_started and num_queries_finished, etc. to a per QueryRunner basis. Anytime we want to find the total number of queries started/finished/cancelled, etc., we aggregate the values from all the runners. We synchronize access by adding a new lock called the _query_runners_lock. In _wait_for_test_to_finish(), we periodically check if a QueryRunner has died, and if it has, we make sure to update the num_queries_finished to num_queries_started, since it may have died before updating the 'finished' value, and we also count the error. Other changes: * Boilerplate code is reduced by storing all metrics in a dictionary keyed by the metric name, instead of stamping out the code for 10+ variables. * Added more comments and debug strings * Reformatted some code. Testing: Ran the stress test with the new patch locally and against a cluster. Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af --- M tests/stress/concurrent_select.py 1 file changed, 222 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/12521/6 -- To view, visit http://gerrit.cloudera.org:8080/12521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af Gerrit-Change-Number: 12521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2162/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 00:53:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 3: Code-Review+2 (1 comment) Addressed Tim's last comment, carrying his +2. http://gerrit.cloudera.org:8080/#/c/12524/3/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/3/tests/webserver/test_web_pages.py@124 PS3, Line 124: memz_breakdown = self.get_debug_page(self.MEMZ_URL)['detailed'] > I think everything except close_query needs to be inside the try block - we You're right. Fixed it. m( -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 00:42:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12524 to look at the new patch set (#4). Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. IMPALA-8209: Include fragment instance ID in memz/ breakdown The change for IMPALA-7694 had accidentally removed the fragment instance ID from the memz/ breakdown. This change puts it back and adds a test to make sure it's there. This change also pads query IDs with zeros when printing them in the backend. Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 --- M be/src/runtime/runtime-state.cc M be/src/util/debug-util.cc M tests/webserver/test_web_pages.py 3 files changed, 19 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12524/4 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3796/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 00:42:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN
Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/12438 )
Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..
Patch Set 5:
(2 comments)
Addressed comments. Found one more cardinality field to abbreviate. Because the
EXPLAIN output changed again, need to rerun all the pre-review tests. Will post
an update when that completes.
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java
File fe/src/main/java/org/apache/impala/common/PrintUtils.java:
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java@64
PS5, Line 64: if (value == -1) return "unavailable";
> Shouldn't we instead use printCardinality() or some other helper instead of
Done
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java@66
PS5, Line 66: if (value >= TERA) return new
DecimalFormat(".00T").format(result / TERA);
> Curious if we should be consistent with the backend metrics which print wit
Created an "exact cardinality" method to format the values as suggested. Then,
I went in search of which methods should use the "estimated" cardinality format
vs. the "exact" cardinality format. I found that, in the planner, all
cardinalities are estimates.
This rounding shows at least three digits of precision, so if a number is, say,
1, 12 or 123, we'll see the exact number. But, if the number is large, we'll
see an abbreviation, which is about as accurate as the underlying estimate (and
easier to read.)
Can add the extra field later if we discover a place where we have exactly
accurate values.
--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Wed, 20 Feb 2019 00:28:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12438 to look at the new patch set (#6). Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN .. IMPALA-8181: Abbreviate row counts in EXPLAIN A recent fix added node cardinality to the standard EXPLAIN output, displaying a large number like 123456780 as 123.46M. This patch applies the same fix to the remaining row count numbers: metadata, extrapolated rows, etc. Tests: * Rebased PlannerTest .test files as needed for the new row count format. * Reran all tests to check for dependencies on the old format. Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c --- M fe/src/main/java/org/apache/impala/common/PrintUtils.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-planner/queries/PlannerTest/union.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_stats_extrapolation.py M tests/metadata/test_stats_extrapolation.py 20 files changed, 546 insertions(+), 527 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12438/6 -- To view, visit http://gerrit.cloudera.org:8080/12438 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c Gerrit-Change-Number: 12438 Gerrit-PatchSet: 6 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 3: Code-Review+2 (1 comment) Thanks for fixing the padding, that should make things more consistent http://gerrit.cloudera.org:8080/#/c/12524/3/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/3/tests/webserver/test_web_pages.py@124 PS3, Line 124: memz_breakdown = self.get_debug_page(self.MEMZ_URL)['detailed'] I think everything except close_query needs to be inside the try block - we don't want it to run if wait_for_state throws an exception, right? -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 20 Feb 2019 00:32:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2161/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 20 Feb 2019 00:22:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12524 to look at the new patch set (#3). Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. IMPALA-8209: Include fragment instance ID in memz/ breakdown The change for IMPALA-7694 had accidentally removed the fragment instance ID from the memz/ breakdown. This change puts it back and adds a test to make sure it's there. This change also pads query IDs with zeros when printing them in the backend. Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 --- M be/src/runtime/runtime-state.cc M be/src/util/debug-util.cc M tests/webserver/test_web_pages.py 3 files changed, 18 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12524/3 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. Patch Set 2: Code-Review+2 I think get_pids() works better here. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 20 Feb 2019 00:03:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. Patch Set 3: Code-Review+2 Rebased, carrying Phil's +2 -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 20 Feb 2019 00:08:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12501 ) Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3795/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 20 Feb 2019 00:08:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. Patch Set 12: Code-Review+2 (1 comment) Looks great! http://gerrit.cloudera.org:8080/#/c/12020/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/12020/12/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2637 PS12, Line 2637: globalState_.maskedPrivilegeReqs.add(Pair.create(privReq, authErrorMsg_)); Nice! -- To view, visit http://gerrit.cloudera.org:8080/12020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f Gerrit-Change-Number: 12020 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (402) Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 20 Feb 2019 00:06:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12529 ) Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into .. Patch Set 1: Balazs, A generated google doc was shared with you for review. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/12529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657 Gerrit-Change-Number: 12529 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 20 Feb 2019 00:03:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12529 Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into .. IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657 --- M docs/topics/impala_file_formats.xml 1 file changed, 176 insertions(+), 99 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/12529/1 -- To view, visit http://gerrit.cloudera.org:8080/12529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657 Gerrit-Change-Number: 12529 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12529 ) Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/223/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657 Gerrit-Change-Number: 12529 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 20 Feb 2019 00:02:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8191: Wait for additional breakpad processes during test
Hello Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12501 to look at the new patch set (#2). Change subject: IMPALA-8191: Wait for additional breakpad processes during test .. IMPALA-8191: Wait for additional breakpad processes during test The Breakpad signal handler forks off a process to write a minidump. During the breakpad tests we send signals to the Impala daemons and then wait for all processes to go away. Prior to this change we did this by waiting on the PID returned by process.get_pid(). It is determined by iterating over psutil.get_pid_list() which is an ordered list of PIDs running on the system. We return the first process in the list with a matching command line. In cases where the PID space rolled over, this could have been the forked off breakpad process and we'd wait on that one. During the subsequent check that all processes are indeed gone, we could then pick up the original Impala daemon that had forked off to write the minidump and was still in the process of shutting down. To fix this, we wait for every process twice. Processes are identified by their command and iterating through them twice makes sure we catch both the original daemon and it's breakpad child. This change also contains improvements to the logging of processes in our tests. This should make it easier to identify similar issues in the future. Testing: I ran the breakpad tests in exhaustive mode. I didn't try to exercise it around a PID roll-over, but we shouldn't see the issue in IMPALA-8191 again. Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca --- M tests/common/impala_cluster.py M tests/custom_cluster/test_breakpad.py 2 files changed, 55 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12501/2 -- To view, visit http://gerrit.cloudera.org:8080/12501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia4dcc5fecb9b5f38ae1504aae40f099837cf1bca Gerrit-Change-Number: 12501 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-8207: Fix query loading for perf and stress tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12503 ) Change subject: IMPALA-8207: Fix query loading for perf and stress tests .. IMPALA-8207: Fix query loading for perf and stress tests Problems with perf queries (run-workload.py): - TPCH picks up stress test specific queries (TPCH-AGG1/2/3) - TPCDS picks up queries that were intended just to validate that data was loaded properly but that aren't interesting from a perf perspective (TPCDS-COUNT-) - TPCDS picks up both decimal_v1 and decimal_v2 queries. This is mostly harmless as for queries with matching names only one gets run but it causes some queries with mismatched names to be run twice (TPCDS-Q39-1/2 vs. TPCDS-Q39.1/2) Problems with stress queries (concurrent_select.py): - TPCDS fails to pick up Q22A as it does not use the decimal_v2 queries, even though decimal_v2 is the default now. This problem is exacerbated by the fact that the two scripts have different code paths for selecting the queries, so in the past changes that were made to one path were not always made to the other. This patch merges the two paths to reduce code duplication and prevent these sorts of issues in the future, and fixes the above issues. One complication is that historically the stress test has used query names in the form 'q1' whereas the perf test has used query names in the form 'TPCH-Q1'. This patch standardizes on using 'TPCH-Q1'. Testing: - Added a test that checks that the perf tests pick up the expected number of queries. - Manually ran the scripts and verified that the correct queries are selected. Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Reviewed-on: http://gerrit.cloudera.org:8080/12503 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-1.test M testdata/workloads/tpcds/queries/tpcds-decimal_v2-q39-2.test A tests/infra/test_perf_infra.py M tests/infra/test_stress_infra.py M tests/performance/workload.py M tests/stress/concurrent_select.py M tests/util/parse_util.py M tests/util/test_file_parser.py 8 files changed, 107 insertions(+), 63 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8207: Fix query loading for perf and stress tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12503 ) Change subject: IMPALA-8207: Fix query loading for perf and stress tests .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 22:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12519 ) Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2160/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 19 Feb 2019 22:29:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix misspelling in stress test code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12520 ) Change subject: Fix misspelling in stress test code .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 19 Feb 2019 22:09:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix misspelling in stress test code
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12520 ) Change subject: Fix misspelling in stress test code .. Fix misspelling in stress test code Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Reviewed-on: http://gerrit.cloudera.org:8080/12520 Reviewed-by: David Knupp Tested-by: Impala Public Jenkins --- M tests/stress/concurrent_select.py 1 file changed, 7 insertions(+), 7 deletions(-) Approvals: David Knupp: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8214: Fix bad plan in load nested.py
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12519 Change subject: IMPALA-8214: Fix bad plan in load_nested.py .. IMPALA-8214: Fix bad plan in load_nested.py The previous plan had the larger input on the build side of the join and did a broadcast join, which is very suboptimal. This speeds up data loading on my minicluster - 18s vs 31s and has a more significant impact on a real cluster, where queries execute much faster, the memory requirement is significantly reduced and the data loading can potentially be broken up into fewer chunks. I also considered computing stats on the table to let Impala generate the same plan, but this achieves the same goal more efficiently. Testing: Run core tests. Resource estimates in planner tests changed slightly because of the different distribution of data. Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb --- M testdata/bin/load_nested.py M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test 3 files changed, 12 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/12519/3 -- To view, visit http://gerrit.cloudera.org:8080/12519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I55e0ca09590a90ba530efe4e8f8bf587dde3eeeb Gerrit-Change-Number: 12519 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5031: oversized signed shifts are undefined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12514 ) Change subject: IMPALA-5031: oversized signed shifts are undefined .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7debbd1ca5bd3ae640701ce0e95a12c5059abfd7 Gerrit-Change-Number: 12514 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 21:39:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: oversized signed shifts are undefined
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12514 )
Change subject: IMPALA-5031: oversized signed shifts are undefined
..
IMPALA-5031: oversized signed shifts are undefined
Standard section [expr.shift] says that E1 << E2 is undefined if E1 is
of signed type and the result cannot be represented in the
corresponding unsigned type. We can't simply change 1 << bit_width to
1u << bit_width, though, becuase it is the second argument of the
modulo operator, and following [expr.mul], "If the second operand of /
or % is zero the behavior is undefined."
This expression is tripped in RleTest.ValueSkippingFuzzy, with the
following backtrace:
util/rle-test.cc:304:29: runtime error: shift exponent 32 is too large
for 32-bit type 'int'
#0 RleTest::MakeRandomSequence(unsigned int, int, int, int)::
{lambda(int)#1}::operator()(int) const util/rle-test.cc:304:29
#1 RleTest::MakeRandomSequence(unsigned int, int, int, int)
util/rle-test.cc:315:15
#2 RleTest_ValueSkippingFuzzy_Test::TestBody()
util/rle-test.cc:392:25
Change-Id: I7debbd1ca5bd3ae640701ce0e95a12c5059abfd7
Reviewed-on: http://gerrit.cloudera.org:8080/12514
Reviewed-by: Tim Armstrong
Tested-by: Impala Public Jenkins
---
M be/src/util/rle-test.cc
1 file changed, 1 insertion(+), 0 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/12514
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7debbd1ca5bd3ae640701ce0e95a12c5059abfd7
Gerrit-Change-Number: 12514
Gerrit-PatchSet: 3
Gerrit-Owner: Jim Apple
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [DOCS] A typo was fixed in impala dedicated coordinator.xml
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12525 ) Change subject: [DOCS] A typo was fixed in impala_dedicated_coordinator.xml .. [DOCS] A typo was fixed in impala_dedicated_coordinator.xml Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Reviewed-on: http://gerrit.cloudera.org:8080/12525 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_dedicated_coordinator.xml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Gerrit-Change-Number: 12525 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] A typo was fixed in impala dedicated coordinator.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12525 ) Change subject: [DOCS] A typo was fixed in impala_dedicated_coordinator.xml .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/222/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Gerrit-Change-Number: 12525 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Feb 2019 20:53:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2158/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f Gerrit-Change-Number: 12020 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (402) Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 19 Feb 2019 20:51:32 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A typo was fixed in impala dedicated coordinator.xml
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12525 ) Change subject: [DOCS] A typo was fixed in impala_dedicated_coordinator.xml .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Gerrit-Change-Number: 12525 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Feb 2019 20:51:14 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A typo was fixed in impala dedicated coordinator.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12525 ) Change subject: [DOCS] A typo was fixed in impala_dedicated_coordinator.xml .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/222/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Gerrit-Change-Number: 12525 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Feb 2019 20:51:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12521 ) Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2159/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af Gerrit-Change-Number: 12521 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 20:50:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A typo was fixed in impala dedicated coordinator.xml
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12525 Change subject: [DOCS] A typo was fixed in impala_dedicated_coordinator.xml .. [DOCS] A typo was fixed in impala_dedicated_coordinator.xml Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e --- M docs/topics/impala_dedicated_coordinator.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/12525/1 -- To view, visit http://gerrit.cloudera.org:8080/12525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic71fb69e9b59ce82f204ed569089d5ec8dfee66e Gerrit-Change-Number: 12525 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Fredy Wijaya has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. IMPALA-7917 (Part 1): Decouple Sentry from Impala The first part of this patch is to provide an initial work to decouple Sentry from Impala by creating a generic authorization provider interface that Sentry implements. The idea is to allow more authorization providers in the future. The patch updates the following: - Renamed Authorizeable to Authorizable to fix typographical error. - Moved any clases that uses Sentry specific code to org.apache.impala.authorization.sentry package and created interfaces when necessary. - Moved all generic authorization related classes to org.apache.impala.authorization package. - Minor clean up on authorization related code. In this patch, switching the authorization provider implementation still requires updating the code in many different places. A follow up patch will make it easy to switch an authorization provider implementation. This patch has no functionality change. Testing: - Ran all FE tests - Ran all E2E authorization tests Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f --- M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java R fe/src/main/java/org/apache/impala/authorization/Authorizable.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java R fe/src/main/java/org/apache/impala/authorization/AuthorizationException.java R fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java C fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java D fe/src/main/java/org/apache/impala/authorization/AuthorizeableColumn.java D fe/src/main/java/org/apache/impala/authorization/AuthorizeableUri.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequest.java M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java A fe/src/main/java/org/apache/impala/authorization/sentry/ImpalaAction.java R fe/src/main/java/org/apache/impala/authorization/sentry/ImpalaActionFactory.java R fe/src/main/java/org/apache/impala/authorization/sentry/ImpalaPrivilegeModel.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthProvider.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizable.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableColumn.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableDb.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFn.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableServer.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableTable.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableUri.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryConfig.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyReaderException.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryUnavailableException.java R fe/src/main/java/org/apache/impala/authorization/sentry/SentryUtil.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12521 ) Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. Patch Set 5: This is a rebased version of https://gerrit.cloudera.org/#/c/9635/ with some additional cleanup and a bugfix for the timeout logic. -- To view, visit http://gerrit.cloudera.org:8080/12521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af Gerrit-Change-Number: 12521 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 20:08:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Tim Armstrong has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12521 ) Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. IMPALA-6662: Make stress test resilient to hangs due to client crashes Thanks to Sailesh Mukil for the initial version of this patch. The concurrent_select.py process starts multiple sub processes (called query runners), to run the queries. It also starts 2 threads called the query producer thread and the query consumer thread. The query producer thread adds queries to a query queue and the query consumer thread pulls off the queue and feeds the queries to the query runners. The query runner, once it gets queries, does the following: ... with _submit_query_lock: increment(num_queries_started) run_query()# One runner crashes here. increment(num_queries_finished) ... One of the runners crash inside run_query(), thereby never incrementing num_queries_finished. Another thread that's supposed to check for memory leaks (but actually doesn't), periodically acquires '_submit_query_lock' and waits for the number of running queries to reach 0 before releasing the lock. However, in the above case, the number of running queries will never reach 0 because one of the query runners hasn't incremented 'num_queries_finished' and exited. Therefore, the poll_mem_usage() function will hold the lock indefinitely, causing no new queries to be submitted, nor the stress test to complete running. This patch fixes the problem by changing the global trackers of num_queries_started and num_queries_finished, etc. to a per QueryRunner basis. Anytime we want to find the total number of queries started/finished/cancelled, etc., we aggregate the values from all the runners. We synchronize access by adding a new lock called the _query_runners_lock. In _wait_for_test_to_finish(), we periodically check if a QueryRunner has died, and if it has, we make sure to update the num_queries_finished to num_queries_started, since it may have died before updating the 'finished' value, and we also count the error. Other changes: * Boilerplate code is reduced by storing all metrics in a dictionary keyed by the metric name, instead of stamping out the code for 10+ variables. * Added more comments and debug strings * Reformatted some code. Testing: Ran the stress test with the new patch locally and against a cluster. Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af --- M tests/stress/concurrent_select.py 1 file changed, 222 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/12521/5 -- To view, visit http://gerrit.cloudera.org:8080/12521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I525bf13e0f3dd660c0d9f5c2bf6eb292e7ebb8af Gerrit-Change-Number: 12521 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7917 (Part 1): Decouple Sentry from Impala
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12020 ) Change subject: IMPALA-7917 (Part 1): Decouple Sentry from Impala .. Patch Set 12: Rebased to fix merge conflicts. Paul/Bharath, can you take a look at this again? -- To view, visit http://gerrit.cloudera.org:8080/12020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1fd1df0b38ddd7cfa41299e95f5827f8a9e9c1f Gerrit-Change-Number: 12020 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (402) Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 19 Feb 2019 20:07:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12524/2/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/2/tests/webserver/test_web_pages.py@129 PS2, Line 129: self.client.cancel(query_handle) Could maybe use finally to run this on both code paths. Should this be close_query() instead of cancel() as well to unregister the query? -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 19:42:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 19:15:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12524/1/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/1/tests/webserver/test_web_pages.py@124 PS1, Line 124: self.client.cancel(query_handle) > Shouldn't we cancel the query after getting the memz output? Good point, fixed. Looks like the cancelation was slow enough to work in my testing. :( -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:58:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12524/2/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/2/tests/webserver/test_web_pages.py@123 PS2, Line 123: e flake8: E722 do not use bare except' -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:58:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2156/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:57:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12524 to look at the new patch set (#2). Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. IMPALA-8209: Include fragment instance ID in memz/ breakdown The change for IMPALA-7694 had accidentally removed the fragment instance ID from the memz/ breakdown. This change puts it back and adds a test to make sure it's there. Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 --- M be/src/runtime/runtime-state.cc M tests/webserver/test_web_pages.py 2 files changed, 19 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12524/2 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12438 )
Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java
File fe/src/main/java/org/apache/impala/common/PrintUtils.java:
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java@64
PS5, Line 64: if (value == -1) return "unavailable";
Shouldn't we instead use printCardinality() or some other helper instead of
doing it here?
http://gerrit.cloudera.org:8080/#/c/12438/5/fe/src/main/java/org/apache/impala/common/PrintUtils.java@66
PS5, Line 66: if (value >= TERA) return new
DecimalFormat(".00T").format(result / TERA);
Curious if we should be consistent with the backend metrics which print with
the following format - pretty-printed-val (actual val)
Ex:
PeakMemoryUsage: 149.50 KB (153088)
- PeakReservation: 8.00 KB (8192)
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 169.50 KB (173568)
- RowsProduced: 1 (1)
I think we probably want exact values in some cases, ex: off-by-one cases in
cardinality etc.
--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Tue, 19 Feb 2019 18:35:32 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Add support for sles12
Hello Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12422 to look at the new patch set (#2). Change subject: Add support for sles12 .. Add support for sles12 We add support for building native-toolchain sles images. Since these images require access to a mirror, they're not built unless the SLES_MIRROR environment variable is set. Change-Id: If8a60e3f4ee94f9a4cd76128b8658802870daca9 --- M Makefile M docker/all/assert-dependencies-present.py M docker/buildall.sh A docker/sles12.df M in-docker.py 5 files changed, 74 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/22/12422/2 -- To view, visit http://gerrit.cloudera.org:8080/12422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If8a60e3f4ee94f9a4cd76128b8658802870daca9 Gerrit-Change-Number: 12422 Gerrit-PatchSet: 2 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Enable publishing on the docker-based build
hector.aco...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12423 ) Change subject: Enable publishing on the docker-based build .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/12423/2/in-docker.py File in-docker.py: http://gerrit.cloudera.org:8080/#/c/12423/2/in-docker.py@57 PS2, Line 57: Maps docker images to BUILD_TARGET_LABELs which is ultimately included : # in the path for each built package > It may be worth mentioning here that Impala/bin/bootstrap_toolchain.py cont Done http://gerrit.cloudera.org:8080/#/c/12423/2/in-docker.py@130 PS2, Line 130: PUBLISH_D > Maybe add the AWS credential variables to the passthrough list, in case the Done -- To view, visit http://gerrit.cloudera.org:8080/12423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32966bf5b5ad32a3caaf2aee0430ea3a67ecb5d6 Gerrit-Change-Number: 12423 Gerrit-PatchSet: 3 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: hector.aco...@cloudera.com Gerrit-Comment-Date: Tue, 19 Feb 2019 18:27:20 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add support for sles12
hector.aco...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/12422 )
Change subject: Add support for sles12
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12422/1/docker/buildall.sh
File docker/buildall.sh:
http://gerrit.cloudera.org:8080/#/c/12422/1/docker/buildall.sh@25
PS1, Line 25: if [[ -n "${SLES_MIRROR:-""}" ]]; then
: BUILD_ARGS+=(--build-arg="SLES_MIRROR=$SLES_MIRROR")
> Could you mention this dependency in the commit message?
Done
--
To view, visit http://gerrit.cloudera.org:8080/12422
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If8a60e3f4ee94f9a4cd76128b8658802870daca9
Gerrit-Change-Number: 12422
Gerrit-PatchSet: 2
Gerrit-Owner: hector.aco...@cloudera.com
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: hector.aco...@cloudera.com
Gerrit-Comment-Date: Tue, 19 Feb 2019 18:27:15 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Initial support for building the toolchain in docker
hector.aco...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12285 ) Change subject: Initial support for building the toolchain in docker .. Patch Set 6: (3 comments) http://gerrit.cloudera.org:8080/#/c/12285/5/docker/redhat6.df File docker/redhat6.df: http://gerrit.cloudera.org:8080/#/c/12285/5/docker/redhat6.df@4 PS5, Line 4: # We need to pin our yum updates to keep old library versions. > Repo pinning seems to be missing from here. Is that intentional? Done http://gerrit.cloudera.org:8080/#/c/12285/5/docker/redhat6.df@6 PS5, Line 6: 's > nit: seems to be redundant with "-y" also present in the docker/redhat/yum- Done http://gerrit.cloudera.org:8080/#/c/12285/5/in-docker.py File in-docker.py: http://gerrit.cloudera.org:8080/#/c/12285/5/in-docker.py@112 PS5, Line 112: passthrough_env > Maybe add the env vars needed for publishing to the S3 bucket: Done -- To view, visit http://gerrit.cloudera.org:8080/12285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If42c9bc06a3d303642eb37dea784b61e2a1f5cc6 Gerrit-Change-Number: 12285 Gerrit-PatchSet: 6 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: hector.aco...@cloudera.com Gerrit-Comment-Date: Tue, 19 Feb 2019 18:27:09 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add support for Ubuntu 18.04
hector.aco...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12286 ) Change subject: Add support for Ubuntu 18.04 .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/12286/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12286/2//COMMIT_MSG@13 PS2, Line 13: ./in-docker.py impala-toolchain-ubuntu18 ./buildall.sh > Ignore this, I saw the other patchset. Done http://gerrit.cloudera.org:8080/#/c/12286/2/docker/ubuntu18.df File docker/ubuntu18.df: http://gerrit.cloudera.org:8080/#/c/12286/2/docker/ubuntu18.df@5 PS2, Line 5: > libtool seems to be missing here; the Kudu build broke for me with: Done -- To view, visit http://gerrit.cloudera.org:8080/12286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie1e63fd41ad293008dec2423a5a361cabadcdb8a Gerrit-Change-Number: 12286 Gerrit-PatchSet: 7 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: hector.aco...@cloudera.com Gerrit-Comment-Date: Tue, 19 Feb 2019 18:27:02 + Gerrit-HasComments: Yes
[native-toolchain-CR] Initial support for building the toolchain in docker
Hello Lars Volker, Laszlo Gaal, Philip Zeyliger, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12285 to look at the new patch set (#6). Change subject: Initial support for building the toolchain in docker .. Initial support for building the toolchain in docker Adds Dockerfiles and supporting scripts to build the toolchain in a docker container. It should now be possible to run any build*.sh script in a supported platform by running: ./in-docker.py impala-toolchain-debian8 "./build.sh gflags 2.2.0-p2" - To build gflags version 2.2.0-p2 in debian8 It is also possible to build the entire toolchain for all platforms by running: make -j $P Change-Id: If42c9bc06a3d303642eb37dea784b61e2a1f5cc6 --- A Makefile A docker/all/assert-dependencies-present.py A docker/all/postinstall.sh A docker/buildall.sh A docker/debian7.df A docker/debian8.df A docker/redhat/CentOS-6.6-Base.repo A docker/redhat/CentOS-7.2.1511-Base.repo A docker/redhat/yum-install A docker/redhat6.df A docker/redhat7.df A docker/ubuntu1204.df A docker/ubuntu1404.df A docker/ubuntu1604.df M functions.sh A in-docker.py 16 files changed, 728 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/85/12285/6 -- To view, visit http://gerrit.cloudera.org:8080/12285 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If42c9bc06a3d303642eb37dea784b61e2a1f5cc6 Gerrit-Change-Number: 12285 Gerrit-PatchSet: 6 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: hector.aco...@cloudera.com
[native-toolchain-CR] Enable publishing on the docker-based build
Hello Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12423 to look at the new patch set (#3). Change subject: Enable publishing on the docker-based build .. Enable publishing on the docker-based build In order to enable publishing from each container this commit: * Adds maven and aws commands * Modifies the upload function to conditionally upload to artifactory/s3 * Generate TOOLCHAIN_ID (if not set) and share it with all the containers. This is done in the Makefile so that all containers get the same TOOLCHAIN_ID. * Generate the BUILD_TARGET_LABEL based on the docker image name. Change-Id: I32966bf5b5ad32a3caaf2aee0430ea3a67ecb5d6 --- M Makefile M docker/all/assert-dependencies-present.py M docker/debian7.df M docker/debian8.df M docker/redhat6.df M docker/redhat7.df M docker/sles12.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M functions.sh M in-docker.py M init.sh 14 files changed, 185 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/23/12423/3 -- To view, visit http://gerrit.cloudera.org:8080/12423 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32966bf5b5ad32a3caaf2aee0430ea3a67ecb5d6 Gerrit-Change-Number: 12423 Gerrit-PatchSet: 3 Gerrit-Owner: hector.aco...@cloudera.com Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-8207: Fix query loading for perf and stress tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12503 ) Change subject: IMPALA-8207: Fix query loading for perf and stress tests .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3793/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:24:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8207: Fix query loading for perf and stress tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12503 ) Change subject: IMPALA-8207: Fix query loading for perf and stress tests .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:24:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog server web-ui should expose top-n tables with most number of file. The commit includes: Add functions in CatalogUsageMonitor to monitor and report the catalog
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12483 )
Change subject: IMPALA-6897: Catalog server web-ui should expose top-n tables
with most number of file. The commit includes: Add functions in
CatalogUsageMonitor to monitor and report the catalog usage of the tables
have the most number of files Add methods to ex
..
Patch Set 3:
(10 comments)
Can you paste a screenshot of how this looks like?
http://gerrit.cloudera.org:8080/#/c/12483/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12483/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-6897: Catalog server web-ui should expose top-n tables with
most
: number of file.
nit: rephrase to limit it to 72 chars.
http://gerrit.cloudera.org:8080/#/c/12483/3//COMMIT_MSG@10
PS3, Line 10: Add functions in CatalogUsageMonitor to monitor and report the
catalog
nit: Indent these better. Also, try to avoid implementation detail and
summarize the functionality that was added?
http://gerrit.cloudera.org:8080/#/c/12483/3/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:
http://gerrit.cloudera.org:8080/#/c/12483/3/be/src/catalog/catalog-server.h@187
PS3, Line 187: .
Update?
http://gerrit.cloudera.org:8080/#/c/12483/3/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/12483/3/common/thrift/JniCatalog.thrift@710
PS3, Line 710: that table has
nit:...in this table.
http://gerrit.cloudera.org:8080/#/c/12483/3/common/thrift/JniCatalog.thrift@711
PS3, Line 711: 4: optional i64 num_files
nit: Clarify that for partitioned tables, this includes file counts across all
the partitions?
http://gerrit.cloudera.org:8080/#/c/12483/3/common/thrift/JniCatalog.thrift@724
PS3, Line 724: most_num_files_tables
rename to something like high_file_count_tables or something? (or any other
better name you can think of)
http://gerrit.cloudera.org:8080/#/c/12483/3/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12483/3/fe/src/main/java/org/apache/impala/catalog/Table.java@152
PS3, Line 152: getNumOfFiles
getNumFiles()?
http://gerrit.cloudera.org:8080/#/c/12483/3/fe/src/main/java/org/apache/impala/catalog/Table.java@168
PS3, Line 168: public void setNumOfFiles(long numOfFiles) {
: numOfFiles_.set(numOfFiles);
: if (!isStoredInImpaladCatalogCache()) {
: CatalogUsageMonitor.INSTANCE.updateMostFilesTables(this);
: }
: }
This does not make sense for non-HDFS tables. Instead do this in HdfsTable
class?
http://gerrit.cloudera.org:8080/#/c/12483/3/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/12483/3/tests/webserver/test_web_pages.py@271
PS3, Line 271: es")
Could you also validate the number of files?
http://gerrit.cloudera.org:8080/#/c/12483/3/tests/webserver/test_web_pages.py@272
PS3, Line 272: def __test_catalog_object(self, db_name, tbl_name):
newline
--
To view, visit http://gerrit.cloudera.org:8080/12483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b
Gerrit-Change-Number: 12483
Gerrit-PatchSet: 3
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Tue, 19 Feb 2019 18:22:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12524 ) Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12524/1/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12524/1/tests/webserver/test_web_pages.py@124 PS1, Line 124: self.client.cancel(query_handle) Shouldn't we cancel the query after getting the memz output? -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 18:17:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8209: Include fragment instance ID in memz/ breakdown
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12524 Change subject: IMPALA-8209: Include fragment instance ID in memz/ breakdown .. IMPALA-8209: Include fragment instance ID in memz/ breakdown The change for IMPALA-7694 had accidentally removed the fragment instance ID from the memz/ breakdown. This change puts it back and adds a test to make sure it's there. Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 --- M be/src/runtime/runtime-state.cc M tests/webserver/test_web_pages.py 2 files changed, 17 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12524/1 -- To view, visit http://gerrit.cloudera.org:8080/12524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I73bf06bf95c88186b16fd03243de9bac946c5cc8 Gerrit-Change-Number: 12524 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] Fix misspelling in stress test code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12520 ) Change subject: Fix misspelling in stress test code .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3792/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 19 Feb 2019 18:09:53 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix misspelling in stress test code
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/12520 ) Change subject: Fix misspelling in stress test code .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 19 Feb 2019 17:45:32 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix misspelling in stress test code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12520 ) Change subject: Fix misspelling in stress test code .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2155/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 19 Feb 2019 17:32:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8207: Fix query loading for perf and stress tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12503 ) Change subject: IMPALA-8207: Fix query loading for perf and stress tests .. Patch Set 2: Code-Review+2 I tested this out a bit in my stress runs and it helps a lot. -- To view, visit http://gerrit.cloudera.org:8080/12503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id1966d6ca8babdda07d47e089b75ba06d0318c0d Gerrit-Change-Number: 12503 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 17:27:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: oversized signed shifts are undefined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12514 ) Change subject: IMPALA-5031: oversized signed shifts are undefined .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3791/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7debbd1ca5bd3ae640701ce0e95a12c5059abfd7 Gerrit-Change-Number: 12514 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 17:24:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: oversized signed shifts are undefined
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12514 ) Change subject: IMPALA-5031: oversized signed shifts are undefined .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12514 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7debbd1ca5bd3ae640701ce0e95a12c5059abfd7 Gerrit-Change-Number: 12514 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 17:16:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6662: Make stress test resilient to hangs due to client crashes
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9635 ) Change subject: IMPALA-6662: Make stress test resilient to hangs due to client crashes .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/9635/6/tests/stress/concurrent_select.py File tests/stress/concurrent_select.py: http://gerrit.cloudera.org:8080/#/c/9635/6/tests/stress/concurrent_select.py@698 PS6, Line 698: timeout = solo_runtime * max(10, query_runner._num_queries_started.value - \ I'm pretty sure this change is the cause of the increased timeouts that Sailesh saw. The problem is that it's using the query-runner-local counts to try to calculate the number of active queries globally. But each runner has only one query at a time, so the second argument to max() is either 0 or 1. The 10x factor here is also kind-of broken anyway since there's no reason queries can't run more than 10x slower under load. -- To view, visit http://gerrit.cloudera.org:8080/9635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c5dc9b8c2fffc471bac2279e348bc1d1fec3b7 Gerrit-Change-Number: 9635 Gerrit-PatchSet: 6 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Nithya Janarthanan Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Feb 2019 16:57:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix misspelling in stress test code
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12520 Change subject: Fix misspelling in stress test code .. Fix misspelling in stress test code Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 --- M tests/stress/concurrent_select.py 1 file changed, 7 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/12520/1 -- To view, visit http://gerrit.cloudera.org:8080/12520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I410c7ad55693f2fdc6bf7edcc4ae3aeb7a408c43 Gerrit-Change-Number: 12520 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2154/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 19 Feb 2019 16:08:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Hello Zoltan Borok-Nagy, Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12168 to look at the new patch set (#16). Change subject: IMPALA-6503: Support reading complex types from ORC .. IMPALA-6503: Support reading complex types from ORC We've supported reading primitive types from ORC files (IMPALA-5717). In this patch we add support for complex types (struct/array/map). In IMPALA-5717, we leverage the ORC lib to parse ORC binaries (data in io buffer read from DiskIoMgr). The ORC lib can materialize ORC column binaries into its representation (orc::ColumnVectorBatch). Then we transform values in orc::ColumnVectorBatch into impala::Tuples in hdfs-orc-scanner. We don't need to do anything about decoding/decompression since they are handled by the ORC lib. Fortunately, the ORC lib already supports complex types, we can still leverage it to support complex types. What we need to add in IMPALA-6503 are two things: 1. Specify which nested columns we need in the form required by the ORC lib (Get list of ORC type ids from tuple descriptors) 2. Transform outputs of ORC lib (nested orc::ColumnVectorBatch) into Impala's representation (Slots/Tuples/RowBatches) To format the materialization, we implement several ORC column readers in hdfs-orc-scanner. Each kind of reader treats a column type and transforms outputs of the ORC lib into tuple/slot values. Tests: * Enable existing tests for complex types (test_nested_types.py, test_tpch_nested_queries.py) for ORC. Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 --- M be/src/exec/CMakeLists.txt M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h A be/src/exec/orc-column-readers.cc A be/src/exec/orc-column-readers.h A be/src/exec/orc-metadata-utils.cc A be/src/exec/orc-metadata-utils.h M be/src/runtime/descriptors.cc M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java A testdata/ComplexTypesTbl/README A testdata/ComplexTypesTbl/nonnullable.orc A testdata/ComplexTypesTbl/nullable.orc M testdata/bin/create-load-data.sh M testdata/bin/load_nested.py M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/complex-types-file-formats.test M testdata/workloads/functional-query/queries/QueryTest/max-nesting-depth.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-limit.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit-single-node.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test M testdata/workloads/tpch_nested/tpch_nested_core.csv M testdata/workloads/tpch_nested/tpch_nested_dimensions.csv M testdata/workloads/tpch_nested/tpch_nested_exhaustive.csv M testdata/workloads/tpch_nested/tpch_nested_pairwise.csv M tests/query_test/test_nested_types.py M tests/query_test/test_tpch_nested_queries.py 29 files changed, 1,889 insertions(+), 462 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/12168/16 -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 16 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
