[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13932 ) Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4689/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Jul 2019 05:53:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13932 )
Change subject: IMPALA-8661 : Add randomized tests to stress
MetastoreEventsProcessor
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1343
PS3, Line 1343: // when multiple partitions are added in HMS they are
all added as one transaction
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java
File fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java:
http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@142
PS3, Line 142: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME,
getNoAuthConnectionStr("binary"));
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/13932/3/fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java@150
PS3, Line 150: return createClient(TestUtils.HIVE_SERVER2_DRIVER_NAME,
getNoAuthConnectionStr("http"));
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce
Gerrit-Change-Number: 13932
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 30 Jul 2019 05:42:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13932 ) Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor .. IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor This change adds a new stress test for MetastoreEventsProcessor. This test randomly executes hive queries to generate a lot of events. The event processor is invoked at random intervals so that a variable batch of events is processed everytime. After each batch is processed, the test checks the status of events processor. By default, on CDH builds the test is configured to run with 8 concurrent Hive clients and each of the client runs 100 random Hive queries. These defaults can be overridden by passing system properties using maven command arguments "-DnumClients" and "-DnumQueriesPerClients". Additionally, the test also creates impala clients which keep issuing refresh table commands on the test databases to make sure that eventProcessor is doing some real work rather than invalidating/refreshing tables which are already incomplete. This test is added as a junit test and uses Hive JDBC to issue the sqls. This is much faster than the end-to-end python test which issues each hive query in a separate beeline sessions which re-establishes the connection every time. The test already found a bug which is caused when a Hive issues a alter table add if not exists partition" query and the partition is not really added since it is preexisting. In such a case the ADD_PARTITION events is still generated but with a empty list of partitions in the events. Such events are now ignored. Notes: 1. Ran the test with defaults. It generates about 2100 events and runs for close to 15 min. This can be changed to a lower value if we see significant increased delay in the test job runtimes. 3. On CDP builds the concurrent hive queries run very slow due to container provisioning time on the minicluster. I have left this as a TODO to investigate. The test runs in single threaded mode with increased number of queries when running against Hive-3 Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java A fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A fe/src/test/java/org/apache/impala/catalog/events/RandomHiveQueryRunner.java A fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java 8 files changed, 1,516 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/13932/3 -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4684/ -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 05:28:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4072/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 04:47:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: widen Thrift enum to placate UBSAN
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13940 ) Change subject: IMPALA-5031: widen Thrift enum to placate UBSAN .. Patch Set 1: I'm a bit reluctant about diverging parquet.thrift from the upstream Apache Parquet thrift. Need to think about whether this is the right way. -- To view, visit http://gerrit.cloudera.org:8080/13940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I48090e8e0c6c6f18bb1ad3c32c1f5fbffc908844 Gerrit-Change-Number: 13940 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jul 2019 04:11:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 04:06:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4688/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 10 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 04:06:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 9: Code-Review+2 (1 comment) carrying forward http://gerrit.cloudera.org:8080/#/c/13868/8/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/13868/8/be/src/scheduling/executor-blacklist.h@41 PS8, Line 41: Maintenan > nit: typo Done -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 04:05:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13868 to look at the new patch set (#9). Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. When a query is scheduled and there is currently some blacklisted executors, a new line 'Blacklisted Executors:' is added to the profile listing the hostnames of all such executors. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added e2e test cases for killing and restarting an impalad. - Manual randomized testing locally with iptables. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-blacklist.cc A be/src/scheduling/executor-blacklist.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_blacklist.py 15 files changed, 735 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13868/9 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4071/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 04:00:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8534: data cache for dockerised tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13934 ) Change subject: IMPALA-8534: data cache for dockerised tests .. IMPALA-8534: data cache for dockerised tests This adds support for the data cache in dockerised clusters in start-impala-cluster.py. It is handled similarly to the log directories - we ensure that a separate data cache directory is created for each container, then mount it at /opt/impala/cache inside the container. This is then enabled by default for the dockerised tests. Testing: Did a dockerised test run. Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Reviewed-on: http://gerrit.cloudera.org:8080/13934 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/jenkins/dockerized-impala-run-tests.sh M bin/start-impala-cluster.py 2 files changed, 26 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Gerrit-Change-Number: 13934 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8534: data cache for dockerised tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13934 ) Change subject: IMPALA-8534: data cache for dockerised tests .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Gerrit-Change-Number: 13934 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 30 Jul 2019 03:59:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 8: Code-Review+2 (1 comment) Spotted a typo but otherwise LGTM. Thanks for adding the comment! http://gerrit.cloudera.org:8080/#/c/13868/8/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/13868/8/be/src/scheduling/executor-blacklist.h@41 PS8, Line 41: Maintence nit: typo -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 03:29:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13868 to look at the new patch set (#8). Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. When a query is scheduled and there is currently some blacklisted executors, a new line 'Blacklisted Executors:' is added to the profile listing the hostnames of all such executors. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added e2e test cases for killing and restarting an impalad. - Manual randomized testing locally with iptables. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-blacklist.cc A be/src/scheduling/executor-blacklist.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_blacklist.py 15 files changed, 735 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13868/8 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/13868/7/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/13868/7/be/src/scheduling/executor-blacklist.h@139 PS7, Line 139: Percent > nit: this is really a multiplier (e.g. multiply by 1.2, not add 1.2%) Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc File be/src/scheduling/executor-blacklist.cc: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc@86 PS6, Line 86: } > This means there is a window for false positives here, right? If so, I thin Added some more comments -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 30 Jul 2019 03:19:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13952 ) Change subject: IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4070/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13952 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I52d07bce88f9332a34bfe2f9b31570203485d544 Gerrit-Change-Number: 13952 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Jul 2019 02:41:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7486: Add specialized estimation scheme for dedicated coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated coordinators .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4681/ -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jul 2019 02:36:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4069/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 02:01:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4068/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 01:55:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13952 )
Change subject: IMPALA-8761 Configuration validation introduced in IMPALA-8559
can be improved
..
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@286
PS1, Line 286: throw new CatalogException(String.format("Found %d
metastore configuration(s) incorrectly"+"" +
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@287
PS1, Line 287: " set. Event processing cannot be started. See
error log for more details.", results.size()));
line too long (108 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@221
PS1, Line 221:* Testing if validateConfigs() is working as expected. If
multiple configuration keys of the metastore are incorrect,
line too long (120 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@222
PS1, Line 222:* then we collect all the configuration validations together
and then present the results together in case of
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@223
PS1, Line 223:* failures so that user can change all the required changes
in one go.This test would assert both for
line too long (104 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@231
PS1, Line 231:
Mockito.when(mockMetastoreEventsProcessor.getConfigValueFromMetastore(configKey,
"")).thenReturn("false");
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@238
PS1, Line 238:
assertTrue(e.getMessage().contains(String.format(errorMessage, majorVersion >=
2 ? 1 : 2)));
line too long (100 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13952
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I52d07bce88f9332a34bfe2f9b31570203485d544
Gerrit-Change-Number: 13952
Gerrit-PatchSet: 1
Gerrit-Owner: Sharanitha Harish
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 30 Jul 2019 01:30:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved
Sharanitha Harish has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13952 Change subject: IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved .. IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved This patch aims to improve the validation of configuration keys from the metastore server. The issue with configuration validation in IMPALA-8559 is that it validates one configuration at a time and fails as soon as there is a validation error. Since there are more than one configuration keys to validate, user may have to restart HMS again and again if there are multiple configuration changes which are needed. This is not a great user experience. This patch presents all of the incorrect configuration validations and results together in case of failures so that user can change all the required changes in one go. Added a test testValidateConfigs() to assert if multiple incorrect values are thrown together according the MetastoreShim.getMajorVersion(). Change-Id: I52d07bce88f9332a34bfe2f9b31570203485d544 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 28 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13952/1 -- To view, visit http://gerrit.cloudera.org:8080/13952 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I52d07bce88f9332a34bfe2f9b31570203485d544 Gerrit-Change-Number: 13952 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. IMPALA-8627: Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Also, fixes the test_observability which fails on catalog-v2 since the profile emits different metadata load events. Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M tests/query_test/test_observability.py 4 files changed, 41 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/5 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4687/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 01:11:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: [WIP] Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: [WIP] Enable catalog-v2 in tests .. IMPALA-8627: [WIP] Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Also, fixes the test_observability which fails on catalog-v2 since the profile emits different metadata load events. Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M tests/query_test/test_observability.py 4 files changed, 41 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/4 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13932 ) Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4680/ -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Jul 2019 01:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8807: fix OPTIMIZE PARTITION KEY SCANS docs
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13949 ) Change subject: IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs .. IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs The docs were inaccurate about the cases in which the optimisation applied. Happily, it actually works in a much wider set of cases. Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Reviewed-on: http://gerrit.cloudera.org:8080/13949 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_optimize_partition_key_scans.xml 1 file changed, 22 insertions(+), 6 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Gerrit-Change-Number: 13949 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8807: fix OPTIMIZE PARTITION KEY SCANS docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13949 ) Change subject: IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/413/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Gerrit-Change-Number: 13949 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 00:50:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4685/ -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jul 2019 00:49:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4686/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jul 2019 00:49:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8785: give debug docker images a different name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13905 ) Change subject: IMPALA-8785: give debug docker images a different name .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4685/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32d2e19cb671beacceebb2642aba01191bd7a244 Gerrit-Change-Number: 13905 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 30 Jul 2019 00:49:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8807: fix OPTIMIZE PARTITION KEY SCANS docs
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13949 ) Change subject: IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Gerrit-Change-Number: 13949 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 00:41:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8807: fix OPTIMIZE PARTITION KEY SCANS docs
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13949 Change subject: IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs .. IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs The docs were inaccurate about the cases in which the optimisation applied. Happily, it actually works in a much wider set of cases. Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d --- M docs/topics/impala_optimize_partition_key_scans.xml 1 file changed, 22 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13949/1 -- To view, visit http://gerrit.cloudera.org:8080/13949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Gerrit-Change-Number: 13949 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8807: fix OPTIMIZE PARTITION KEY SCANS docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13949 ) Change subject: IMPALA-8807: fix OPTIMIZE_PARTITION_KEY_SCANS docs .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/413/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8909b23bfe2b90470fc559fbc01f1e3aa3caa85d Gerrit-Change-Number: 13949 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jul 2019 00:30:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4067/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 6 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 29 Jul 2019 23:55:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: [WIP] Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: [WIP] Enable catalog-v2 in tests .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Jul 2019 23:34:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13945 ) Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. IMPALA-8802: Switch to pgrep for graceful shutdown helper Some places discourage the use of pidof and favor pgrep instead. This change switches usage to the latter in the graceful shutdown helper introduced in IMPALA-8798. Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Reviewed-on: http://gerrit.cloudera.org:8080/13945 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/graceful_shutdown_backends.sh 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13945 ) Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 23:33:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Hello Abhishek Rawat, Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13857 to look at the new patch set (#6). Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. IMPALA-8549: Add support for scanning DEFLATE text files Hadoop tools such as Hive and MapReduce support reading and writing text files compressed using the deflate algorithm. In Hadoop, the zlib library (an implementation of the DEFLATE algorithm) is used to compress text files into .DEFLATE files, which are not in the raw deflate format but rather the zlib format (the zlib library supports three flavors of deflate, and Hadoop is using the flavor that compresses data into deflate with zlib wrappings rather than the raw deflate format) This patch adds support to Impala for scanning .DEFLATE files of tables stored as text. To avoid confusion, it should be noted that although these files have a compression type of DEFLATE in Impala, they should be treated as if their compression type is ZLIB. Testing: There is a pre-existing unit test that validates compressing/decompressing data with compression type DEFLATE. Also, modified existing end-to-end testing that simulates querying files of various formats and compression types. All core tests pass. Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a --- M be/src/exec/hdfs-text-scanner.cc M be/src/exec/hdfs-text-scanner.h M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/functional-query_exhaustive.csv M tests/query_test/test_compressed_formats.py 5 files changed, 19 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/13857/6 -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 6 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Ethan Xue has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/13857/5/be/src/util/codec.cc File be/src/util/codec.cc: http://gerrit.cloudera.org:8080/#/c/13857/5/be/src/util/codec.cc@149 PS5, Line 149: decompressor->reset(new Gz > it seems like .deflate files are considered as DEFAULT compression (https:/ Done -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 6 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 29 Jul 2019 23:13:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: [WIP] Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: [WIP] Enable catalog-v2 in tests .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13933/3/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13933/3/tests/query_test/test_observability.py@296 PS3, Line 296: p flake8: E501 line too long (99 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13933/3/tests/query_test/test_observability.py@306 PS3, Line 306: flake8: E201 whitespace after '[' -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Jul 2019 22:53:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: [WIP] Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: [WIP] Enable catalog-v2 in tests .. IMPALA-8627: [WIP] Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Testing is in progress. Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M tests/query_test/test_observability.py 4 files changed, 39 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/3 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8627: [WIP] Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: [WIP] Enable catalog-v2 in tests .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4684/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Jul 2019 22:53:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8534: data cache for dockerised tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13934 ) Change subject: IMPALA-8534: data cache for dockerised tests .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4683/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Gerrit-Change-Number: 13934 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 29 Jul 2019 21:23:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8534: data cache for dockerised tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13934 ) Change subject: IMPALA-8534: data cache for dockerised tests .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Gerrit-Change-Number: 13934 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 29 Jul 2019 21:23:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13765 ) Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. Patch Set 5: (1 comment) Sorry that it takes time for me to examine all the code paths in the Planner side. This patch makes sense to me. But I need to go deeper to make sure whether this is the best place to add an If-statement for this case. http://gerrit.cloudera.org:8080/#/c/13765/5/fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java File fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java: http://gerrit.cloudera.org:8080/#/c/13765/5/fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java@163 PS5, Line 163: // If the expr is a complex type, return false directly because I think we need to explain why the return type of an expression can be in complex type. In this case the expression is a SlofRef for a complex type column. It's produced by the SubPlan node. In the case of "count(*)", there're no IllegalStateException even without this patch. The reason is that AggregateInfo will generate intermediate TupleDescriptor which doesn't contain the complex type SlotRef. Some relative JIRAs: IMPALA-2875, IMPALA-2138 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Mon, 29 Jul 2019 21:06:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8534: data cache for dockerised tests
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13934 ) Change subject: IMPALA-8534: data cache for dockerised tests .. Patch Set 6: Code-Review+2 Nice. -- To view, visit http://gerrit.cloudera.org:8080/13934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c75d4a5c1eea7a540d051bb175537163dec0e29 Gerrit-Change-Number: 13934 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 29 Jul 2019 21:06:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13883 ) Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 15 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 21:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc@123
PS11, Line 123: // For now, if num_results < batch->num_rows(), we
terminate returning results
: // early.
: if (num_results > 0 && num_results < batch->num_rows()) {
> I think we should avoid CHECK on codepaths that you could get to by togglin
Makes sense. CHECK() seems a bit an overkill. Sorry for the bad suggestion.
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 11
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Jul 2019 20:56:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc@123
PS11, Line 123: // For now, if num_results < batch->num_rows(), we
terminate returning results
: // early.
: if (num_results > 0 && num_results < batch->num_rows()) {
> Yes, its possible if the result cache is enabled, but I think only if the c
I think we should avoid CHECK on codepaths that you could get to by toggling a
query option (since there's nothing to stop a malicious user from toggling it).
It's fine to return an error though.
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 11
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Jul 2019 20:29:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc@123
PS11, Line 123: // For now, if num_results < batch->num_rows(), we
terminate returning results
: // early.
: if (num_results > 0 && num_results < batch->num_rows()) {
> Isn't passing different fetch size possible if the result cache is enabled
Yes, its possible if the result cache is enabled, but I think only if the
client restarts the fetch to the beginning of the result set.
I changed it so it returns "Status::Expected(TErrorCode::NOT_IMPLEMENTED_ERROR,
...)"
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 11
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Jul 2019 20:25:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/13868/7/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/13868/7/be/src/scheduling/executor-blacklist.h@139 PS7, Line 139: Percent nit: this is really a multiplier (e.g. multiply by 1.2, not add 1.2%) http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc File be/src/scheduling/executor-blacklist.cc: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc@86 PS6, Line 86: } > ClusterMembershipMgr uses this to determine if the removed BE would be pres This means there is a window for false positives here, right? If so, I think it would be good to mention it in the header comment. -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 29 Jul 2019 20:23:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13883 to look at the new patch set (#15). Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl .. IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl Improves the encapsulation of RowBatchQueue by the doing the following re-factoring: * Renames RowBatchQueue to BlockingRowBatchQueue which is more indicitive of what the queue does * Re-factors the timers managed by the scan-node into the BlockingRowBatchQueue implementation * Favors composition over inheritance by re-factoring BlockingRowBatchQueue to own a BlockingQueue rather than extending one The re-factoring lays the groundwork for introducing a generic RowBatchQueue that all RowBatch queues inherit from. Adds a new DequeRowBatchQueue which is a simple wrapper around a std::deque that (1) stores unique_ptr to queued RowBatch-es and (2) has a maximum capacity. Implements BufferedPlanRootSink using the new DequeRowBatchQueue. DequeRowBatchQueue is generic enough that replacing it with a SpillableQueue (queue backed by a BufferedTupleStream) should be straightforward. BufferedPlanRootSink is synchronized to protect access to DequeRowBatchQueue since the queue is not thread safe. BufferedPlanRootSink FlushFinal blocks until the consumer thread has processed all RowBatches. This ensures that the coordinator fragment stays alive until all results are fetched, but allows all other fragments to be shutdown immediately. Testing: * Running core tests * Updated tests/query_test/test_result_spooling.py Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/scanner-context.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/blocking-row-batch-queue.cc A be/src/runtime/blocking-row-batch-queue.h A be/src/runtime/deque-row-batch-queue.cc A be/src/runtime/deque-row-batch-queue.h D be/src/runtime/row-batch-queue.cc D be/src/runtime/row-batch-queue.h M be/src/util/blocking-queue.h M tests/query_test/test_result_spooling.py 20 files changed, 550 insertions(+), 205 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13883/15 -- To view, visit http://gerrit.cloudera.org:8080/13883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be Gerrit-Change-Number: 13883 Gerrit-PatchSet: 15 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7486: Add specialized estimation scheme for dedicated coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated coordinators .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 20:21:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7486: Add specialized estimation scheme for dedicated coordinators
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13740 ) Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated coordinators .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13740 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21 Gerrit-Change-Number: 13740 Gerrit-PatchSet: 16 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 20:16:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc@123
PS11, Line 123: // For now, if num_results < batch->num_rows(), we
terminate returning results
: // early.
: if (num_results > 0 && num_results < batch->num_rows()) {
> > the case in which the total number of result rows is not a multiple of ro
Isn't passing different fetch size possible if the result cache is enabled ? In
general, this seems to be making very big assumption about what the client will
pass in. Not sure if this is exactly a good practice.
I suppose my point is that we should fail stop or at least indicate some sort
of failures to the clients for this known to be wrong case in the code. CHECK()
is the fail-stop approach, which seems acceptable as this sink type is not
anticipated to be used. An alternative would be to log this error case and
return an error status to the clients.
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 11
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Jul 2019 19:48:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13947 ) Change subject: IMPALA-7975 : Improve supportability of the automatic invalidate feature .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4064/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib113a5b9458dcf483b183e927544a6c6d46e1af3 Gerrit-Change-Number: 13947 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Jul 2019 19:22:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4063/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 29 Jul 2019 18:53:08 + Gerrit-HasComments: No
[Impala-ASF-CR] This commit adds support to display the metric last-synced-event-id as Catalogd/metrics#evnts page whereas previously it was displayed only on the catalod/events page. Added code to th
Sharanitha Harish has abandoned this change. ( http://gerrit.cloudera.org:8080/13925 ) Change subject: This commit adds support to display the metric last-synced-event-id as Catalogd/metrics#evnts page whereas previously it was displayed only on the catalod/events page. Added code to the MetaStoreEventsProcessorTest class under testEventProcessorMetrics() .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/13925 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I75b966f2b4eaafbcf7d80358f53501bb7ade67e7 Gerrit-Change-Number: 13925 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature
Sharanitha Harish has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13947 Change subject: IMPALA-7975 : Improve supportability of the automatic invalidate feature .. IMPALA-7975 : Improve supportability of the automatic invalidate feature This commit adds support to display the metric last-synced-event-id as Catalogd/metrics#events page whereas previously it was displayed only on the catalod/events page. Added code to the MetaStoreEventsProcessorTest class under testEventProcessorMetrics() and testEventProcessorWhenNotActive() to check 1)If the metric updated as it should when new events were processed. 2)If the metric was not set when the event processor was not active. Change-Id: Ib113a5b9458dcf483b183e927544a6c6d46e1af3 --- M be/src/util/event-metrics.cc M be/src/util/event-metrics.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 35 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/13947/1 -- To view, visit http://gerrit.cloudera.org:8080/13947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib113a5b9458dcf483b183e927544a6c6d46e1af3 Gerrit-Change-Number: 13947 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] This commit adds support to display the metric last-synced-event-id as Catalogd/metrics#evnts page whereas previously it was displayed only on the catalod/events page. Added code to th
Sharanitha Harish has restored this change. ( http://gerrit.cloudera.org:8080/13925 ) Change subject: This commit adds support to display the metric last-synced-event-id as Catalogd/metrics#evnts page whereas previously it was displayed only on the catalod/events page. Added code to the MetaStoreEventsProcessorTest class under testEventProcessorMetrics() .. Restored restoring the change -- To view, visit http://gerrit.cloudera.org:8080/13925 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: I75b966f2b4eaafbcf7d80358f53501bb7ade67e7 Gerrit-Change-Number: 13925 Gerrit-PatchSet: 1 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13932 ) Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4680/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Jul 2019 18:29:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13868 to look at the new patch set (#7). Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. IMPALA-8339: Add local executor blacklist to coordinators This patch adds the concept of a blacklist of executors to the coordinator, which removes executors from consideration for query scheduling. Blacklisting decisions are local to a given coordinator and are not included in statestore updates. The intention is to allow coordinators to be more aggressive about deciding that an exeutor is unhealthy or unavailable, to minimize failed queries in environments where cluster membership may be more variable, rather than having to wait on the statestore heartbeat mechanism to decide that the executor is down. For the first patch, executors will only be blacklisted if the KRPC status for Exec() is an error. Followup work will add blacklisting of executors in more complex scenarios, eg. if an executor appears to be a straggler. When a query is scheduled and there is currently some blacklisted executors, a new line 'Blacklisted Executors:' is added to the profile listing the hostnames of all such executors. Testing: - Added a case to the cluster mgr BE unit test that uses blacklisting. - Added e2e test cases for killing and restarting an impalad. - Manual randomized testing locally with iptables. TODO - Add an e2e test case where an impalad becomes briefly unreachable. - Manual/stress tests on a real cluster. Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/scheduling/CMakeLists.txt M be/src/scheduling/admission-controller.cc M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/executor-blacklist.cc A be/src/scheduling/executor-blacklist.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h A tests/custom_cluster/test_blacklist.py 15 files changed, 733 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13868/7 -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8339: Add local executor blacklist to coordinators
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13868 ) Change subject: IMPALA-8339: Add local executor blacklist to coordinators .. Patch Set 7: (11 comments) http://gerrit.cloudera.org:8080/#/c/13868/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13868/6//COMMIT_MSG@23 PS6, Line 23: a straggler. > Mention the change to the profile in the commit message? Done http://gerrit.cloudera.org:8080/#/c/13868/6//COMMIT_MSG@28 PS6, Line 28: > Are you planning to do these in this change? I was intending do this as follow up work http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/cluster-membership-mgr-test.cc File be/src/scheduling/cluster-membership-mgr-test.cc: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/cluster-membership-mgr-test.cc@350 PS6, Line 350: const int BLACKLIST_TIMEOUT_SLEEP_US = 10; > const int, uppercase? I think that the frequencies could also be closer to Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/cluster-membership-mgr-test.cc@354 PS6, Line 354: EXPECT_EQ(NUM_BACKENDS, backends_.size()); > nit: single line Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/cluster-membership-mgr-test.cc@374 PS6, Line 374: NU > nit: BE, or backend. I tripped over "a be to" :) Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/cluster-membership-mgr-test.cc@415 PS6, Line 415: > nit: extra word? Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.h@39 PS6, Line 39: probation > I think "greylisting" is another term that we could use which sounds less l I think 'probation' is clearer since its a more common term, but willing to change it if others want http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.h@43 PS6, Line 43: When > When Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc File be/src/scheduling/executor-blacklist.cc: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc@86 PS6, Line 86: } > Does this not need to check the timeouts? Or do we assume that the caller c ClusterMembershipMgr uses this to determine if the removed BE would be present in the snapshot's 'executor_groups', i.e. if it was previously blacklisted it won't be in 'executor_groups'. We only transition things to probation during Maitenance() because we need to do the 'unblacklisting' step at the same time to keep things consistent. http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/scheduling/executor-blacklist.cc@191 PS6, Line 191: g re > This can be a variable for clarity. Done http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/statestore/statestore.h File be/src/statestore/statestore.h: http://gerrit.cloudera.org:8080/#/c/13868/6/be/src/statestore/statestore.h@183 PS6, Line 183: time > Please specify the time unit. Done -- To view, visit http://gerrit.cloudera.org:8080/13868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacb6e73b84042c33cd475b82470a975d04ee9b74 Gerrit-Change-Number: 13868 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 29 Jul 2019 18:13:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13945 ) Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4062/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 17:18:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13945 ) Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4679/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 16:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13945 ) Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 16:42:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8802: Switch to pgrep for graceful shutdown helper
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13945 Change subject: IMPALA-8802: Switch to pgrep for graceful shutdown helper .. IMPALA-8802: Switch to pgrep for graceful shutdown helper Some places discourage the use of pidof and favor pgrep instead. This change switches usage to the latter in the graceful shutdown helper introduced in IMPALA-8798. Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 --- M bin/graceful_shutdown_backends.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13945/1 -- To view, visit http://gerrit.cloudera.org:8080/13945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iaa8cc7112002a98c42b4dcfbe30b99ae0cfadf83 Gerrit-Change-Number: 13945 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker
[Impala-ASF-CR] IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13883 )
Change subject: IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and
BufferedPRS impl
..
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13883/11/be/src/exec/buffered-plan-root-sink.cc@123
PS11, Line 123: // For now, if num_results < batch->num_rows(), we
terminate returning results
: // early.
: if (num_results > 0 && num_results < batch->num_rows()) {
> the case in which the total number of result rows is not a multiple of row
> batch size.
That case is still handled properly. The value of num_results is the same as
the fetch size configured by any client. For impala-shell, I believe that is
1024, for other drivers such as Simba the docs state that the
"RowsFetchedPerBlock" ("The maximum number of rows that a query returns at a
time.") is 1.
This condition is only hit if a user configures a fetch size under 1024.
Otherwise, everything works as expected (no partial results) regardless of the
number of rows returned by the actual query.
The only way partial results could be returned is if a user sets the fetch size
to a value above 1024, fetches rows, sets the fetch size to under 1024, and
then fetches rows. However, thats non-default behavior, and a bit of an odd
pattern, which is why I decided to defer fixing this to a future patch.
Do we want to a CHECK for this? That would cause a crash that affects all
Impala users, which is presumably worse than the current behavior?
--
To view, visit http://gerrit.cloudera.org:8080/13883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Gerrit-Change-Number: 13883
Gerrit-PatchSet: 11
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 29 Jul 2019 16:25:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8516: Fix the sha512sum check for the Maven download
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13876 ) Change subject: IMPALA-8516: Fix the sha512sum check for the Maven download .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13876/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13876/1//COMMIT_MSG@21 PS1, Line 21: athe typo: the -- To view, visit http://gerrit.cloudera.org:8080/13876 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8aa4d78b34d60a01fc8f6afc336d66ee5a005cb Gerrit-Change-Number: 13876 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 29 Jul 2019 16:26:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8198: DATE: Read from avro.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13944 ) Change subject: IMPALA-8198: DATE: Read from avro. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4061/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a9d5b93a22cf3a00244037e187f8c145cacc959 Gerrit-Change-Number: 13944 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Jul 2019 13:36:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8198: DATE: Read from avro.
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13944 Change subject: IMPALA-8198: DATE: Read from avro. .. IMPALA-8198: DATE: Read from avro. This change is a follow-up to IMPALA-7368 and adds support for DATE type to the avro scanner. Similarly to parquet, avro uses DATE logical type for dates. DATE logical type annotates an INT32 that stores the number of days since the unix epoch, 1 January 1970. This representation introduces an avro interoperability issue between Impala and older versions of Hive: - Before version 3.1, Hive used Julian calendar to represent dates up to 1582-10-05 and Gregorian calendar for dates starting with 1582-10-15. Dates between 1582-10-05 and 1582-10-15 were lost. - Impala uses proleptic Gregorian calendar, extending the Gregorian calendar backward to dates preceding its official introduction in 1582-10-15. This means that pre-1582-10-15 dates written to an avro table by Hive will be read back incorrectly by Impala. Note that Hive 3.1 switched to proleptic Gregorian calendar too, so for Hive 3.1+ this is no longer an issue. Dependency changes: - BE uses avro 1.7.4-p5 from native-toolchain. Change-Id: I7a9d5b93a22cf3a00244037e187f8c145cacc959 --- M be/src/codegen/gen_ir_descriptions.py M be/src/exec/hdfs-avro-scanner-ir.cc M be/src/exec/hdfs-avro-scanner-test.cc M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/util/avro-util.cc M bin/impala-config.sh M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/util/AvroSchemaConverter.java M fe/src/main/java/org/apache/impala/util/AvroSchemaParser.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M testdata/avro_schema_resolution/create_table.sql M testdata/avro_schema_resolution/file_schema2.avsc M testdata/avro_schema_resolution/records2.avro M testdata/avro_schema_resolution/records2.json M testdata/bad_avro_snap/README A testdata/bad_avro_snap/hive2_pre_gregorian_date.avro A testdata/bad_avro_snap/out_of_range_date.avro M testdata/bin/generate-schema-statements.py M testdata/data/README M testdata/data/date_tbl.avro A testdata/data/date_tbl.orc M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/DataErrorsTest/avro-errors.test M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test A testdata/workloads/functional-query/queries/QueryTest/avro_date.test M testdata/workloads/functional-query/queries/QueryTest/date-fileformat-support.test M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test M testdata/workloads/functional-query/queries/QueryTest/hive2-pre-gregorian-date.test M testdata/workloads/functional-query/queries/QueryTest/out-of-range-date.test M tests/query_test/test_date_queries.py 35 files changed, 338 insertions(+), 77 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13944/1 -- To view, visit http://gerrit.cloudera.org:8080/13944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7a9d5b93a22cf3a00244037e187f8c145cacc959 Gerrit-Change-Number: 13944 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4060/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 22 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 12:08:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 22: (3 comments) http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py@93 PS22, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.Tuple' imported but unused http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py@93 PS22, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.Optional' imported but unused http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py@93 PS22, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.List' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 22 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 11:28:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 22: (3 comments) http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/benchmarks/bit-packing-benchmark.cc File be/src/benchmarks/bit-packing-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/benchmarks/bit-packing-benchmark.cc@39 PS22, Line 39: // BitReader 9.85e+03 9.98e+03 1.01e+04 1X 1X 1X Updated the benchmarks. http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/bit-packing.inline.h File be/src/util/bit-packing.inline.h: http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/bit-packing.inline.h@284 PS22, Line 284: // TODO: Decide if this function is needed. It seems that it is only used in the I think we could remove this method. It is only used in the old benchmark where we compare the original BitReader that unpacks a single value at a time, this method and UnpackValues that can unpack any number of values (not just 32). Many times the last one is faster than Unpack32Values which is not very intuitive so maybe there is some problem with the benchmark, too. So I think we could either remove this function completely, also from the benchmarks, or if we need it there, we could move it to the benchmark file instead of BitPacking. http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/22/be/src/util/vectorised_bit_unpacking_generator.py@1281 PS22, Line 1281: elif bit_width in range(5, 9): Here, with bit width 6, BitScatter is a little bit faster according to my benchmark results. We could branch here in and use BitScatter in that case but that 1-2% is probably not worth it. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 22 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 11:27:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/exec/parquet/parquet-bool-decoder.h M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.generated.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 12 files changed, 6,293 insertions(+), 198 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/22 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 22 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4677/ -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Jul 2019 10:28:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4678/ -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 29 Jul 2019 10:12:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4059/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 08:44:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 21: (3 comments) http://gerrit.cloudera.org:8080/#/c/13807/21/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/21/be/src/util/vectorised_bit_unpacking_generator.py@93 PS21, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.Tuple' imported but unused http://gerrit.cloudera.org:8080/#/c/13807/21/be/src/util/vectorised_bit_unpacking_generator.py@93 PS21, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.Optional' imported but unused http://gerrit.cloudera.org:8080/#/c/13807/21/be/src/util/vectorised_bit_unpacking_generator.py@93 PS21, Line 93: from typing import List, Optional, Tuple flake8: F401 'typing.List' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 08:04:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. IMPALA-8741: Speed up bit unpacking by vectorisation Adding a vectorised implementation to bit unpacking using AVX, AVX2 and BMI2 instructions through compiler intrinsics. Vectorised bit unpacking is implemented for bit widths from 1 to 16. Higher bit widths would not benefit from this vectorisation algorithm. We check at runtime whether the required instructions are available on the CPU and fall back to the scalar implementation if not. The vectorised unpacking functions are in the file be/src/util/bit-packing-vectorized.h, which is generated by the python script in be/src/util/vectorised_bit_unpacking_generator.py. Also adding benchmarks comparing the scalar and the vectorised implementations. Testing: - Added tests for the vectorised unpacking implementations. Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 --- M be/src/benchmarks/bit-packing-benchmark.cc M be/src/exec/parquet/parquet-bool-decoder.h M be/src/util/bit-packing-test.cc A be/src/util/bit-packing-vectorized.generated.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h M be/src/util/cpu-info.cc M be/src/util/cpu-info.h A be/src/util/vectorised_bit_unpacking_generator.py 12 files changed, 6,194 insertions(+), 99 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13807/21 -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 21: (2 comments) http://gerrit.cloudera.org:8080/#/c/13807/19/be/src/util/vectorised_bit_unpacking_generator.py File be/src/util/vectorised_bit_unpacking_generator.py: http://gerrit.cloudera.org:8080/#/c/13807/19/be/src/util/vectorised_bit_unpacking_generator.py@76 PS19, Line 76: > Removed metaclass inheritance, so I'm going to remove this. Accidentally I found a way to use abstract base classes without the metaclass syntax, I should have just looked at the abc module documentation. I simply derive the class from ABC instead of using ABCMeta as a metaclass. This way flake8 has no problem with it. http://gerrit.cloudera.org:8080/#/c/13807/19/be/src/util/vectorised_bit_unpacking_generator.py@89 PS19, Line 89: > Yeah we can ignore this, it will only get re-flagged if someone touches the I added a comment describing why we have those imports. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 08:03:22 + Gerrit-HasComments: Yes
