[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4037/ -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Apr 2019 05:15:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 04:52:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4041/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 04:52:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 04:52:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2833/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 03:34:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Bharath Krishna has posted comments on this change. (
http://gerrit.cloudera.org:8080/13049 )
Change subject: IMPALA-8149 : Add support for alter_database events
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@978
PS1, Line 978: public void process() throws DatabaseNotFoundException,
CatalogException {
Looks like DatabaseNotFoundException is already captured in CatalogException,
so probaly we can remove DatabaseNotFoundException from throws.
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3812
PS1, Line 3812: long newCatalogVersion =
catalog_.incrementAndGetCatalogVersion();
Just asking to understand this part:
Do we need to lock the DB when we update the version?
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3813
PS1, Line 3813: addCatalogServiceIdentifiers(db,
catalog_.getCatalogServiceId(), newCatalogVersion);
I see that the only alter operation here is SET_OWNER, so shouldn't we update
the catalogServiceIdentifiers only when the operation is ALTER DB SET OWNER?
Otherwise, why should we just update the version?
http://gerrit.cloudera.org:8080/#/c/13049/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3844
PS1, Line 3844: synchronized (metastoreDdlLock_) {
Should we do addCatalogServiceIdentifiers after acquiring the metastoreDdlLock_
?
--
To view, visit http://gerrit.cloudera.org:8080/13049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b
Gerrit-Change-Number: 13049
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 18 Apr 2019 03:29:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 This change updates Impyla to 0.15.0 and then uses Impyla to retrieve thrift profiles through the HS2 api. Unfortunately, some of the current usages of get_thrift_profile rely on the Beeswax query states and the ImpylaHS2Connection does not have the required functionality yet. We will have to update these in a future change, once we unified the query states. This change also adds a self-contained test for IMPALA-2063 Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Reviewed-on: http://gerrit.cloudera.org:8080/12530 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M infra/python/deps/compiled-requirements.txt M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/query_test/test_cancellation.py M tests/query_test/test_observability.py 6 files changed, 91 insertions(+), 87 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 14 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 13 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Apr 2019 03:11:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 4: (1 comment) Thanks Vihang for the comment. Bharath, I have updated the patch accordingly. http://gerrit.cloudera.org:8080/#/c/13058/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@700 PS3, Line 700: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_TABLE operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_TABLE event processing. This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp > I think it would be good to add the following comment in the DropTableEvent Done -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 03:02:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase The test fails because of two Databases getting created with same CREATION_TIME. Hence, adding a sleep of 2 seconds to avoid this case. Also fixing other tests with similar use-case. Testing - Fixed the unit tests Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 40 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13058/4 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 20: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 20 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 01:59:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 19: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 01:59:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2832/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 01:38:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. IMPALA-8426: Logging error in DROP_TABLE event processing Fixing the bug in condition check while logging in DROP_TABLE event processing. Also updating EVENTS_SKIPPED metric to keep track of the number of drop table events skipped when CREATION_TIME matches. Testing: - Added metric check to unit test. - Ran existing unit tests. Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Reviewed-on: http://gerrit.cloudera.org:8080/13056 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 14 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 01:20:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12973 ) Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. Patch Set 8: Code-Review+2 Rebased and carrying over Lars's +2. Will test it on all supported OSes before merging it in. -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Apr 2019 01:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 19: (2 comments) Thanks Bharath. Changed the tests to account for slower builds. http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py File tests/custom_cluster/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@122 PS18, Line 122: time.sleep(bu > Lets use a longer timeout for slower builds like ASAN. How about 2 for regu Done http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@133 PS18, Line 133: > Breaks if someone updates the page. Parse it into keyvalue pairs and pick u Done -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 18 Apr 2019 00:57:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 575 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/19 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 19 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type
..
Patch Set 23: Code-Review+1
(1 comment)
Looked at the delta, good catch by csaba
http://gerrit.cloudera.org:8080/#/c/12481/19/be/src/exprs/cast-functions-ir.cc
File be/src/exprs/cast-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/12481/19/be/src/exprs/cast-functions-ir.cc@312
PS19, Line 312: if (UNLIKELY(!dv.IsValid())) {
: ctx->SetError("String to Date parse failed.");
: return DateVal::null();
: }
> Casts from STRING to most types don't return an error and don't emit a warn
Yeah this was a good point that I missed. We've tried to be stricter with new
functionality, e.g. DECIMAL_V2 errors in more cases than before.
I think in principle we should avoid different behaviour between constants and
variables, i.e. f(1) and f(x) where x=1 should have exactly the same behaviour.
This is an invariant that most programmers would use to reason about their
programs.
So the current patch's behaviour is good.
--
To view, visit http://gerrit.cloudera.org:8080/12481
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f
Gerrit-Change-Number: 12481
Gerrit-PatchSet: 23
Gerrit-Owner: Attila Jeges
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 18 Apr 2019 00:31:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7957: Fix slot equivalences may be enforced multiple times
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13051 ) Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times .. Patch Set 2: (1 comment) I'm struggling to wrap my head around this but in the meantime have a couple of ideas for ests. http://gerrit.cloudera.org:8080/#/c/13051/2/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test File testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test: http://gerrit.cloudera.org:8080/#/c/13051/2/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test@1760 PS2, Line 1760: Can you add some tests with ORDER BY LIMIT and GROUP BY in the first branch of the union? Just to confirm that there isn't a variant of the bug since there's some different logic with those code paths. -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Apr 2019 00:15:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12973 ) Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2831/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 18 Apr 2019 00:14:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13061 ) Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. Patch Set 1: (5 comments) We've already discussed this a bunch, but now looking at the solution in real code, I think this is good in that it solves the problem without getting into any tricky distributed systems problems. http://gerrit.cloudera.org:8080/#/c/13061/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13061/1//COMMIT_MSG@17 PS1, Line 17: Testing: Added an e2e test Just to confirm, this test failed before your change? http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h File be/src/statestore/statestore-subscriber.h: http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h@130 PS1, Line 130: int64 int64_t is the standard one, I guess int64 comes from some header we've pulled in. Can this be a const method, also? http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@100 PS1, Line 100: def test_restart_statestore_query_resilience(self): If you didn't already, can you loop this test for a while, maybe with 'stress' running in the background. Just want additional assurance that it's not flaky. http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@117 PS1, Line 117: / > flake8: E226 missing whitespace around arithmetic operator Might be slightly cleaner to also default CANCELLATION_GRACE_PERIOD_S above http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@119 PS1, Line 119: # Now restart statestore and kill an impalaD while it is down, and make sure the We might need to reconcile this with IMPALA-2990, which adds another mechanism to kill queries that doesn't depend on the statestore. I don't think there's action required now, just thinking out loud. -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 23:57:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8270: fix MemTracker teardown in FeSupport
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13057 ) Change subject: IMPALA-8270: fix MemTracker teardown in FeSupport .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2830/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If815130cd4db00917746f10b28514f779ee254f0 Gerrit-Change-Number: 13057 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 23:50:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13061 ) Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2829/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 23:46:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Hello Lars Volker, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12973 to look at the new patch set (#7). Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. IMPALA-8401: SIGRTMIN initiates the graceful shutdown process This patch enables a user that has access to the impalad process, to initiate the graceful shutdown process with a deadline of one year by sending SIGRTMIN signal to it. Sample usage: "kill -SIGRTMIN " Testing: Added relevant e2e tests. TODO: test the shutdown signal handeling on all supported OSes Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 --- M be/src/common/init.cc M be/src/common/init.h M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M tests/custom_cluster/test_restart_services.py 5 files changed, 124 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/12973/7 -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12973 ) Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@236 PS5, Line 236: exit(1); > Oh you're right. In that case you could also use cerr << above, no? Done -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 6 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 23:40:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8414: Skip header when parsing /proc/net/dev
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13016 ) Change subject: IMPALA-8414: Skip header when parsing /proc/net/dev .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13016/1/be/src/util/system-state-info.cc File be/src/util/system-state-info.cc: http://gerrit.cloudera.org:8080/#/c/13016/1/be/src/util/system-state-info.cc@174 PS1, Line 174: const StringPiece const StringPiece& -- To view, visit http://gerrit.cloudera.org:8080/13016 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ff931671050d44926d0baa77ec374afed1f8225 Gerrit-Change-Number: 13016 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 17 Apr 2019 23:09:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8270: fix MemTracker teardown in FeSupport
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13057 Change subject: IMPALA-8270: fix MemTracker teardown in FeSupport .. IMPALA-8270: fix MemTracker teardown in FeSupport This patch tries to simplify and standardise the order in which control structures are torn down. As a consequence the bug is fixed. I've described the bug below. The changes are: * Make more control structures owned directly by QueryState::obj_pool_, so that they are all destroyed at the same time via ~QueryState. * Tear down local_query_state_ explicitly before other destructors run. Either change is sufficient to fix the bug, but I preferred to do both to reduce the chances of similar bugs in future. Description of bug: === In the normal query execution flow: - RuntimeState is in QueryState::obj_pool_ - RuntimeState owns RuntimeState::instance_mem_tracker_ via unique_ptr - QueryState::query_mem_tracker_ is in QueryState::obj_pool_ - QueryState::query_mem_tracker_ has a reference to RuntimeState::instance_mem_tracker_ The tear-down works because ~QueryState unregisters query_mem_tracker_ from its parent, making the whole subtree unreachable before destroying QueryState::obj_pool_. It is thus safe to destroy instance_mem_tracker_ along with the rest of obj_pool_. FeSupport messes this up by having RuntimeState own the QueryState RuntimeState::local_query_state_ via a unique_ptr, and the implied destructor order means that RuntimeState::instance_mem_tracker_ is destroyed before RuntimeState::local_query_state_, which breaks the above flow and the destroyed local_query_state_ is reachable from the process MemTracker via QueryState::query_mem_tracker_ for a small window until it is unregistered. Testing: Added a backend test that reproduced the ASAN use-after-free failure when run against unmodified RuntimeState code. I did not make it a unified backend test so that it would be easier to backport this fix to older versions that don't have unified tests. Change-Id: If815130cd4db00917746f10b28514f779ee254f0 --- M be/src/runtime/CMakeLists.txt A be/src/runtime/runtime-state-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h 4 files changed, 99 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/13057/1 -- To view, visit http://gerrit.cloudera.org:8080/13057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If815130cd4db00917746f10b28514f779ee254f0 Gerrit-Change-Number: 13057 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13061 ) Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@26 PS1, Line 26: from time import sleep, time flake8: F811 redefinition of unused 'time' from line 23 http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@98 PS1, Line 98: . flake8: E131 continuation line unaligned for hanging indent http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@117 PS1, Line 117: / flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@130 PS1, Line 130: / flake8: E226 missing whitespace around arithmetic operator -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 23:04:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13061 Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore restart .. IMPALA-7665: Fix unwarranted query cancellation on statestore restart Currently, if the statestore restarts and disseminates an inconsistent view of cluster membership to the coordinators, then they might believe that the backends no longer in the membership update are down and would start canceling queries that are running or scheduled to run on those allegedly failed backends. This patch adds a grace period after statestore recovery/successful registration that give it enough time to gather a consistent state of the cluster. Testing: Added an e2e test Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 --- M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M tests/custom_cluster/test_restart_services.py 4 files changed, 66 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/13061/1 -- To view, visit http://gerrit.cloudera.org:8080/13061 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959 Gerrit-Change-Number: 13061 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 12 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 22:36:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 22:10:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. IMPALA-8392: fix parallel docker_images build I made the other targets depend on targets, not the timestamp file, according to the suggested solution in: https://gitlab.kitware.com/cmake/cmake/issues/17585 Testing: Ran "make -j 8 docker_images" locally, which now succeeds. Running dockerised tests. Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Reviewed-on: http://gerrit.cloudera.org:8080/13053 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/jenkins/dockerized-impala-run-tests.sh M docker/CMakeLists.txt 2 files changed, 11 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 22:16:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4039/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 13 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 22:07:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. IMPALA-8410: enable TestTpcdsInsert by default Fix incorrect output to match current behaviour. The test takes 24s on my system but can be run in parallel with other tests. Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Reviewed-on: http://gerrit.cloudera.org:8080/13055 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/tpcds-insert/queries/partitioned-insert.test M tests/query_test/test_tpcds_queries.py 2 files changed, 512 insertions(+), 33 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 13: Code-Review+2 PS 12 fixes Bikram's last comment, PS13 rebases the change. Carrying Bikram's +2. -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 13 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 22:07:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Hello Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12530 to look at the new patch set (#12). Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 This change updates Impyla to 0.15.0 and then uses Impyla to retrieve thrift profiles through the HS2 api. Unfortunately, some of the current usages of get_thrift_profile rely on the Beeswax query states and the ImpylaHS2Connection does not have the required functionality yet. We will have to update these in a future change, once we unified the query states. This change also adds a self-contained test for IMPALA-2063 Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e --- M infra/python/deps/compiled-requirements.txt M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/query_test/test_cancellation.py M tests/query_test/test_observability.py 6 files changed, 91 insertions(+), 87 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/12530/12 -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 12 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12973 ) Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. Patch Set 6: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc File be/src/common/init.cc: http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@236 PS5, Line 236: exit(1); > do we need that here? since this is on the main thread and not in the signa Oh you're right. In that case you could also use cerr << above, no? -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 6 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 22:05:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/12973 )
Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
..
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.h
File be/src/common/init.h:
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.h@25
PS5, Line 25: SIGNAL
> nit: lowercase
Done
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc
File be/src/common/init.cc:
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@232
PS5, Line 232: void HandleSysCallError(const int syscall_ret_val, const string&
msg) {
> Maybe call this method "AbortIfError" in reference to the macros we already
Done
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@236
PS5, Line 236: exit(1);
> Use _exit here because exit() is also not async-signal-safe. See comment ab
do we need that here? since this is on the main thread and not in the signal
handler thread like in L226.
http://gerrit.cloudera.org:8080/#/c/12973/5/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/12973/5/tests/custom_cluster/test_restart_services.py@97
PS5, Line 97: match = re.match(r'shutdown grace period left: ([0-9ms]*),
deadline left: ([0-9ms]*),'
> nit: have space trailing the previous line, or lead the current line, but p
Done
--
To view, visit http://gerrit.cloudera.org:8080/12973
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86
Gerrit-Change-Number: 12973
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 17 Apr 2019 21:49:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13003 ) Change subject: IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2 .. Patch Set 2: gvo failed due to IMPALA-8430 -- To view, visit http://gerrit.cloudera.org:8080/13003 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3feddaccb9be3a15220ce9e59aa7ed41d41b8ab6 Gerrit-Change-Number: 13003 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 21:56:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2826/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 17 Apr 2019 21:55:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13003 ) Change subject: IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4033/ -- To view, visit http://gerrit.cloudera.org:8080/13003 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3feddaccb9be3a15220ce9e59aa7ed41d41b8ab6 Gerrit-Change-Number: 13003 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 21:53:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Hello Lars Volker, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12973 to look at the new patch set (#6). Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process .. IMPALA-8401: SIGRTMIN initiates the graceful shutdown process This patch enables a user that has access to the impalad process, to initiate the graceful shutdown process with a deadline of one year by sending SIGRTMIN signal to it. Sample usage: "kill -SIGRTMIN " Testing: Added relevant e2e tests. TODO: test the shutdown signal handeling on all supported OSes Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 --- M be/src/common/init.cc M be/src/common/init.h M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M tests/custom_cluster/test_restart_services.py 5 files changed, 125 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/12973/6 -- To view, visit http://gerrit.cloudera.org:8080/12973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86 Gerrit-Change-Number: 12973 Gerrit-PatchSet: 6 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 11: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/12530/11/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/12530/11/tests/common/impala_connection.py@314 PS11, Line 314: "FINISHED_STATE" > TCLIService.TOperationState is an enum and we need to use _VALUES_TO_NAMES I agree it wont be good for readability. Maybe just mention in a comment that it comes from TOperationState so we have an idea of where to look for other states http://gerrit.cloudera.org:8080/#/c/12530/11/tests/query_test/test_cancellation.py File tests/query_test/test_cancellation.py: http://gerrit.cloudera.org:8080/#/c/12530/11/tests/query_test/test_cancellation.py@221 PS11, Line 221: assert any(client.get_state(handle) == 'RUNNING_STATE' or sleep(1) :for _ in range(5)), 'Query failed to start' > ImpalaTestSuite.wait_for_state uses self.client, which is a beeswax client. wfm. didnt notice it was using beeswax. -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 11 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 21:41:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13060/1/tests/custom_cluster/test_rpc_timeout.py File tests/custom_cluster/test_rpc_timeout.py: http://gerrit.cloudera.org:8080/#/c/13060/1/tests/custom_cluster/test_rpc_timeout.py@129 PS1, Line 129: \ flake8: E502 the backslash is redundant between brackets -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 17 Apr 2019 21:14:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13060 Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY This patch removes the FAULT_INJECTION_RPC_DELAY macro and replaces its uses with DebugAction which is more flexible. For example, it supports JITTER which injects random delays. Every backend rpc has a debug action of the form RPC_NAME_DELAY. DebugAction has previously always been used via query options. However, for the rpcs considered here there is not always a query with an accessible TQUeryOptions available (for example, we do not send any query info with the RemoteShutdown rpc), so this patch introduces a flag, '--debug_actions', which is used to control these rpc delay debug actions. Testing: - Updated existing tests to use the new mechanism. Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b --- M be/src/common/global-flags.cc M be/src/service/control-service.cc M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/testutil/fault-injection-util.cc M be/src/testutil/fault-injection-util.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M tests/custom_cluster/test_rpc_timeout.py 9 files changed, 52 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13060/1 -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); > Like we discussed offlines, this is probably not needed. Consider removing Yes, I guess we don't need the sleep here because the first create will be filtered out, and the drop event will just be a no-op as the Table/DB is not yet created. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 21:14:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: Code-Review+1 (2 comments) Vihang, can you take a quick look too? http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); > Yes, so the same applies to Hive as well. We don't want events to have same Like we discussed offlines, this is probably not needed. Consider removing it. http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@709 PS1, Line 709: This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp. > It is expected to rely on the ctime, but limitation is that we don't expect Not totally understand but looks like you are talking about the precision of the underlying timestamp and it being rounded off to a nearest second. If so, clarify that? -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 21:09:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 18: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py File tests/custom_cluster/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@109 PS18, Line 109: : nit: formatting http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@122 PS18, Line 122: time.sleep(2) Lets use a longer timeout for slower builds like ASAN. How about 2 for regular and 4s for slower? (Look for the usages of build_flavor_timeout) http://gerrit.cloudera.org:8080/#/c/12889/18/tests/custom_cluster/test_event_processing.py@133 PS18, Line 133: 31 Breaks if someone updates the page. Parse it into keyvalue pairs and pick up the value mapping to the key? -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 18 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 20:50:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 20:39:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 20:18:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4038/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 20:18:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 20:18:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Krishna has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. IMPALA-8426: Logging error in DROP_TABLE event processing Fixing the bug in condition check while logging in DROP_TABLE event processing. Also updating EVENTS_SKIPPED metric to keep track of the number of drop table events skipped when CREATION_TIME matches. Testing: - Added metric check to unit test. - Ran existing unit tests. Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 14 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/13056/2 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12939 ) Change subject: IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/12939/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12939/4//COMMIT_MSG@15 PS4, Line 15: For example, consider the following tables and a query: Thanks for the nice explanation! http://gerrit.cloudera.org:8080/#/c/12939/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: http://gerrit.cloudera.org:8080/#/c/12939/4/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1190 PS4, Line 1190: Expr expr = e.trySubstitute(inlineViewRef.getBaseTblSmap(), analyzer, false); Intuitively this makes sense to me, since it's doing a "deeper" substitution that will "collapse" more expressions from the view into the same expressions on the base table. I'm trying to convince myself that there are not edge cases where the previous check would have identified an equivalence predicate I.e. where smap[e1] == smap[e2] but baseTblSmap[e1] != baseTableSmap[e2] I think generally the first implies the second, but it's possible there are some other latent bugs. I have a couple of requests for different tests that might catch any bugs with smap/baseTableSmap and the corresponding resultExprs_/baseTableResultExprs_ state. http://gerrit.cloudera.org:8080/#/c/12939/4/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test File testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test: http://gerrit.cloudera.org:8080/#/c/12939/4/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test@1705 PS4, Line 1705: Can we add a test where each subquery has an ORDER BY/LIMIT. This case is semi-interesting since there's an additional layer of substitution in createSortTupleInfo(). Also tests with a UNION inside the subquery since there is logic in UnionNode manipulating the different ResultExprs. -- To view, visit http://gerrit.cloudera.org:8080/12939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa Gerrit-Change-Number: 12939 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 19:43:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 3: Sorry, I didn't refresh before posting my own comments, so didn't notice what Sudhanshu wrote. -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 19:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: (2 comments) Hi Bharath, thanks for the review. I have added comments to explain the limitation about CREATION_TIME, let me know if it makes sense. http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); > Do you need this if all the creates are via Hive? (event processor is expec Yes, so the same applies to Hive as well. We don't want events to have same CREATION_TIME as mentioned below. http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@709 PS1, Line 709: This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp. > I'm not totally sure I understand how this is a limitation. Isn't the proce It is expected to rely on the ctime, but limitation is that we don't expect a CREATE_TABLE and DROP_TABLE on the same table happen in the same second. In that case, the logic would fail to filter out the event. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 19:51:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 3: (6 comments) http://gerrit.cloudera.org:8080/#/c/13005/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13005/3//COMMIT_MSG@31 PS3, Line 31: t to nit: wrap at 72 chars in the commit message http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java File fe/src/main/java/org/apache/impala/analysis/StringLiteral.java: http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/analysis/StringLiteral.java@96 PS3, Line 96: /** :* Copied from Apache Hive's BaseSemanticAnalyzer. This method has not changed :* since last several years so hoping that it is fairly stable by now. Sourcing it from :* the Hive's code without copying brings along with it a lot of other unnecessary :* dependencies :* @param b :* @return :*/ Can you move the parts copied from Hive to a separate file/directory? It will be easier to remove them later if they are at a single place and not spread around at call sites. http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java: http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@122 PS3, Line 122: //TODO (Vihang) this pulls in hive-hbase-handler dependency which brings all the : // other stuff we don't need. Perhaps we just need to copy the constants and mark : // them public API in Hive source code : private static final String HBASE_COLUMNS_MAPPING = "hbase.columns.mapping"; : private static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE = "hbase.table.default" : + ".storage.type"; : private static final String HBASE_KEY_COL = ":key"; : private static final String HBASE_TABLE_NAME = "hbase.table.name"; Constants copied from Hive 3 could be copied to a common place. http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@189 PS3, Line 189: ` nit: mistyping? http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java File fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java: http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java@29 PS3, Line 29: nit: wrong indentation http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java File fe/src/main/java/org/apache/impala/service/MetadataOp.java: http://gerrit.cloudera.org:8080/#/c/13005/3/fe/src/main/java/org/apache/impala/service/MetadataOp.java@345 PS3, Line 345: case VIRTUAL_VIEW: : case MATERIALIZED_VIEW: : return TABLE_TYPE_VIEW; This returns TABLE_TYPE_VIEW even for EXTERNAL_TABLE and MANAGED_TABLE - is this intentional to avoid changing any table? -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 19:50:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13006 ) Change subject: IMPALA-7892: [DOCS] Described the new network I/O throughput in query profiles .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13006/1/docs/topics/impala_explain_plan.xml File docs/topics/impala_explain_plan.xml: http://gerrit.cloudera.org:8080/#/c/13006/1/docs/topics/impala_explain_plan.xml@238 PS1, Line 238: and other processes : running on the same system. This does not include processes on the same system (we exclude the loopback interface when aggregating the network counters). It only includes traffic to/from other hosts. -- To view, visit http://gerrit.cloudera.org:8080/13006 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I25b128bc23f418347b400ca9e694d9d591935592 Gerrit-Change-Number: 13006 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 17 Apr 2019 19:18:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4037/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 19:15:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 19:15:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4865: Reject Expr Rewrite When Appropriate
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12814 ) Change subject: IMPALA-4865: Reject Expr Rewrite When Appropriate .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12814 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b078113ccc1aa49b0cea0c86dff2e02e1dd0e23 Gerrit-Change-Number: 12814 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 19:15:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13056/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13056/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@724 PS1, Line 724: assertEquals(1, : eventsProcessor_.getMetrics() : .getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).getCount() : - numFilteredEvents); > nit: assertEquals(numFilteredevents +1 , ..getCount() ? +1 to above suggestion. -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 19:03:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 19:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13056/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13056/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@724 PS1, Line 724: assertEquals(1, : eventsProcessor_.getMetrics() : .getCounter(MetastoreEventsProcessor.EVENTS_SKIPPED_METRIC).getCount() : - numFilteredEvents); nit: assertEquals(numFilteredevents +1 , ..getCount() ? I think that is more readable. -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 19:00:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@278 PS1, Line 278: // Adding sleep here to make sure that the CREATION_TIME is not same : // as the previous CREATE_DB operation, so as to trigger the filtering logic : // based on CREATION_TIME in DROP_DB event processing. This is currently a : // limitation : the DROP_DB event filtering expects that while processing events, : // the CREATION_TIME of two Databases with same name won't have the same : // creation timestamp. : sleep(2000); Do you need this if all the creates are via Hive? (event processor is expected to replay the events in the same order) http://gerrit.cloudera.org:8080/#/c/13058/1/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@709 PS1, Line 709: This is currently a : // limitation : the DROP_TABLE event filtering expects that while processing events, : // the CREATION_TIME of two tables with same name won't have the same : // creation timestamp. I'm not totally sure I understand how this is a limitation. Isn't the processor expected to rely on ctime? (same question in other places too) -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 18:57:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Thomas Marshall has abandoned this change. ( http://gerrit.cloudera.org:8080/12297 ) Change subject: IMPALA-8138: Reintroduce rpc debugging options .. Abandoned we've decided to go in a different direction -- To view, visit http://gerrit.cloudera.org:8080/12297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 Gerrit-Change-Number: 12297 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.cc File be/src/runtime/io/data-cache.cc: http://gerrit.cloudera.org:8080/#/c/12987/2/be/src/runtime/io/data-cache.cc@343 PS2, Line 343: insertion_offset)); insertion_offset + bytes_written -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 18:35:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12973 )
Change subject: IMPALA-8401: SIGRTMIN initiates the graceful shutdown process
..
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.h
File be/src/common/init.h:
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.h@25
PS5, Line 25: SIGNAL
nit: lowercase
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc
File be/src/common/init.cc:
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@232
PS5, Line 232: void HandleSysCallError(const int syscall_ret_val, const string&
msg) {
Maybe call this method "AbortIfError" in reference to the macros we already
have? That also makes it clear at the call-site what the behavior will be.
"Handle..." still leaves it open what happens.
http://gerrit.cloudera.org:8080/#/c/12973/5/be/src/common/init.cc@236
PS5, Line 236: exit(1);
Use _exit here because exit() is also not async-signal-safe. See comment above
in L226.
http://gerrit.cloudera.org:8080/#/c/12973/5/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/12973/5/tests/custom_cluster/test_restart_services.py@97
PS5, Line 97: match = re.match(r'shutdown grace period left: ([0-9ms]*),
deadline left: ([0-9ms]*),'
nit: have space trailing the previous line, or lead the current line, but
prefer not to mix
--
To view, visit http://gerrit.cloudera.org:8080/12973
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I521ffd7526ac9a8a5c4996994eb68d6a855aef86
Gerrit-Change-Number: 12973
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 17 Apr 2019 18:28:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13058 ) Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 18:08:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 23: Code-Review+1 Carry +1 -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 23 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 18:09:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12530 ) Change subject: IMPALA-8158: Retrieve thrift profiles through Impyla 0.15.0 .. Patch Set 11: (2 comments) http://gerrit.cloudera.org:8080/#/c/12530/11/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/12530/11/tests/common/impala_connection.py@314 PS11, Line 314: "FINISHED_STATE" > do we have a constant for this? like in TCLIService.TOperationState TCLIService.TOperationState is an enum and we need to use _VALUES_TO_NAMES to map it to a string. This makes this whole check: return cursor.status() == TOperationState._VALUES_TO_NAMES[TOperationState.FINISHED_STATE] Since this is thrift generated, the enum value and its string will always be the same and using the constant+map will not give us any more flexibility in the future, but is much less readable. Let me know if you feel strongly about it or if I'm missing anything, but otherwise I think I prefer the string. http://gerrit.cloudera.org:8080/#/c/12530/11/tests/query_test/test_cancellation.py File tests/query_test/test_cancellation.py: http://gerrit.cloudera.org:8080/#/c/12530/11/tests/query_test/test_cancellation.py@221 PS11, Line 221: assert any(client.get_state(handle) == 'RUNNING_STATE' or sleep(1) :for _ in range(5)), 'Query failed to start' > we can use wait_for_state() here and at Line 228 ImpalaTestSuite.wait_for_state uses self.client, which is a beeswax client. Here we want to use the hs2_client though, and I thought that this was easier than adding another set of methods to the test suite. Let me know if you prefer adding a new method wait_for_state_using_client() there. Once we switch our tests to HS2, we can update the usage here. -- To view, visit http://gerrit.cloudera.org:8080/12530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I769a99f0843297dd2b20f2f5b1a9046c97bb131e Gerrit-Change-Number: 12530 Gerrit-PatchSet: 11 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 18:08:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13058 Change subject: IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase .. IMPALA-8430: Fix flakiness in testCreateDropCreateDatabase The test fails because of two Databases getting created with same CREATION_TIME. Hence, adding a sleep of 2 seconds to avoid this case. Also fixing other tests with similar use-case. Testing - Fixed the unit tests Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d --- M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 1 file changed, 27 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13058/1 -- To view, visit http://gerrit.cloudera.org:8080/13058 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I30bf4535d54c9cd8d257b528dc7a1b42f106800d Gerrit-Change-Number: 13058 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 17:13:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 17:20:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4035/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 17:20:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13056 ) Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2823/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 17 Apr 2019 17:15:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4034/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 17:01:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 17:01:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8410: enable TestTpcdsInsert by default
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13055 ) Change subject: IMPALA-8410: enable TestTpcdsInsert by default .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13055/1/tests/query_test/test_tpcds_queries.py File tests/query_test/test_tpcds_queries.py: http://gerrit.cloudera.org:8080/#/c/13055/1/tests/query_test/test_tpcds_queries.py@a519 PS1, Line 519: I think this is fine as a quick fix for this test, but it would be nice to actually be able to run non-"functional-query" tests only in exhaustive, and I suspect a bunch more tests are affected by issues like this. Fwiw, this is already tracked by IMPALA-3947 -- To view, visit http://gerrit.cloudera.org:8080/13055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf9a279d57ad74de0c77a90dde69e5c4dc563a3f Gerrit-Change-Number: 13055 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 17 Apr 2019 16:58:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8392: fix parallel docker images build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13053 ) Change subject: IMPALA-8392: fix parallel docker_images build .. Patch Set 2: The dockerised build succeeded with this change - https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/55/ -- To view, visit http://gerrit.cloudera.org:8080/13053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb658ee156eb9b186ff3fcc3e4a40ad87ed7c0ce Gerrit-Change-Number: 13053 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 16:56:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13003 ) Change subject: IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13003 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3feddaccb9be3a15220ce9e59aa7ed41d41b8ab6 Gerrit-Change-Number: 13003 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 16:51:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13003 ) Change subject: IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4033/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13003 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3feddaccb9be3a15220ce9e59aa7ed41d41b8ab6 Gerrit-Change-Number: 13003 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 16:51:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8426: Logging error in DROP TABLE event processing
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13056 Change subject: IMPALA-8426: Logging error in DROP_TABLE event processing .. IMPALA-8426: Logging error in DROP_TABLE event processing Fixing the bug in condition check while logging in DROP_TABLE event processing. Also updating EVENTS_SKIPPED metric to keep track of the number of drop table events skipped when CREATION_TIME matches. Testing: - Added metric check to unit test. - Ran existing unit tests. Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 14 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/13056/1 -- To view, visit http://gerrit.cloudera.org:8080/13056 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0a2ca10f82d183fd2821014e30b109b9f4474db4 Gerrit-Change-Number: 13056 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2822/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Apr 2019 16:27:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13003 ) Change subject: IMPALA-8407: Warn when Impala shell fails to connect due to tlsv1.2 .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13003/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13003/1/shell/impala_shell.py@844 PS1, Line 844: and "EOF occurred in violation of protocol" in str(e): Using pure string matching feels hacky but I can't think of anything concrete that will go wrong, so I guess this is fine :). -- To view, visit http://gerrit.cloudera.org:8080/13003 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3feddaccb9be3a15220ce9e59aa7ed41d41b8ab6 Gerrit-Change-Number: 13003 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 17 Apr 2019 15:40:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Hello Michael Ho, Lars Volker, Pooja Nilangekar, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12065 to look at the new patch set (#14). Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. IMPALA-5843: Use page index in Parquet files to skip pages This commit implements page filtering based on the Parquet page index. The read and evaluation of the page index is done by the HdfsParquetScanner. At first, we determine the row ranges we are interested in, and based on the row ranges we determine the candidate pages for each column that we are reading. We still issue one ScanRange per column chunk, but we specify sub-ranges that store the candidate pages, i.e. we don't read the whole column chunk, but only fractions of it. Pages are not aligned across column chunks, i.e. page #2 of column A might store completely different rows than page #2 of column B. It means we need to implement some kind of row-skipping logic when we read the data pages. This logic is implemented in BaseScalarColumnReader and ScalarColumnReader. Collection column readers know nothing about page filtering. Page filtering can be turned off by setting the query option 'read_parquet_page_index' to false. Testing: * added some unit tests for the row range and page selection logic * generated various Parquet files with Parquet-MR * enabled Page index writing and wrote selective queries against tables written by Impala. Current tests are likely to use page filtering transparently. Performance: * Measured locally, observed 3x to 20x speedup for selective queries. The speedup was proportional to the IO operations need to be done. * The TPCH benchmark didn't show a significant performance change. It is not a suprise since the data is not being sorted in any useful way. So the main goal was to not introduce perf regression. TODO: * measure performance for remote reads Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a --- M be/src/common/global-flags.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h A be/src/exec/parquet/parquet-common-test.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-level-decoder.h A be/src/exec/parquet/parquet-page-index-test.cc A be/src/exec/parquet/parquet-page-index.cc A be/src/exec/parquet/parquet-page-index.h M be/src/exprs/literal.cc M be/src/runtime/scoped-buffer.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/data/README A testdata/data/alltypes_tiny_pages.parquet A testdata/data/alltypes_tiny_pages_plain.parquet A testdata/data/decimals_1_10.parquet A testdata/data/double_nested_decimals.parquet A testdata/data/nested_decimals.parquet A testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages-plain.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-large.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/query_test/test_parquet_stats.py 36 files changed, 3,391 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12065/14 -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2821/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 23 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 14:23:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has uploaded a new patch set (#23). (
http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type
..
IMPALA-7368: Add initial support for DATE type
DATE values describe a particular year/month/day in the form
-MM-dd. For example: DATE '2019-02-15'. DATE values do not have a
time of day component. The range of values supported for the DATE type
is -01-01 to -12-31.
This initial DATE type support covers TEXT and HBASE fileformats only.
'DateValue' is used as the internal type to represent DATE values.
The changes are as follows:
- Support for DATE literal syntax.
- Explicit casting between DATE and other types (note that invalid
casts will fail with an error just like invalid DECIMAL_V2 casts,
while failed casts to other types do no lead to warning or error):
- from STRING to DATE. The string value must be formatted as
-MM-dd HH:mm:ss.S. The date component is mandatory,
the time component is optional. If the time component is
present, it will be truncated silently.
- from DATE to STRING. The resulting string value is formatted as
-MM-dd.
- from TIMESTAMP to DATE. The source timestamp's time of day
component is ignored.
- from DATE to TIMESTAMP. The target timestamp's time of day
component is set to 00:00:00.
- Implicit casting between DATE and other types:
- from STRING to DATE if the source string value is used in a
context where a DATE value is expected.
- from DATE to TIMESTAMP if the source date value is used in a
context where a TIMESTAMP value is expected.
- Since STRING -> DATE, STRING -> TIMESTAMP and DATE -> TIMESTAMP
implicit conversions are now all possible, the existing function
overload resolution logic is not adequate anymore.
For example, it resolves the
if(false, '2011-01-01', DATE '1499-02-02') function call to the
if(BOOLEAN, TIMESTAMP, TIMESTAMP) version of the overloaded
function, instead of the if(BOOLEAN, DATE, DATE) version.
This is clearly wrong, so the function overload resolution logic had
to be changed to resolve function calls to the best-fit overloaded
function definition if there are multiple applicable candidates.
An overloaded function definition is an applicable candidate for a
function call if each actual parameter in the function call either
matches the corresponding formal parameter's type (without casting)
or is implicitly castable to that type.
When looking for the best-fit applicable candidate, a parameter
match score (i.e. the number of actual parameters in the function
call that match their corresponding formal parameter's type without
casting) is calculated and the applicable candidate with the highest
parameter match score is chosen.
There's one more issue that the new resolution logic has to address:
if two applicable candidates have the same parameter match score and
the only difference between the two is that the first one requires a
STRING -> TIMESTAMP implicit cast for some of its parameters while
the second one requires a STRING -> DATE implicit cast for the same
parameters then the first candidate has to be chosen not to break
backward compatibility.
E.g: year('2019-02-15') function call must resolve to
year(TIMESTAMP) instead of year(DATE). Note, that year(DATE) is not
implemented yet, so this is not an issue at the moment but it will
be in the future.
When the resolution algorithm considers overloaded function
definitions, first it orders them lexicographically by the types in
their parameter lists. To ensure the backward compatible behavior
Primitivetype.DATE enum value has to come after
PrimitiveType.TIMESTAMP.
- Codegen infrastructure changes for expression evaluation.
- 'IS [NOT] NULL' and '[NOT] IN' predicates.
- Common comparison operators (including the 'BETWEEN' operator).
- Infrastructure changes for built-in functions.
- Some built-in functions: conditional, aggregate, analytical and
math functions.
- C++ UDF/UDA support.
- Support partitioning and grouping by DATE.
- Beeswax, HiveServer2 support.
These items are tightly coupled and it makes sense to implement them
in one change-set.
Testing:
- A new partitioned TEXT table 'functional.date_tbl' (and the
corresponding HBASE table 'functional_hbase.date_tbl') was
introduced for DATE-related tests.
- BE and FE tests were extended to cover DATE type.
- E2E tests:
- since DATE type is supported for TEXT and HBASE fileformats
only, most DATE tests were implemented separately in
tests/query_test/test_date_queries.py.
Note, that this change-set is not a complete DATE type implementation,
but it lays the foundation for future work:
- Add date support to the random query generator.
- Implement a complete set of built-in functions.
- Add Parquet support.
- Add Kudu support.
-
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 23: > Uploaded patch set 23. Change has been rebased -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 23 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 17 Apr 2019 13:42:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Apr 2019 13:29:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/12065/13/be/src/exec/parquet/parquet-page-index.cc File be/src/exec/parquet/parquet-page-index.cc: http://gerrit.cloudera.org:8080/#/c/12065/13/be/src/exec/parquet/parquet-page-index.cc@112 PS13, Line 112: VLOG_QUERY << "io_buffer->len(): " << io_buffer->len() << ", page_index_buffer_.Size(): " << page_index_buffer_.Size(); line too long (121 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 17 Apr 2019 12:48:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/12065 )
Change subject: IMPALA-5843: Use page index in Parquet files to skip pages
..
Patch Set 11:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc@644
PS8, Line 644: // {min: 10, max: 20}, and query is 'select * from T
where A = 8'.
> This is not the only case when it can happen. Another example is multi key
Thanks, yeah, so basically it can happen anytime when you have predicates
against different columns, and the separate row ranges that pass the predicates
don't have a common subset. I extended the comment.
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/hdfs-parquet-scanner.cc@618
PS11, Line 618: COUNTER_ADD(num_row_groups_counter_, 1);
> optional: This loop grew really large, > 100 lines, doesn't even fit to my
This change is already quite huge, I would rather not add unrelated
refactorings.
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h
File be/src/exec/parquet/parquet-page-index.h:
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@39
PS11, Line 39: /// It reads the raw bytes of the whole page index and stores
it in an
: /// internal buffer.
: /// It doesn't expect that the Page index in a particular
layout, it only
: /// expects that the whole page index layed out continuously
in the file.
: /// It needs to be called before the serialization methods.
: Status ReadAll();
> Can you mention that this reads the page index for all row groups? For me t
Changed this function to only read the indexes belonging to a single row group.
Updated the comment.
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@61
PS11, Line 61: /// Common helper for deserialization
> nit: other member comments have . at the end
Done
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.h@65
PS11, Line 65: /// The scanner that created this object
> nit: other member comments have . at the end
Done
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.cc
File be/src/exec/parquet/parquet-page-index.cc:
http://gerrit.cloudera.org:8080/#/c/12065/11/be/src/exec/parquet/parquet-page-index.cc@81
PS11, Line 81: page_index_buffer_.TryAllocate(size)
> Sorry for the wrong assumptions, I got lost somewhere in HdfsParquetScanner
Made this change, now we only read the indexes of a single row group, and after
evaluating it we release the allocated buffer.
--
To view, visit http://gerrit.cloudera.org:8080/12065
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a
Gerrit-Change-Number: 12065
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 17 Apr 2019 12:50:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Hello Michael Ho, Lars Volker, Pooja Nilangekar, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12065 to look at the new patch set (#13). Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. IMPALA-5843: Use page index in Parquet files to skip pages This commit implements page filtering based on the Parquet page index. The read and evaluation of the page index is done by the HdfsParquetScanner. At first, we determine the row ranges we are interested in, and based on the row ranges we determine the candidate pages for each column that we are reading. We still issue one ScanRange per column chunk, but we specify sub-ranges that store the candidate pages, i.e. we don't read the whole column chunk, but only fractions of it. Pages are not aligned across column chunks, i.e. page #2 of column A might store completely different rows than page #2 of column B. It means we need to implement some kind of row-skipping logic when we read the data pages. This logic is implemented in BaseScalarColumnReader and ScalarColumnReader. Collection column readers know nothing about page filtering. Page filtering can be turned off by setting the query option 'read_parquet_page_index' to false. Testing: * added some unit tests for the row range and page selection logic * generated various Parquet files with Parquet-MR * enabled Page index writing and wrote selective queries against tables written by Impala. Current tests are likely to use page filtering transparently. Performance: * Measured locally, observed 3x to 20x speedup for selective queries. The speedup was proportional to the IO operations need to be done. * The TPCH benchmark didn't show a significant performance change. It is not a suprise since the data is not being sorted in any useful way. So the main goal was to not introduce perf regression. TODO: * measure performance for remote reads Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a --- M be/src/common/global-flags.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h A be/src/exec/parquet/parquet-common-test.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-level-decoder.h A be/src/exec/parquet/parquet-page-index-test.cc A be/src/exec/parquet/parquet-page-index.cc A be/src/exec/parquet/parquet-page-index.h M be/src/exprs/literal.cc M be/src/runtime/scoped-buffer.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/data/README A testdata/data/alltypes_tiny_pages.parquet A testdata/data/alltypes_tiny_pages_plain.parquet A testdata/data/decimals_1_10.parquet A testdata/data/double_nested_decimals.parquet A testdata/data/nested_decimals.parquet A testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages-plain.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-large.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/query_test/test_parquet_stats.py 36 files changed, 3,393 insertions(+), 96 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12065/13 -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2819/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 8 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 17 Apr 2019 12:33:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2818/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 7 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 17 Apr 2019 12:29:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2817/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 6 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 17 Apr 2019 12:21:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2816/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 5 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 17 Apr 2019 12:13:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12901 to look at the new patch set (#8). Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger Wrote unit tests to capture above assertions Ran fe unit and e2e tests Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 306 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/8 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 8 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
