[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 22:45:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Reviewed-on: http://gerrit.cloudera.org:8080/13111 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 79 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3006/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 18:08:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4121/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/8 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115 PS7, Line 2115: @throws > typo: @throws Done http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@551 PS7, Line 551: TPart > TPartiton is the name of the struct, so it's more readable if we don't shor Done http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@568 PS7, Line 568: P > nit: this should be an upper case P Done -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 17:22:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115 PS7, Line 2115: @thorws typo: @throws http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@551 PS7, Line 551: TPart TPartiton is the name of the struct, so it's more readable if we don't shorten this. TPartition? http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@568 PS7, Line 568: p nit: this should be an upper case P -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 14:43:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 07:12:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/7 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: (8 comments) http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2113 PS6, Line 2113: Throws CatalogException if partition reload is unsuccessful. Throws :* DatabaseNotFoundException if Db doesn't exist We should use @throws CatalogException and @throws DatabaseNotFoundException javadoc instead. http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1252 PS6, Line 1252: /** nit: add a new line after L1251 http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1255 PS6, Line 1255: getTPartSpecFromHmsPart I think it's better to not shorten partition as part since it can be confusing. http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1272 PS6, Line 1272: constructPartStringFromTpart nit: constructPartitionStringFromTPartition http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1288 PS6, Line 1288: List addedPartitions_; can this be private final? http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1463 PS6, Line 1463: private final List> droppedPartitions_; nit: add a new empty line after this variable declaration http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1523 PS6, Line 1523: a typo: an http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1573 PS6, Line 1573: Collection partsAfterAdd = We don't have to do it in this CR, but the method here is getting too big. We should have a separate helper method for each event. It' makes the code easier to read. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 30 Apr 2019 00:54:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 23:28:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2978/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:23:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 6: Resolved merge conflicts with IMPALA-8454. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 22:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 201 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/6 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 5: Rebased after insert events patch was merged. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 21:37:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 200 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/5 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 19:23:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 4: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13111/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@56 PS4, Line 56: ClassUtil remove if unused -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 29 Apr 2019 18:52:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 205 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/4 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
..
Patch Set 3:
(5 comments)
Patch looks good to me. some nits below and its good to go from my side.
http://gerrit.cloudera.org:8080/#/c/13111/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/3//COMMIT_MSG@17
PS3, Line 17: affetced
spell-check
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1081
PS3, Line 1081: org.apache.hadoop.hive.common.FileUtils
Can we import this instead of using fully qualified classname?
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1153
PS3, Line 1153: infoLog
no need to have add_partitions in the log message since infoLog provides that
information
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1307
PS3, Line 1307: {} partitions dropped from table {} in alter_partition event
This statement doesn't make sense to me. Why would there be dropped partitions
in alter_partition event? Also, the log doesn't need to have a event type in it
since infoLog method takes care of it.
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1315
PS3, Line 1315: debugLog
no need to say "after a drop_partition event" here since the debugLog prints
both the event id and event type
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 29 Apr 2019 17:31:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2927/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 26 Apr 2019 02:31:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
..
Patch Set 3:
(15 comments)
Thanks for your comments Vihang and Bharath. Please review my comments.
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7973: Add support for fine grained events processing for
> nit: mention about "events" in the title? Otherwise it looks very general .
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13
PS2, Line 13:
> nit: ..affected..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14
PS2, Line 14: refresh affected partitions
: in case of add/d
> Not sure I understand this, can you please clarify?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19
PS2, Line 19: on to
> nit:typo
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085:
> nit: ..it..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085: s. Returns true if reload of the partition succeeds,
:* false othe
> Instead of referring to specific methods, rephrase it to something like re
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089
PS2, Line 2089: reloadP
> nit:call "reload" to be consistent ?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102
PS2, Line 1102: per(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ADD_PARTITION));
: if (event.getMessage() == null) {
: throw new IllegalStateException(debugString("Event
messag
> nit: Not super clear what this is. How are HMS transactional semantics rela
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106:
> Would be good if you can log a useful information about the event like numb
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: {
: AddPartitionMessage addPartitionMessage_ =
: MetastoreEventsProcessor.getMessageFactory()
: .getDeserializer()
: .getAddPartitionMessage(event.getMessage());
: addedPartitions_ =
:
Lists.newArrayList(addPartitionMessage_.getPartitionObjs());
: Preconditions.checkState(addedPartitions_.size() > 0);
: //
> We can avoid one unnecessary conversion from Partition -> partSpecMap -> TP
Created static method in base class to do this.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118
PS2, Line 1118:
> nit: include the partition name too
This scenario is only if the table is not present in the catalog, Next load of
the table will refresh all the partitions anyway. Do you think logging which
partition failed is useful to the user in this case?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125
PS2, Line 1125:
> include the partition name
In success case, it can be a really log list of partitions. (eg:- dynamically
loaded partitions)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: */
: private AlterPartitionEvent(CatalogServiceCatalog catalog,
Metrics metrics,
: NotificationEvent event) throws
MetastoreNotificationException {
: super(catalog, metrics, event);
:
Preconditions.checkState(eventType_.equals(MetastoreEventType.ALTER_PARTITION));
: Preconditions.checkNotNull(event.getMessage());
: AlterPartitionMessage alterPartitionMessage =
:
MetastoreEventsProcessor.getMessageFactory().getDeserializer()
:
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. .. IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affetced in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 208 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/3 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
