[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matches with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which has been dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Reviewed-on: http://gerrit.cloudera.org:8080/16129 Reviewed-by: Anurag Mantripragada Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) Approvals: Anurag Mantripragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 01 Jul 2020 00:16:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6470/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:35:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6469/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:32:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6083/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:23:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 30 Jun 2020 19:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16129 ) Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matches with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which has been dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/16129/2 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9913: Use table id to match the table in drop table event
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16129 Change subject: IMPALA-9913: Use table id to match the table in drop table event .. IMPALA-9913: Use table id to match the table in drop table event This patch modifies the matching logic of the table while processing a DROP_TABLE event. The EventsProcessor currently relies on createTime of the table to determine if the table in the catalog matching with the table in the event. It is possible that if there is a time-skew between the different HMS instances and a same table is dropped and recreated, the the creation time of the table which is present in the catalog is less than or equals to the creationTime of a table which is dropped. In such case, there is a false negative during the matching and the DROP_TABLE event is skipped. HIVE-20556 exposes a id field of the table in which can be used to uniquely identify a table. A table which is dropped and recreated with the same name has a different id. This patch makes use of this id to match the table in catalog with the table from the HMS event. Note that the DROP_DATABASE event continues to use createTime for matching. We should use a database id for it when it is available. I have created HIVE-23785 for tracking that on the HMS side. Testing: 1. Ran Events processor tests. Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/16129/1 -- To view, visit http://gerrit.cloudera.org:8080/16129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id6a80bbf5757e46318af1b57911fc127d7dd1f01 Gerrit-Change-Number: 16129 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
