[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. IMPALA-8633 : Insert event should not error when table does not exists When a insert event is received by the events processor it is possible that the table has already been removed from metastore. In such cases, the reloadTable method throws a NoSuchObjectException which is unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state. The patch adds exception handling for such cases so that the error can be ignored and a warning is logged. Ideally, if the table has been removed from metastore, this insert event should be followed by a drop-table event and hence the table is cleaned up from the catalog subsequently. Hence Event processor does not need to remove the table from the catalog. Also, the patch adds the exception trace to the error messages during processing of Insert events to improve debugging ability in case of such errors. Testing: Refactored the existing test for insert events into util methods which can be reused for other tests. Added a new test case which generates a insert event on a table which has been dropped. This test reproduces the scenario seen by debugging test failures in IMPALA-8567. The newly added test succeeds after the patch. Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Reviewed-on: http://gerrit.cloudera.org:8080/13548 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 185 insertions(+), 152 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Jun 2019 01:44:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3616/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 21:04:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 20:16:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4471/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 20:16:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 20:16:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. IMPALA-8633 : Insert event should not error when table does not exists When a insert event is received by the events processor it is possible that the table has already been removed from metastore. In such cases, the reloadTable method throws a NoSuchObjectException which is unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state. The patch adds exception handling for such cases so that the error can be ignored and a warning is logged. Ideally, if the table has been removed from metastore, this insert event should be followed by a drop-table event and hence the table is cleaned up from the catalog subsequently. Hence Event processor does not need to remove the table from the catalog. Also, the patch adds the exception trace to the error messages during processing of Insert events to improve debugging ability in case of such errors. Testing: Refactored the existing test for insert events into util methods which can be reused for other tests. Added a new test case which generates a insert event on a table which has been dropped. This test reproduces the scenario seen by debugging test failures in IMPALA-8567. The newly added test succeeds after the patch. Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 185 insertions(+), 153 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13548/7 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 6: Carrying forward Todd's +2 from earlier -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 20:15:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/13548/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13548/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@825 PS6, Line 825: e : . > tiny nit: can you move 'e' to the next line? this breaking is pretty unnatu Fixed. My IDE formats it like this for some reason. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 20:15:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 6: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13548/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13548/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@825 PS6, Line 825: e : . tiny nit: can you move 'e' to the next line? this breaking is pretty unnatural -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 13 Jun 2019 19:56:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 4: > (1 comment) > > Is there any e2e test we could write to stress these paths? eg if > you were to run the metastore poller with a 5 second frequency, and > you did some random sequence of hive operations, would it end up > triggering it? I'm wondering whether we have similar issues in > other events. For example, if you ALTER TABLE followed by DROP, > will Impala get desynchronized? Created IMPALA-8661 to create random batch of event to stress event processor. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 12 Jun 2019 20:44:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 11 Jun 2019 20:50:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3565/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 11 Jun 2019 20:47:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@755 PS4, Line 755: BLE event is processe > I see you are creating the insert events by calling the HMS API. You can pr Done -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 11 Jun 2019 20:05:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. IMPALA-8633 : Insert event should not error when table does not exists When a insert event is received by the events processor it is possible that the table has already been removed from metastore. In such cases, the reloadTable method throws a NoSuchObjectException which is unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state. The patch adds exception handling for such cases so that the error can be ignored and a warning is logged. Ideally, if the table has been removed from metastore, this insert event should be followed by a drop-table event and hence the table is cleaned up from the catalog subsequently. Hence Event processor does not need to remove the table from the catalog. Also, the patch adds the exception trace to the error messages during processing of Insert events to improve debugging ability in case of such errors. Testing: Refactored the existing test for insert events into util methods which can be reused for other tests. Added a new test case which generates a insert event on a table which has been dropped. This test reproduces the scenario seen by debugging test failures in IMPALA-8567. The newly added test succeeds after the patch. Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 185 insertions(+), 153 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13548/6 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. IMPALA-8633 : Insert event should not error when table does not exists When a insert event is received by the events processor it is possible that the table has already been removed from metastore. In such cases, the reloadTable method throws a NoSuchObjectException which is unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state. The patch adds exception handling for such cases so that the error can be ignored and a warning is logged. Ideally, if the table has been removed from metastore, this insert event should be followed by a drop-table event and hence the table is cleaned up from the catalog subsequently. Hence Event processor does not need to remove the table from the catalog. Also, the patch adds the exception trace to the error messages during processing of Insert events to improve debugging ability in case of such errors. Testing: Refactored the existing test for insert events into util methods which can be reused for other tests. Added a new test case which generates a insert event on a table which has been dropped. This test reproduces the scenario seen by debugging test failures in IMPALA-8567. The newly added test succeeds after the patch. Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 184 insertions(+), 134 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13548/5 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13548 )
Change subject: IMPALA-8633 : Insert event should not error when table does not
exists
..
Patch Set 4:
(1 comment)
> (1 comment)
>
> Is there any e2e test we could write to stress these paths? eg if
> you were to run the metastore poller with a 5 second frequency, and
> you did some random sequence of hive operations, would it end up
> triggering it? I'm wondering whether we have similar issues in
> other events. For example, if you ALTER TABLE followed by DROP,
> will Impala get desynchronized?
For all the other events the event processor takes action if the
table/partition exists. For example in case of alter table, drop table event
sequence, the alter event does a invalidateTableIfExists so in this particular
case it will be no-op.
I have been thinking of what is the best way to add a test stress these code
paths. I was thinking may be writing a junit test which creates random batches
of the events for the eventProcessor is simpler and faster way. An equivalent
e2e test would be to run random hive queries for a set of given queries but its
needs to have a more complex logic of all the various ways you can write a
alter query (change col, add/drop partition, rename etc). Any thoughts? I can
create a separate JIRA to do that in a separate patch if that is okay with you.
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@829
PS4, Line 829: }
> should this have an else { throw e }?
Yes, Thanks for catching that.
--
To view, visit http://gerrit.cloudera.org:8080/13548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693
Gerrit-Change-Number: 13548
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 11 Jun 2019 18:24:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 4: (1 comment) Thanks for creating a test case for non-existent entities. Patch looks good to me other than a minor comment below. http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@755 PS4, Line 755: createFakeInsertEvent I see you are creating the insert events by calling the HMS API. You can probably remove this method? -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 10 Jun 2019 20:46:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13548 )
Change subject: IMPALA-8633 : Insert event should not error when table does not
exists
..
Patch Set 4:
(1 comment)
Is there any e2e test we could write to stress these paths? eg if you were to
run the metastore poller with a 5 second frequency, and you did some random
sequence of hive operations, would it end up triggering it? I'm wondering
whether we have similar issues in other events. For example, if you ALTER TABLE
followed by DROP, will Impala get desynchronized?
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13548/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@829
PS4, Line 829: }
should this have an else { throw e }?
--
To view, visit http://gerrit.cloudera.org:8080/13548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693
Gerrit-Change-Number: 13548
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 10 Jun 2019 19:10:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3531/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 07 Jun 2019 01:33:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. IMPALA-8633 : Insert event should not error when table does not exists When a insert event is received by the events processor it is possible that the table has already been removed from metastore. In such cases, the reloadTable method throws a NoSuchObjectException which is unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state. The patch adds exception handling for such cases so that the error can be ignored and a warning is logged. Ideally, if the table has been removed from metastore, this insert event should be followed by a drop-table event and hence the table is cleaned up from the catalog subsequently. Hence Event processor does not need to remove the table from the catalog. Also, the patch adds the exception trace to the error messages during processing of Insert events to improve debugging ability in case of such errors. Testing: Refactored the existing test for insert events into util methods which can be reused for other tests. Added a new test case which generates a insert event on a table which has been dropped. This test reproduces the scenario seen by debugging test failures in IMPALA-8567. The newly added test succeeds after the patch. Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 180 insertions(+), 131 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13548/4 -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13548 ) Change subject: IMPALA-8633 : Insert event should not error when table does not exists .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13548/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/13548/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@631 PS2, Line 631: msTbl = metaStoreClient.getHiveClient().getTable(TEST_DB_NAME, createInsertDropTable); > line too long (92 > 90) Done http://gerrit.cloudera.org:8080/#/c/13548/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@739 PS2, Line 739: Path parentPath = partition == null ? new Path(msTbl.getSd().getLocation()) : new Path(partition.getSd().getLocation()); > line too long (124 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/13548 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693 Gerrit-Change-Number: 13548 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 07 Jun 2019 01:14:39 + Gerrit-HasComments: Yes
