[
https://issues.apache.org/jira/browse/IMPALA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated IMPALA-8968:
-----------------------------------------
Description:
When event processing is turned on, self-events are not detected for DATABASE
level events (create/alter/drop database). Reproduced using the below
statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the events processor could not detect self-events, it will try to process
the ALTER_DATABASE event created by dropping the function. However, it doesn't
find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.044441 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525]
Unexpected exception received while processing eventJava exception
follows:org.apache.impala.catalog.events.MetastoreNotificationException: Unable
to process event 30078 of type ALTER_DATABASE. Event processing will be
stopped. at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.impala.catalog.DatabaseNotFoundException: Database test not found at
org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
at
org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
Notification event is nullW0923 16:09:56.057376 6727
MetastoreEventsProcessor.java:504] Event processing is skipped since status is
ERROR. Last synced event id is 30077{code}
was:
Self-events are not detected for DATABASE level events (create/alter/drop
database). Reproduced using the below statements:
{code:java}
CREATE DATABASE test;
CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
'/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
DROP FUNCTION test.fn_test(INT, STRING);
DROP DATABASE test CASCADE; {code}
Since the events processor could not detect self-events, it will try to process
the ALTER_DATABASE event created by dropping the function. However, it doesn't
find the DB and errors out like below:
{code:java}
I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
events. Start event id : 30077I0923 16:09:46.042317 6727
MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
database testI0923 16:09:46.044441 6727 MetastoreEvents.java:382] EventId:
30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total number
of events received: 2 Total number of events filtered out: 0I0923
16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
ALTER_DATABASE Received exception Database test not found. Ignoring self-event
evaluationE0923 16:09:46.056273 6727 MetastoreEventsProcessor.java:525]
Unexpected exception received while processing eventJava exception
follows:org.apache.impala.catalog.events.MetastoreNotificationException: Unable
to process event 30078 of type ALTER_DATABASE. Event processing will be
stopped. at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.impala.catalog.DatabaseNotFoundException: Database test not found at
org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
at
org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
Notification event is nullW0923 16:09:56.057376 6727
MetastoreEventsProcessor.java:504] Event processing is skipped since status is
ERROR. Last synced event id is 30077{code}
> Fix self-event detection on database events.
> --------------------------------------------
>
> Key: IMPALA-8968
> URL: https://issues.apache.org/jira/browse/IMPALA-8968
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Anurag Mantripragada
> Assignee: Anurag Mantripragada
> Priority: Major
>
> When event processing is turned on, self-events are not detected for DATABASE
> level events (create/alter/drop database). Reproduced using the below
> statements:
> {code:java}
> CREATE DATABASE test;
> CREATE FUNCTION test.fn_test (INT, STRING) RETURNS BOOLEAN LOCATION
> '/test-warehouse/dummy.jar' SYMBOL='com.cloudera.impala.TestUdf';
> DROP FUNCTION test.fn_test(INT, STRING);
> DROP DATABASE test CASCADE; {code}
> Since the events processor could not detect self-events, it will try to
> process the ALTER_DATABASE event created by dropping the function. However,
> it doesn't find the DB and errors out like below:
> {code:java}
> I0923 16:09:46.042317 6727 MetastoreEventsProcessor.java:480] Received 2
> events. Start event id : 30077I0923 16:09:46.042317 6727
> MetastoreEventsProcessor.java:480] Received 2 events. Start event id :
> 30077I0923 16:09:46.042853 6727 MetastoreEvents.java:382] EventId: 30078
> EventType: ALTER_DATABASE Creating event 30078 of type ALTER_DATABASE on
> database testI0923 16:09:46.044441 6727 MetastoreEvents.java:382] EventId:
> 30079 EventType: DROP_DATABASE Creating event 30079 of type DROP_DATABASE on
> database testI0923 16:09:46.050657 6727 MetastoreEvents.java:231] Total
> number of events received: 2 Total number of events filtered out: 0I0923
> 16:09:46.051167 6727 MetastoreEvents.java:382] EventId: 30078 EventType:
> ALTER_DATABASE Received exception Database test not found. Ignoring
> self-event evaluationE0923 16:09:46.056273 6727
> MetastoreEventsProcessor.java:525] Unexpected exception received while
> processing eventJava exception
> follows:org.apache.impala.catalog.events.MetastoreNotificationException:
> Unable to process event 30078 of type ALTER_DATABASE. Event processing will
> be stopped. at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:614)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:511)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.impala.catalog.DatabaseNotFoundException: Database test not found
> at
> org.apache.impala.catalog.CatalogServiceCatalog.updateDb(CatalogServiceCatalog.java:1060)
> at
> org.apache.impala.catalog.events.MetastoreEvents$AlterDatabaseEvent.process(MetastoreEvents.java:1225)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:316)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:609)
> ... 8 moreE0923 16:09:46.056489 6727 MetastoreEventsProcessor.java:631]
> Notification event is nullW0923 16:09:56.057376 6727
> MetastoreEventsProcessor.java:504] Event processing is skipped since status
> is ERROR. Last synced event id is 30077{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
