Weizisheng created IMPALA-12181:
-----------------------------------
Summary: Catalogd can't process hms event once adding an already
exists partition
Key: IMPALA-12181
URL: https://issues.apache.org/jira/browse/IMPALA-12181
Project: IMPALA
Issue Type: Bug
Components: Catalog, Frontend
Affects Versions: Impala 3.4.0
Reporter: Weizisheng
Add a already exists parttion from Hive using
{code:java}
alter table xxx add if not exists partition (ds="***"){code}
Impala will received 2 events, the 1st one contains no partition and the 2nd
one contains the existing partition but with null parameters which will cause
NPE， then event process status turns to ERROR...
{code:java}
I0526 17:14:57.637362 2146 MetastoreEventsProcessor.java:482] Received 2
events. Start event id : 12
I0526 17:14:57.652266 2146 MetastoreEvents.java:233] Total number of events
received: 2 Total number of events filtered out: 0
I0526 17:14:57.652372 2146 MetastoreEvents.java:366] EventId: 13 EventType:
ADD_PARTITION Partition list is empty. Ignoring this event.
E0526 17:14:57.654448 2146 MetastoreEventsProcessor.java:527] Unexpected
exception received while processing event
Java exception follows:
java.lang.NullPointerException
at java.util.HashMap.putMapEntries(HashMap.java:501)
at java.util.HashMap.putAll(HashMap.java:785)
at
org.apache.impala.catalog.events.MetastoreEvents$AddPartitionEvent.getSelfEventContext(MetastoreEvents.java:1320)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.isSelfEvent(MetastoreEvents.java:425)
at
org.apache.impala.catalog.events.MetastoreEvents$AddPartitionEvent.process(MetastoreEvents.java:1333)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:311)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:615)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:513)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
E0526 17:14:57.654541 2146 MetastoreEventsProcessor.java:637] Notification
event is null
W0526 17:14:58.654865 2146 MetastoreEventsProcessor.java:506] Event processing
is skipped since status is ERROR. Last synced event id is 13 {code}
PS:
before reporting this issue i've already found IMPALA-8847 and seems it's been
fixed in Impala-3.3.0. I'm using Impala-3.4.0 with Hive-3.1.0
From
[https://github.com/apache/impala/commit/0ff4f45#diff-531be00fa82d527a36363426497585e5cfcfaabba18766ef875d609c8151975fR1372]
, IMPALA-8847 Ignore such events when {{addedPartitions_.isEmpty()}} is ture.
But as i said, hms will generate 2 events. The 1st event can be ignore, the 2nd
one will be handled...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
