Quanlong Huang created IMPALA-12680:
---------------------------------------
Summary: NullPointerException in addHmsPartitions() during
MetastoreEventsProcessor switch state from PAUSED to ACTIVE
Key: IMPALA-12680
URL: https://issues.apache.org/jira/browse/IMPALA-12680
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Event processing is paused during a global INVALIDATE METADATA.
catalog_.isEventProcessingActive() returns false in this state.
If an AlterTableAddPartition statement is running during that time, we could
pass in a null value for the 'partitionToEventId' map in here:
{code:java}
Map<String, Long> partitionToEventId = catalog_.isEventProcessingActive()
?
Maps.newHashMap() : null;
List<Partition> addedHmsPartitions =
addHmsPartitionsInTransaction(msClient,
tbl, allHmsPartitionsToAdd, partitionToEventId, ifNotExists); {code}
https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4404-L4407
If the global INVALIDATE METADATA finishes and event processing is back to the
ACTIVE state before the AlterTableAddPartition statement runs into
addHmsPartitions(), we will have an non-empty 'partitionToEventSubMap' here:
{code:java}
List<NotificationEvent> events =
getNextMetastoreEventsIfEnabled(eventId,
event -> AddPartitionEvent.ADD_PARTITION_EVENT_TYPE
.equals(event.getEventType())
&& msTbl.getDbName().equalsIgnoreCase(event.getDbName())
&&
msTbl.getTableName().equalsIgnoreCase(event.getTableName()));
Map<Partition, Long> partitionToEventSubMap = Maps.newHashMap();
getPartitionsFromEvent(events, partitionToEventSubMap);
// set the eventId to last one which we received so the we fetch the
next
// set of events correctly
if (!events.isEmpty()) {
eventId = events.get(events.size() - 1).getEventId();
}
if (partitionToEventSubMap.isEmpty()) {
// if partitions couldn't be fetched from events, use the one
returned by
// add_partitions call above.
addedHmsPartitions.addAll(addedPartitions);
} else {
Preconditions.checkNotNull(partitionToEventId); // <-- This will
fail{code}
https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5052-L5069
Then the AlterTableAddPartition statement fails with NullPointerException:
{code:java}
I0104 02:46:32.075830 1010 jni-util.cc:302] 4a4eae34f60ba947:b6b2bcfc00000000]
java.lang.NullPointerException
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889)
at
org.apache.impala.service.CatalogOpExecutor.addHmsPartitions(CatalogOpExecutor.java:5051)
at
org.apache.impala.service.CatalogOpExecutor.addHmsPartitionsInTransaction(CatalogOpExecutor.java:5082)
at
org.apache.impala.service.CatalogOpExecutor.alterTableAddPartitions(CatalogOpExecutor.java:4388)
at
org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1136)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:450)
at
org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:304)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:233)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:247)
at
org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:303){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
