[
https://issues.apache.org/jira/browse/IMPALA-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved IMPALA-12680.
--------------------------------------------
Fix Version/s: Impala 4.5.0
Resolution: Fixed
> NullPointerException in addHmsPartitions() during MetastoreEventsProcessor
> switch state from PAUSED to ACTIVE
> -------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12680
> URL: https://issues.apache.org/jira/browse/IMPALA-12680
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Fix For: Impala 4.5.0
>
> Attachments: alterTableAddPartitionProfile.txt, catalogd.INFO.gz,
> impalad.INFO.gz
>
>
> Event processing is paused during a global INVALIDATE METADATA.
> catalog_.isEventProcessingActive() returns false in this state.
>
> If an AlterTableAddPartition statement is running during that time, we could
> pass in a null value for the 'partitionToEventId' map in here:
> {code:java}
> Map<String, Long> partitionToEventId =
> catalog_.isEventProcessingActive() ?
> Maps.newHashMap() : null;
> List<Partition> addedHmsPartitions =
> addHmsPartitionsInTransaction(msClient,
> tbl, allHmsPartitionsToAdd, partitionToEventId, ifNotExists); {code}
>
> https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4404-L4407
> If the global INVALIDATE METADATA finishes and event processing is back to
> the ACTIVE state before the AlterTableAddPartition statement runs into
> addHmsPartitions(), we will have an non-empty 'partitionToEventSubMap' here:
> {code:java}
> List<NotificationEvent> events =
> getNextMetastoreEventsIfEnabled(eventId,
> event -> AddPartitionEvent.ADD_PARTITION_EVENT_TYPE
> .equals(event.getEventType())
> && msTbl.getDbName().equalsIgnoreCase(event.getDbName())
> &&
> msTbl.getTableName().equalsIgnoreCase(event.getTableName()));
> Map<Partition, Long> partitionToEventSubMap = Maps.newHashMap();
> getPartitionsFromEvent(events, partitionToEventSubMap);
> // set the eventId to last one which we received so the we fetch the
> next
> // set of events correctly
> if (!events.isEmpty()) {
> eventId = events.get(events.size() - 1).getEventId();
> }
> if (partitionToEventSubMap.isEmpty()) {
> // if partitions couldn't be fetched from events, use the one
> returned by
> // add_partitions call above.
> addedHmsPartitions.addAll(addedPartitions);
> } else {
> Preconditions.checkNotNull(partitionToEventId); // <-- This will
> fail{code}
> https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5052-L5069
> Then the AlterTableAddPartition statement fails with NullPointerException:
> {code:java}
> I0104 02:46:32.075830 1010 jni-util.cc:302]
> 4a4eae34f60ba947:b6b2bcfc00000000] java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889)
> at
> org.apache.impala.service.CatalogOpExecutor.addHmsPartitions(CatalogOpExecutor.java:5051)
> at
> org.apache.impala.service.CatalogOpExecutor.addHmsPartitionsInTransaction(CatalogOpExecutor.java:5082)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTableAddPartitions(CatalogOpExecutor.java:4388)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1136)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:450)
> at
> org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:304)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:233)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:247)
> at
> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:303){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
