[
https://issues.apache.org/jira/browse/IMPALA-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-12257.
-------------------------------------
Fix Version/s: Impala 4.3.0
Resolution: Fixed
> createInsertEvents failed by NullPointerException: Invalid partition name
> -------------------------------------------------------------------------
>
> Key: IMPALA-12257
> URL: https://issues.apache.org/jira/browse/IMPALA-12257
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 4.3.0
>
>
> INSERT on partitioned table could fail in createInsertEvents() if some
> updated partitions are missing in catalogd but actually exists in HMS. The
> failure is a NullPointerException, e.g.
> {code:java}
> I0630 14:44:29.779798 30287 jni-util.cc:288] java.lang.NullPointerException:
> Invalid partition name: p=0
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
> at
> org.apache.impala.catalog.HdfsTable.getPartitionsForNames(HdfsTable.java:1758)
> at
> org.apache.impala.service.CatalogOpExecutor.createInsertEvents(CatalogOpExecutor.java:6935)
> at
> org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:6830)
> at
> org.apache.impala.service.JniCatalog.lambda$updateCatalog$16(JniCatalog.java:471)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:230)
> at
> org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:470) {code}
> We've seen this in an issue caused by IMPALA-12256 in which the stale
> DROP_PARTITION event incorrectly drops the partition, causing inconsistency
> between catalogd and HMS on the partition list.
> To steadily reproduce the issue, we can disable HMS event-processing and
> manually make the partition list differs between catalogd and HMS.
> Start Impala with HMS event-processing disabled and create a partitioned
> table. Run a query on it to make it loaded in catalogd:
> {noformat}
> bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0
> impala> create table my_part (id int) partitioned by (p int) stored as
> textfile;
> impala> show partitions my_part;{noformat}
> Add one partition in Hive. Catalogd is not aware of it:
> {code:java}
> hive> alter table my_part add partition(p=0);
> {code}
> Then run INSERT on the same partition in Impala:
> {code:java}
> impala> insert into my_part partition(p=0) values (0);
> ERROR: NullPointerException: Invalid partition name: p=0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
