Quanlong Huang created IMPALA-15242:
---------------------------------------
Summary: getOrLoadTable should hold table write lock in reloading
partitions
Key: IMPALA-15242
URL: https://issues.apache.org/jira/browse/IMPALA-15242
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Attachments: getOrLoadTable-bug-repro.patch
CatalogServiceCatalog.getOrLoadTable() will reload partitions of a
transactional table when there are stale partitions. However, the check and
update operations are not protected by table write lock. A concurrrent
modification, e.g. EventProcessor processing COMMIT_COMPACTION_EVENT, could
modify the partitions map and cause the reload in getOrLoadTable() fails.
Here is how the stale partitions are checked. It's protected by the table read
lock. It then release the lock.
{code:java}
List<HdfsPartition.Builder> partsToBeRefreshed = Collections.emptyList();
...
readLock(tbl, catalogTimeline);
try {
partsToBeRefreshed =
AcidUtils.getPartitionsForRefreshingFileMetadata(this,
(HdfsTable) tbl);
} finally {
tbl.readLock().unlock();
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2864-L2865]
The table write lock will be acquired in refreshFileMetadata() when refreshing
the partitions:
{code:java}
if (!partsToBeRefreshed.isEmpty()) {
return refreshFileMetadata((HdfsTable) tbl, partsToBeRefreshed,
catalogTimeline);
}{code}
[https://github.com/apache/impala/blob/01062ee897703c0b6c54cfedbe104acf363ff410/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2891]
refreshFileMetadata() could fail if it uses stale partition ids.
getOrLoadTable() then fails, which consequently fails the coordinator request.
{noformat}
E20260804 19:20:44.249446 134413 JniUtil.java:198] Error in Getting partial
catalog object of TABLE:catalogd_meta_provider_test.insert_only. Time spent:
2s019ms
I20260804 19:20:44.250008 134413 jni-util.cc:335]
java.lang.IllegalStateException: Updating a non existing partition instance
id=25
at
com.google.common.base.Preconditions.checkState(Preconditions.java:586)
at
org.apache.impala.catalog.HdfsTable.updatePartition(HdfsTable.java:1060)
at
org.apache.impala.catalog.HdfsTable.updatePartitions(HdfsTable.java:1051)
at
org.apache.impala.catalog.CatalogServiceCatalog.refreshFileMetadata(CatalogServiceCatalog.java:4549)
at
org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2893)
at
org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:4379)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4301)
at
org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:4268)
at
org.apache.impala.service.JniCatalog.lambda$getPartialCatalogObject$10(JniCatalog.java:474)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerializeSilentStartAndFinish(JniCatalogOp.java:109)
at
org.apache.impala.service.JniCatalog.execAndSerializeSilentStartAndFinish(JniCatalog.java:256)
at
org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:473){noformat}
The issue can be reproduced by running
CatalogdMetaProviderTest.testTableFileMetadataAfterMinorCompaction after
applying the debug patch [^getOrLoadTable-bug-repro.patch] (on commit
01062ee897).
{code}
(pushd fe && mvn test
-Dtest=CatalogdMetaProviderTest#testTableFileMetadataAfterMinorCompaction){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
