[
https://issues.apache.org/jira/browse/IMPALA-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821453#comment-17821453
]
ASF subversion and git services commented on IMPALA-12827:
----------------------------------------------------------
Commit 5250cc14b63114fa3976784d6bac39d743e4011a in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5250cc14b ]
IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id
is cleaned up
HdfsTable tracks the ValidWriteIdList from HMS. When the table is
reloaded, the ValidWriteIdList is updated to the latest state. An
ABORT_TXN event that is lagging behind could match to aborted write ids
that have already been cleaned up by the HMS housekeeping thread. Such
write ids can't be found in the cached ValidWriteIdList as opened or
aborted write ids. This hits a Precondition check and fails the event
processing.
This patch fixes the check to allow this case. Also adds more logs for
dealing with write ids.
Tests
- Add custom-cluster test to start Hive with the housekeeping thread
turned on and verified that such ABORT_TXN event is processed
correctly.
Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Reviewed-on: http://gerrit.cloudera.org:8080/21071
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Precondition was hit in MutableValidReaderWriteIdList
> -----------------------------------------------------
>
> Key: IMPALA-12827
> URL: https://issues.apache.org/jira/browse/IMPALA-12827
> Project: IMPALA
> Issue Type: Bug
> Reporter: Csaba Ringhofer
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: ACID, catalog
>
> The callstack below led to stopping metastore event processor during an abort
> transaction event:
> {code}
> MetastoreEventsProcessor.java:899] Unexpected exception received while
> processing event
> Java exception follows:
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:486)
> at
> org.apache.impala.hive.common.MutableValidReaderWriteIdList.addAbortedWriteIds(MutableValidReaderWriteIdList.java:274)
> at org.apache.impala.catalog.HdfsTable.addWriteIds(HdfsTable.java:3101)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addWriteIdsToTable(CatalogServiceCatalog.java:3885)
> at
> org.apache.impala.catalog.events.MetastoreEvents$AbortTxnEvent.addAbortedWriteIdsToTables(MetastoreEvents.java:2775)
> at
> org.apache.impala.catalog.events.MetastoreEvents$AbortTxnEvent.process(MetastoreEvents.java:2761)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:522)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1052)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:881)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> Precondition:
> https://github.com/apache/impala/blob/2f14fd29c0b47fc2c170a7f0eb1cecaf6b9704f4/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java#L274
> I was not able to reproduce this so far.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
