Quanlong Huang created IMPALA-13126:
---------------------------------------
Summary: ReloadEvent.isOlderEvent() should hold the table read lock
Key: IMPALA-13126
URL: https://issues.apache.org/jira/browse/IMPALA-13126
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Sai Hemanth Gantasala
Saw an exception like this:
{noformat}
E0601 09:11:25.275251 246 MetastoreEventsProcessor.java:990] Unexpected
exception received while processing event
Java exception follows:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1469)
at java.util.HashMap$ValueIterator.next(HashMap.java:1498)
at
org.apache.impala.catalog.FeFsTable$Utils.getPartitionFromThriftPartitionSpec(FeFsTable.java:616)
at
org.apache.impala.catalog.HdfsTable.getPartitionFromThriftPartitionSpec(HdfsTable.java:597)
at org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:511)
at org.apache.impala.catalog.Catalog.getHdfsPartition(Catalog.java:489)
at
org.apache.impala.catalog.CatalogServiceCatalog.isPartitionLoadedAfterEvent(CatalogServiceCatalog.java:4024)
at
org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.isOlderEvent(MetastoreEvents.java:2754)
at
org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processTableEvent(MetastoreEvents.java:2729)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1107)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:531)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750) {noformat}
For a partition-level RELOAD event, ReloadEvent.isOlderEvent() needs to check
whether the corresponding partition is reloaded after the event. This should be
done after holding the table read lock. Otherwise, EventProcessor could hit the
error above when there are concurrent DDLs/DMLs modifying the partition list.
CC [~VenuReddy]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
