[
https://issues.apache.org/jira/browse/IMPALA-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala resolved IMPALA-12141.
--------------------------------------------
Fix Version/s: Impala 4.5.0
Resolution: Fixed
> IllegalMonitorStateException when trying to release the table lock
> ------------------------------------------------------------------
>
> Key: IMPALA-12141
> URL: https://issues.apache.org/jira/browse/IMPALA-12141
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Labels: catalog-2024
> Fix For: Impala 4.5.0
>
>
> We saw event-processor went into the ERROR state due to an
> IllegalMonitorStateException:
> {noformat}
> I0504 12:28:45.272922 189771 MetastoreEvents.java:401] EventId: 56369449
> EventType: INSERT Incremented events skipped counter to 283902
> I0504 12:28:45.272941 189771 MetastoreEvents.java:401] EventId: 56369449
> EventType: INSERT Not processing the event as it is a self-event
> I0504 14:28:45.283041 189771 MetastoreEvents.java:412] EventId: 56369450
> EventType: INSERT Received exception Error during self-event evaluation for
> table xxx.xxxx due to lock contention. Ignoring self-event evaluation
> E0504 16:28:45.286149 189771 MetastoreEventsProcessor.java:684] Unexpected
> exception received while processing event
> Java exception follows:
> java.lang.IllegalMonitorStateException
> at
> java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
> at
> java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
> at
> java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
> at org.apache.impala.catalog.Table.releaseWriteLock(Table.java:262)
> at
> org.apache.impala.service.CatalogOpExecutor.reloadPartitionIfExists(CatalogOpExecutor.java:3788)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartition(MetastoreEvents.java:633)
> at
> org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:851)
> at
> org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:835)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:346)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:772)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:670)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> E0504 16:28:45.286345 189771 MetastoreEventsProcessor.java:795] Notification
> event is null
> {noformat}
> It's due to the following try-clause:
> {code:java}
> try {
> tryWriteLock(table, reason); // throws InternalException if timeout (2h) to
> get write lock
> ...
> return numOfPartsReloaded;
> } catch (TableLoadingException e) {
> ...
> } catch (InternalException e) {
> throw new CatalogException(
> "Could not acquire lock on the table " + table.getFullName(), e);
> } finally {
> UnlockWriteLockIfErronouslyLocked();
> table.releaseWriteLock();
> }
> {code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4604-L4641
> tryWriteLock() will wait until timeout(2h) to get the table write lock. If
> fails, it throws an InternalException. The finally-clause of
> releaseWriteLock() is always invoked so it fails by lock not held by current
> thread.
> {code:java}
> private void tryWriteLock(Table tbl, String operation) throws
> InternalException {
> String type = tbl instanceof View ? "view" : "table";
> if (!catalog_.tryWriteLock(tbl)) {
> throw new InternalException(String.format("Error %s (for) %s %s due to
> " +
> "lock contention.", operation, type, tbl.getFullName()));
> }
> }{code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L7309-L7323
> We should check if the lock is held by the current thread before calling
> table.releaseWriteLock(), or move tryWriteLock() outside the try-clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
