Quanlong Huang created IMPALA-12141:
---------------------------------------
Summary: IllegalMonitorStateException when releasing the table lock
Key: IMPALA-12141
URL: https://issues.apache.org/jira/browse/IMPALA-12141
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
We saw event-processor went into the ERROR state due to an
IllegalMonitorStateException:
{noformat}
I0504 12:28:45.272922 189771 MetastoreEvents.java:401] EventId: 56369449
EventType: INSERT Incremented events skipped counter to 283902
I0504 12:28:45.272941 189771 MetastoreEvents.java:401] EventId: 56369449
EventType: INSERT Not processing the event as it is a self-event
I0504 14:28:45.283041 189771 MetastoreEvents.java:412] EventId: 56369450
EventType: INSERT Received exception Error during self-event evaluation for
table xxx.xxxx due to lock contention. Ignoring self-event evaluation
E0504 16:28:45.286149 189771 MetastoreEventsProcessor.java:684] Unexpected
exception received while processing event
Java exception follows:
java.lang.IllegalMonitorStateException
at
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
at
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
at org.apache.impala.catalog.Table.releaseWriteLock(Table.java:262)
at
org.apache.impala.service.CatalogOpExecutor.reloadPartitionIfExists(CatalogOpExecutor.java:3788)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartition(MetastoreEvents.java:633)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:851)
at
org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:835)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:346)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:772)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:670)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
E0504 16:28:45.286345 189771 MetastoreEventsProcessor.java:795] Notification
event is null
{noformat}
It's due to the following try-clause:
{code:java}
try {
tryWriteLock(table, reason); // throws InternalException if timeout (2h) to
get write lock
...
return numOfPartsReloaded;
} catch (TableLoadingException e) {
...
} catch (InternalException e) {
throw new CatalogException(
"Could not acquire lock on the table " + table.getFullName(), e);
} finally {
UnlockWriteLockIfErronouslyLocked();
table.releaseWriteLock();
}
{code}
tryWriteLock() will wait until timeout(2h) to get the table write lock. If
fails, it throws an InternalException. The finally-clause of releaseWriteLock()
is always invoked so it fails by lock not held by current thread.
{code:java}
private void tryWriteLock(Table tbl, String operation) throws
InternalException {
String type = tbl instanceof View ? "view" : "table";
if (!catalog_.tryWriteLock(tbl)) {
throw new InternalException(String.format("Error %s (for) %s %s due to " +
"lock contention.", operation, type, tbl.getFullName()));
}
}{code}
We should check if the lock is held by the current thread before calling
table.releaseWriteLock(), or move tryWriteLock() outside the try-clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
