[
https://issues.apache.org/jira/browse/IMPALA-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910422#comment-17910422
]
ASF subversion and git services commented on IMPALA-12141:
----------------------------------------------------------
Commit e7c97439d12644af2a59bcd893aa7b8bdcb83b36 in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e7c97439d ]
IMPALA-12141: EP shouldn't fail while releasing write lock if the lock
is not held previously
Without IMPALA-12832, Event Processor (EP) is going into error state
when there is an issue while obtaining a table write lock because the
finally-clause of releaseWriteLock() is always invoked even if the lock
is not held by current thread. This patch addresses the problem by
checking if the table holds write lock before releasing it.
Note: With IMPALA-12832, the EP invalidates the table when an error is
encountered which is still an overhead. With this patch EP will neither
goes into error state nor invalidates when this issue is encountered.
Testing:
- Added an end-to-end to verify the same.
Change-Id: Ib2e4c965796dd515ab8549efa616f72510ca447f
Reviewed-on: http://gerrit.cloudera.org:8080/22080
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> IllegalMonitorStateException when trying to release the table lock
> ------------------------------------------------------------------
>
> Key: IMPALA-12141
> URL: https://issues.apache.org/jira/browse/IMPALA-12141
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Labels: catalog-2024
>
> We saw event-processor went into the ERROR state due to an
> IllegalMonitorStateException:
> {noformat}
> I0504 12:28:45.272922 189771 MetastoreEvents.java:401] EventId: 56369449
> EventType: INSERT Incremented events skipped counter to 283902
> I0504 12:28:45.272941 189771 MetastoreEvents.java:401] EventId: 56369449
> EventType: INSERT Not processing the event as it is a self-event
> I0504 14:28:45.283041 189771 MetastoreEvents.java:412] EventId: 56369450
> EventType: INSERT Received exception Error during self-event evaluation for
> table xxx.xxxx due to lock contention. Ignoring self-event evaluation
> E0504 16:28:45.286149 189771 MetastoreEventsProcessor.java:684] Unexpected
> exception received while processing event
> Java exception follows:
> java.lang.IllegalMonitorStateException
> at
> java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
> at
> java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
> at
> java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
> at org.apache.impala.catalog.Table.releaseWriteLock(Table.java:262)
> at
> org.apache.impala.service.CatalogOpExecutor.reloadPartitionIfExists(CatalogOpExecutor.java:3788)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartition(MetastoreEvents.java:633)
> at
> org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:851)
> at
> org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:835)
> at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:346)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:772)
> at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:670)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> E0504 16:28:45.286345 189771 MetastoreEventsProcessor.java:795] Notification
> event is null
> {noformat}
> It's due to the following try-clause:
> {code:java}
> try {
> tryWriteLock(table, reason); // throws InternalException if timeout (2h) to
> get write lock
> ...
> return numOfPartsReloaded;
> } catch (TableLoadingException e) {
> ...
> } catch (InternalException e) {
> throw new CatalogException(
> "Could not acquire lock on the table " + table.getFullName(), e);
> } finally {
> UnlockWriteLockIfErronouslyLocked();
> table.releaseWriteLock();
> }
> {code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4604-L4641
> tryWriteLock() will wait until timeout(2h) to get the table write lock. If
> fails, it throws an InternalException. The finally-clause of
> releaseWriteLock() is always invoked so it fails by lock not held by current
> thread.
> {code:java}
> private void tryWriteLock(Table tbl, String operation) throws
> InternalException {
> String type = tbl instanceof View ? "view" : "table";
> if (!catalog_.tryWriteLock(tbl)) {
> throw new InternalException(String.format("Error %s (for) %s %s due to
> " +
> "lock contention.", operation, type, tbl.getFullName()));
> }
> }{code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L7309-L7323
> We should check if the lock is held by the current thread before calling
> table.releaseWriteLock(), or move tryWriteLock() outside the try-clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
