[
https://issues.apache.org/jira/browse/IMPALA-12459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768322#comment-17768322
]
Wenzhe Zhou commented on IMPALA-12459:
--------------------------------------
Event processing was blocked when it tried to acquire write log to add
partitions. Saw following stack traces frequently in jstacks of catalogd.
{code:java}
"MetastoreEventsProcessor [Attempting to write lock table
gdp_1h_ext_acct_sec.yodlee_cf_inv_hol with a timeout of 7200000 ms]" #120
daemon prio=5 os_prio=0 tid=0x0000000018e24800 nid=0x3256
sleeping[0x00007f9fc0e2e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.impala.catalog.CatalogServiceCatalog.tryLock(CatalogServiceCatalog.java:468)
at
org.apache.impala.catalog.CatalogServiceCatalog.tryWriteLock(CatalogServiceCatalog.java:436)
at
org.apache.impala.service.CatalogOpExecutor.tryWriteLock(CatalogOpExecutor.java:6677)
at
org.apache.impala.service.CatalogOpExecutor.addPartitionsIfNotRemovedLater(CatalogOpExecutor.java:3869)
at
org.apache.impala.catalog.events.MetastoreEvents$AddPartitionEvent.process(MetastoreEvents.java:1698)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:434)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:846)
- locked <0x00007fa2ab3393c8> (a
org.apache.impala.catalog.events.MetastoreEventsProcessor)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:687)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor$$Lambda$69/210949780.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
> Try to avoid taking global version lock during HMS event processing
> -------------------------------------------------------------------
>
> Key: IMPALA-12459
> URL: https://issues.apache.org/jira/browse/IMPALA-12459
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Csaba Ringhofer
> Priority: Major
>
> Currently if the event is not filtered out by type and not disabled by table
> property then catalogd checks the database property which needs the version
> lock for a brief time:
> https://github.com/apache/impala/blob/94f4f1d82461d8f71fbd0d2e9082aa29b5f53a89/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L800
> https://github.com/apache/impala/blob/94f4f1d82461d8f71fbd0d2e9082aa29b5f53a89/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L1303
> Getting the readlock shouldn't be a problem in general, but it would be
> better if ignored events couldn't be blocked (for example rename can take the
> lock for a longer time). A solution could be to create a separate map and
> lock just for this property so that events could check it with a less
> contented lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
