[
https://issues.apache.org/jira/browse/IMPALA-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253287#comment-17253287
]
ASF subversion and git services commented on IMPALA-6671:
---------------------------------------------------------
Commit 2fccd82590d747d834b8be6f3b05bb446d9bac12 in impala's branch
refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2fccd82 ]
IMPALA-6671: Skip locked tables from topic updates
This change adds a mechanism for topic-update thread
to skip a table which is locked for more than a
configurable interval from the topic updates. This is
especially useful in scenarios where long running operations on a
locked table (refresh, recover partitions, compute stats) block the
topic update thread. This causes unrelated queries which are
waiting on metadata via topic updates (catalog-v1 mode)
to unnecessarily block.
The ideal solution of this problem would be to make HdfsTable
immutable so that there is no need for table lock. But that
is large change and not easily portable to older releases
of Impala. It would be taken up as a separate patch.
This change introduces 2 new configurations for catalogd:
1. topic_update_tbl_max_wait_time_ms: This defines the
maximum time in msecs the topic update thread waits on a locked table
before skipping the table from that iteration of topic updates.
The default value is 500. If this configuration is set to 0
the lock with timeout for topic update thread is disabled.
2. catalog_max_lock_skipped_topic_updates: This defines
the maximum number of distinct lock operations which are skipped
by topic update thread due to lock contention. Once this limit
is reached, topic update thread will block until it acquires
the table lock and adds it to the updates.
Testing:
1. Added a test case which introduces a simulated delay
in a few potentially long running statements. This causes the table
to be locked for a long time. The topic update thread skips
that table from updates and unrelated queries are unblocked
since they receive the required metadata from updates.
2. Added a test where multiple threads run blocking statements
in a loop to stress the table lock. It makes sure that topic
update thread is not starved and eventually blocks
on table lock by hitting the limit defined by
catalog_max_lock_skipped_topic_updates.
3. Ran exhaustive tests with default configurations.
Change-Id: Ic657b96edbcdc94c6b906e7ca59291f4e4715655
Reviewed-on: http://gerrit.cloudera.org:8080/16549
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Metadata operations that modify a table blocks topic updates for other
> unrelated operations
> -------------------------------------------------------------------------------------------
>
> Key: IMPALA-6671
> URL: https://issues.apache.org/jira/browse/IMPALA-6671
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
> Reporter: Mostafa Mokhtar
> Assignee: Vihang Karajgaonkar
> Priority: Critical
> Labels: catalog-server, perfomance
>
> Metadata operations that mutate the state of a table like "compute stats foo"
> or "alter recover partitions" block topic updates for read only operations
> against unrelated tables as "describe bar".
> Thread for blocked operation
> {code:java}
> "Thread-7" prio=10 tid=0x0000000011613000 nid=0x21b3b waiting on condition
> [0x00007f5f2ef52000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007f6f57ff0240> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDeltaHelper(CatalogServiceCatalog.java:639)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDelta(CatalogServiceCatalog.java:611)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addDatabaseToCatalogDelta(CatalogServiceCatalog.java:567)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getCatalogDelta(CatalogServiceCatalog.java:449)
> at
> org.apache.impala.service.JniCatalog.getCatalogDelta(JniCatalog.java:126)
> {code}
> Thread for blocking operation
> {code:java}
> "Thread-130" prio=10 tid=0x00000000113d5800 nid=0x2499d runnable
> [0x00007f5ef80d0000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00007f5fffcd9f18> (a java.io.BufferedInputStream)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at
> org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at
> org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1639)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1626)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:609)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy10.add_partitions(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTableRecoverPartitions(CatalogOpExecutor.java:2651)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:525)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:262)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
