[
https://issues.apache.org/jira/browse/CASSANDRA-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121729#comment-14121729
]
Pavel Yaskevich commented on CASSANDRA-7594:
--------------------------------------------
Looks like the problem is that thread pool is rejecting execution of the
selector.
{noformat}
"main" prio=9 tid=0x00007fd2a4002800 nid=0xd07 waiting on condition
[0x000000010d915000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007df31bbd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.offer(SynchronousQueue.java:896)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:65)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:150)
at com.lmax.disruptor.WorkerPool.start(WorkerPool.java:133)
at
com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.<init>(TDisruptorServer.java:525)
at
com.thinkaurelius.thrift.TDisruptorServer.<init>(TDisruptorServer.java:221)
at
org.apache.cassandra.thrift.THsHaDisruptorServer.<init>(THsHaDisruptorServer.java:51)
at
org.apache.cassandra.thrift.THsHaDisruptorServer$Factory.buildTServer(THsHaDisruptorServer.java:105)
at
org.apache.cassandra.thrift.TServerCustomFactory.buildTServer(TServerCustomFactory.java:55)
at
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.<init>(ThriftServer.java:131)
at org.apache.cassandra.thrift.ThriftServer.start(ThriftServer.java:58)
at
org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:410)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:470)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
{noformat}
[~philipthompson] Does dtest try to stop/start thrift after every test or
something similar?
> Disruptor Thrift server worker thread pool not adjustable
> ---------------------------------------------------------
>
> Key: CASSANDRA-7594
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7594
> Project: Cassandra
> Issue Type: Bug
> Reporter: Rick Branson
> Assignee: Pavel Yaskevich
> Fix For: 2.0.11
>
> Attachments: CASSANDRA-7594.patch,
> disruptor-thrift-server-0.3.6-SNAPSHOT.jar, jstack.txt
>
>
> For the THsHaDisruptorServer, there may not be enough threads to run blocking
> StorageProxy methods. The current number of worker threads is hardcoded at 2
> per selector, so 2 * numAvailableProcessors(), or 64 threads on a 16-core
> hyperthreaded machine. StorageProxy methods block these threads, so this puts
> an upper bound on the throughput if hsha is enabled. If operations take 10ms
> on average, the node can only handle a maximum of 6,400 operations per
> second. This is a regression from hsha on 1.2.x, where the thread pool was
> tunable using rpc_min_threads and rpc_max_threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
