[
https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eduard Tudenhoefner updated CASSANDRA-11615:
--------------------------------------------
Attachment: 11615-3.0.patch
Attached a patch that fixed the issue for me. Now I could use stress without a
problem on a cluster with *100 nodes*, without seeing any blocked threads.
Looks like the problem was that the prepares were done on *all* nodes within a
synchronized block, which would basically take quite long.
I was talking to Andy Tolbert from the Java Drivers team and he suggested to
not prepare *all statements on all nodes*.
> cassandra-stress blocks when connecting to a big cluster
> --------------------------------------------------------
>
> Key: CASSANDRA-11615
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11615
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Eduard Tudenhoefner
> Assignee: Eduard Tudenhoefner
> Fix For: 3.0.x
>
> Attachments: 11615-3.0.patch
>
>
> I had a *100* node cluster and was running
> {code}
> cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20'
> 'limit=1000/s'
> {code}
> Based on the thread dump it looks like it's been blocked at
> https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96
> {code}
> "Thread-20" #245 prio=5 os_prio=0 tid=0x00007f3781822000 nid=0x46c4 waiting
> for monitor entry [0x00007f36cc788000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
> - waiting to lock <0x00000005c003d920> (a
> java.util.concurrent.ConcurrentHashMap)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> "Thread-19" #244 prio=5 os_prio=0 tid=0x00007f3781820000 nid=0x46c3 waiting
> for monitor entry [0x00007f36cc889000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96)
> - waiting to lock <0x00000005c003d920> (a
> java.util.concurrent.ConcurrentHashMap)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> {code}
> I was trying the same with with a smaller cluster (50 nodes) and it was
> working fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
