[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-11615: --- Assignee: Andy Tolbert (was: Eduard Tudenhoefner) > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Andy Tolbert > Fix For: 3.7, 3.0.7 > > Attachments: 11615-3.0-2nd.patch, 11615-3.0.patch > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-11615: --- Resolution: Fixed Fix Version/s: (was: 3.0.x) 3.0.7 3.7 Status: Resolved (was: Patch Available) committed thx > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Andy Tolbert > Fix For: 3.7, 3.0.7 > > Attachments: 11615-3.0-2nd.patch, 11615-3.0.patch > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-11615: Reviewer: T Jake Luciani > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.x > > Attachments: 11615-3.0-2nd.patch, 11615-3.0.patch > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-11615: Attachment: 11615-3.0-2nd.patch [~tjake] attached a second patch, where it can be controlled with an option. I'm also go ahead and talk to the C* Java driver guys about it > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.x > > Attachments: 11615-3.0-2nd.patch, 11615-3.0.patch > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-11615: Attachment: 11615-3.0.patch Attached a patch that fixed the issue for me. Now I could use stress without a problem on a cluster with *100 nodes*, without seeing any blocked threads. Looks like the problem was that the prepares were done on *all* nodes within a synchronized block, which would basically take quite long. I was talking to Andy Tolbert from the Java Drivers team and he suggested to not prepare *all statements on all nodes*. > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.x > > Attachments: 11615-3.0.patch > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-11615: Fix Version/s: 3.0.x Status: Patch Available (was: Open) > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.x > > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11615) cassandra-stress blocks when connecting to a big cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-11615: Assignee: T Jake Luciani > cassandra-stress blocks when connecting to a big cluster > > > Key: CASSANDRA-11615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11615 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: T Jake Luciani > > I had a *100* node cluster and was running > {code} > cassandra-stress read n=100 no-warmup cl=LOCAL_QUORUM -rate 'threads=20' > 'limit=1000/s' > {code} > Based on the thread dump it looks like it's been blocked at > https://github.com/apache/cassandra/blob/cassandra-3.0/tools/stress/src/org/apache/cassandra/stress/util/JavaDriverClient.java#L96 > {code} > "Thread-20" #245 prio=5 os_prio=0 tid=0x7f3781822000 nid=0x46c4 waiting > for monitor entry [0x7f36cc788000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > "Thread-19" #244 prio=5 os_prio=0 tid=0x7f378182 nid=0x46c3 waiting > for monitor entry [0x7f36cc889000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.cassandra.stress.util.JavaDriverClient.prepare(JavaDriverClient.java:96) > - waiting to lock <0x0005c003d920> (a > java.util.concurrent.ConcurrentHashMap) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation$JavaDriverWrapper.createPreparedStatement(CqlOperation.java:314) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:77) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109) > at > org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261) > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327) > {code} > I was trying the same with with a smaller cluster (50 nodes) and it was > working fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)