Ariel Weisberg created CASSANDRA-10695:
------------------------------------------
Summary: Thrift HSHA appears to not work in 2.1+ with high client
thread counts in stress
Key: CASSANDRA-10695
URL: https://issues.apache.org/jira/browse/CASSANDRA-10695
Project: Cassandra
Issue Type: Bug
Reporter: Ariel Weisberg
In 2.2+ my stress runs make almost no progress. In 2.1 they make some progress
but this assertion is still in the log. Things are generally pretty unstable. I
was able to benchmark with the sync server at 500 threads without issue.
Where I ran into trouble was attempting to benchmark with 2000 threads. The
sync server crapped out due to thread counts and I didn't bother trying to
fiddle with it I switched to HSHA which worked at 500, but fails at 2000.
My test configuration is an OS X laptop running 2.2 stress (sometimes trunk,
didn't seem to matter) and a quad core Linux desktop. I can reproduce this the
other way around as well.
Stress command was
{code}
cassandra-stress write n=19000000 -rate threads=2000 -mode thrift -node
192.168.1.3
{code}
I will attach the YAML. Error was
{code}
java.lang.RuntimeException: java.lang.AssertionError: Invoke called in invalid
state: READY_TO_WRITE
at
com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
~[disruptor-3.0.1.jar:na]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:126)
~[disruptor-3.0.1.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.lang.AssertionError: Invoke called in invalid state:
READY_TO_WRITE
at com.thinkaurelius.thrift.Message.invoke(Message.java:306)
~[thrift-server-0.3.7.jar:na]
at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
~[thrift-server-0.3.7.jar:na]
at
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
~[thrift-server-0.3.7.jar:na]
at
com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
~[thrift-server-0.3.7.jar:na]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
~[disruptor-3.0.1.jar:na]
... 3 common frames omitted
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
