Hi,
has anyone seen the following exception before?
Context:
* Cassandra 3.9,
* single node (20 Cores / 256 GB RAM)
* doing lots of counter mutations
* Whenever this exception happens, CPU spikes, node becomes unresponsive
for a few minutes. Eventually, the node will "die", i.e. become
completely unresponsive. Restarting the node fixes it... until the next
time :(
* Exceptions happen more often under heavy load
WARN [CounterMutationStage-10] 2017-07-25 18:39:31,867
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on
thread Thread[CounterMutationStage-10,5,main]: {}
java.lang.RuntimeException:
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed
out - received only 0 responses.
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2490)
~[apache-cassandra-3.9.jar:3.9]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.8.0_112]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_112]
Caused by: org.apache.cassandra.exceptions.WriteTimeoutException:
Operation timed out - received only 0 responses.
at
org.apache.cassandra.db.CounterMutation.grabCounterLocks(CounterMutation.java:150)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.CounterMutation.applyCounterMutation(CounterMutation.java:122)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$9.runMayThrow(StorageProxy.java:1473)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2486)
~[apache-cassandra-3.9.jar:3.9]
... 5 common frames omitted
Thanks!
David
--
-----------------------------------
Technical Director / Co-Founder
Sandbox Interactive GmbH
http://albiononline.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
