[
https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005303#comment-14005303
]
Jason Punyon commented on CASSANDRA-7144:
-----------------------------------------
[~thobbs] We racked our brains a bit, we can't really conceive of a direction
to go in to replicate it again (remember it's still live on our cluster, we can
run any diagnostics you want), aside from rebuilding and then randomly bringing
power down (what we did the first time).
> CassandraDaemon RowMutation exception
> -------------------------------------
>
> Key: CASSANDRA-7144
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7144
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB /
> 2 Cores in DigitalOcean.
> Reporter: Maxime Lamothe-Brassard
> Assignee: Tyler Hobbs
>
> First time reporting a bug here, apologies if I'm not posting it in the right
> space.
> At what seems like random interval, on random nodes in random situations I
> will get the following exception. After this the hinted handoff start timing
> out and the node stops participating in the cluster.
> I started seeing these after switching to the Cassandra Python-Driver from
> the Python-CQL driver.
> {noformat}
> ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java
> (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java
> (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
> 198) Exception in thread Thread[MutationStage:118,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> at
> org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> at
> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> at
> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> at
> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line
> 198) Exception in thread Thread[MutationStage:117,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> at
> org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> at
> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> at
> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> at
> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> The service must be restarted for the node to come back online. Let me know
> any additional configuration details needed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
