I don't think that's a known issue. Can you open a ticket at https://issues.apache.org/jira/browse/CASSANDRA and attach your schema along with the commitlog files and the mutation that was saved to /tmp?
On Thu, Jan 14, 2016 at 10:56 AM, Jean Tremblay < jean.tremb...@zen-innovations.com> wrote: > Hi, > > I have a small Cassandra Cluster with 5 nodes, having 16MB of RAM. > I use Cassandra 3.1.1. > I use the following setup for the memory: > MAX_HEAP_SIZE="6G" > HEAP_NEWSIZE="496M" > > I have been loading a lot of data in this cluster over the last 24 hours. > The system behaved I think very nicely. It was loading very fast, and > giving excellent read time. There was no error messages until this one: > > > ERROR [SharedPool-Worker-35] 2016-01-14 17:05:23,602 > JVMStabilityInspector.java:139 - JVM state determined to be unstable. > Exiting forcefully due to: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_65] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_65] > at > org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:297) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.BufferCell$Serializer.serialize(BufferCell.java:263) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:183) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:96) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:128) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_65] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > ~[apache-cassandra-3.1.1.jar:3.1.1] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.1.1.jar:3.1.1] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > > 4 nodes out of 5 crashed with this error message. Now when I want to > restart the first node I have the following error; > > ERROR [main] 2016-01-14 17:15:59,617 JVMStabilityInspector.java:81 - > Exiting due to error while processing commit log during initialization. > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Unexpected error deserializing mutation; saved to > /tmp/mutation7465380878750576105dat. This may be caused by replaying a > mutation against a table with the same name but incompatible schema. > Exception follows: org.apache.cassandra.serializers.MarshalException: Not > enough bytes to read a map > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:633) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:556) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:509) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:404) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:151) > [apache-cassandra-3.1.1.jar:3.1.1] > at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) > [apache-cassandra-3.1.1.jar:3.1.1] > at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:549) > [apache-cassandra-3.1.1.jar:3.1.1] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:677) > [apache-cassandra-3.1.1.jar:3.1.1] > > I can no longer start my nodes. > > How can I restart my cluster? > Is this problem known? > Is there a better Cassandra 3 version which would behave better with > respect to this problem? > Would there be a better memory configuration to select for my nodes? > Currently I use MAX_HEAP_SIZE="6G" HEAP_NEWSIZE=“496M” for a 16M RAM node. > > > Thank you very much for your advice. > > Kind regards > > Jean > -- Tyler Hobbs DataStax <http://datastax.com/>
