[
https://issues.apache.org/jira/browse/CASSANDRA-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233551#comment-15233551
]
Mattias W edited comment on CASSANDRA-11528 at 4/9/16 1:41 PM:
---------------------------------------------------------------
It is a out-of-memory error. Cassandra 3.4 on ubuntu 14.04 behaves the same,
and there, the last message in the log is below.
So now I know, select statements can use a lot of heap. The Ubuntu machine only
has 1.5 GB RAM. (The windows machine above had 8 GB)
{noformat}
INFO [SharedPool-Worker-3] 2016-04-09 15:32:34,915 ApproximateTime.java:44 -
Scheduling approximate time-check task with a precision of 10 milliseconds
INFO [Service Thread] 2016-04-09 15:34:50,366 GCInspector.java:284 -
ConcurrentMarkSweep GC in 443ms. CMS Old Gen: 547965232 -> 268786192; Par Eden
Space: 126017056 -> 0; Par Survivor Space: 3420928 -> 0
INFO [Service Thread] 2016-04-09 15:34:50,379 StatusLogger.java:52 - Pool Name
Active Pending Completed Blocked All Time Blocked
ERROR [SharedPool-Worker-2] 2016-04-09 15:34:50,409
JVMStabilityInspector.java:139 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_77]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
at
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:208)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:185)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:110)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:98)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:134)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2467)
~[apache-cassandra-3.4.jar:3.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_77]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.4.jar:3.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.4.jar:3.4]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
INFO [Service Thread] 2016-04-09 15:34:50,412 StatusLogger.java:56 -
MutationStage 0 0 157 0
0
INFO [Service Thread] 2016-04-09 15:34:50,414 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,415 StatusLogger.java:56 - ReadStage
1 0 22 0 0
INFO [Service Thread] 2016-04-09 15:34:50,416 StatusLogger.java:56 -
RequestResponseStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,418 StatusLogger.java:56 -
ReadRepairStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,423 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,424 StatusLogger.java:56 - MiscStage
0 0 0 0 0
INFO [Service Thread] 2016-04-09 15:34:50,428 StatusLogger.java:56 -
CompactionExecutor 0 0 72 0
0
{noformat}
was (Author: mattiasw2):
It is a out-of-memory error. Cassandra 3.4 on ubuntu 14.04 behaves the same,
and there, the last message in the log is
{noformat}
INFO [SharedPool-Worker-3] 2016-04-09 15:32:34,915 ApproximateTime.java:44 -
Scheduling approximate time-check task with a precision of 10 milliseconds
INFO [Service Thread] 2016-04-09 15:34:50,366 GCInspector.java:284 -
ConcurrentMarkSweep GC in 443ms. CMS Old Gen: 547965232 -> 268786192; Par Eden
Space: 126017056 -> 0; Par Survivor Space: 3420928 -> 0
INFO [Service Thread] 2016-04-09 15:34:50,379 StatusLogger.java:52 - Pool Name
Active Pending Completed Blocked All Time Blocked
ERROR [SharedPool-Worker-2] 2016-04-09 15:34:50,409
JVMStabilityInspector.java:139 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_77]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
at
org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:208)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:185)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:110)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:98)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:134)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2467)
~[apache-cassandra-3.4.jar:3.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_77]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.4.jar:3.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.4.jar:3.4]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
INFO [Service Thread] 2016-04-09 15:34:50,412 StatusLogger.java:56 -
MutationStage 0 0 157 0
0
INFO [Service Thread] 2016-04-09 15:34:50,414 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,415 StatusLogger.java:56 - ReadStage
1 0 22 0 0
INFO [Service Thread] 2016-04-09 15:34:50,416 StatusLogger.java:56 -
RequestResponseStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,418 StatusLogger.java:56 -
ReadRepairStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,423 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-04-09 15:34:50,424 StatusLogger.java:56 - MiscStage
0 0 0 0 0
INFO [Service Thread] 2016-04-09 15:34:50,428 StatusLogger.java:56 -
CompactionExecutor 0 0 72 0
0
{noformat}
> Server Crash when select returns more than a few hundred rows
> -------------------------------------------------------------
>
> Key: CASSANDRA-11528
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11528
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: windows 7, 8 GB machine
> Reporter: Mattias W
> Fix For: 3.x
>
> Attachments: datastax_ddc_server-stdout.2016-04-07.log
>
>
> While implementing a dump procedure, which did "select * from" from one table
> at a row, I instantly kill the server. A simple
> {noformat}select count(*) from {noformat}
> also kills it. For a while, I thought the size of blobs were the cause
> I also try to only have a unique id as partition key, I was afraid a single
> partition got too big or so, but that didn't change anything
> It happens every time, both from Java/Clojure and from DevCenter.
> I looked at the logs at C:\Program Files\DataStax-DDC\logs, but the crash is
> so quick, so nothing is recorded there.
> There is a Java-out-of-memory in the logs, but that isn't from the time of
> the crash.
> It only happens for one table, it only has 15000 entries, but there are blobs
> and byte[] stored there, size between 100kb - 4Mb. Total size for that table
> is about 6.5 GB on disk.
> I made a workaround by doing many small selects instead, each only fetching
> 100 rows.
> Is there a setting a can set to make the system log more eagerly, in order to
> at least get a stacktrace or similar, that might help you.
> It is the prun_srv that dies. Restarting the NT service makes Cassandra run
> again
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
