[
https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550468#comment-14550468
]
Gary Ogden commented on CASSANDRA-8917:
---------------------------------------
I'm actually able to reproduce this same error under a different scenario in a
test cluster using devcenter.
One of our nodes got into a funny state but is still actually running. Here's
the log output:
ERROR 13:29:32 JVM state determined to be unstable. Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:69)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:77)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:56)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1832)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableScanner.getScanner(SSTableScanner.java:57)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1602)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:67)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1988)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2105)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:132)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_75]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2
.1.3.jar:2.1.3]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$TraceSessionFutureTask.run(AbstractTracingAwareExecutorService.java:136)
[apache-
cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.3.jar:2.1.3]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo
(Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline
org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned
(Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/
cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
(Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo
(Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline
org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned
(Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/
cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
(Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
When this node is up but in this state and I try to execute a query in
devcenter, I get this same error in the dev center popup, but not every time I
run a query.
> Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-8917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8917
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data
> cassandra 1.1.1, cassandra java driver 2.0.9
> Reporter: Gary Ogden
> Fix For: 2.1.x
>
> Attachments: Screen Shot 2015-05-19 at 10.50.23 AM.png, b_output.log,
> jersey_error.log, node1-cassandra.yaml, node1-system.log,
> node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log
>
>
> We have java apps running on glassfish that read/write to our 3 node cluster
> running on 2.0.9.
> we have the CL set to quorum for all reads and writes.
> When we started to upgrade the first node and did the sstable upgrade on that
> node, we started getting this error on reads and writes:
> com.datastax.driver.core.exceptions.UnavailableException: Not enough replica
> available for query at consistency QUORUM (2 required but only 1 alive)
> How is that possible when we have 3 nodes total, and there was 2 that were up
> and it's saying we can't get the required CL?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
