[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982840#comment-14982840 ] Aleksey Yeschenko commented on CASSANDRA-8917: -- This isn't really actionable as is. Could you provide reproductions steps for us to track it down? > Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions > - > > Key: CASSANDRA-8917 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 > Project: Cassandra > Issue Type: Bug > Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data > cassandra 1.1.1, cassandra java driver 2.0.9 >Reporter: Gary Ogden > Fix For: 2.1.x > > Attachments: Screen Shot 2015-05-19 at 10.50.23 AM.png, b_output.log, > jersey_error.log, node1-cassandra.yaml, node1-system.log, > node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log > > > We have java apps running on glassfish that read/write to our 3 node cluster > running on 2.0.9. > we have the CL set to quorum for all reads and writes. > When we started to upgrade the first node and did the sstable upgrade on that > node, we started getting this error on reads and writes: > com.datastax.driver.core.exceptions.UnavailableException: Not enough replica > available for query at consistency QUORUM (2 required but only 1 alive) > How is that possible when we have 3 nodes total, and there was 2 that were up > and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550468#comment-14550468 ] Gary Ogden commented on CASSANDRA-8917: --- I'm actually able to reproduce this same error under a different scenario in a test cluster using devcenter. One of our nodes got into a funny state but is still actually running. Here's the log output: ERROR 13:29:32 JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:56) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1832) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:78) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableScanner.getScanner(SSTableScanner.java:57) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1602) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:67) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1988) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2105) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:132) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:39) ~[apache-cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2 .1.3.jar:2.1.3] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$TraceSessionFutureTask.run(AbstractTracingAwareExecutorService.java:136) [apache- cassandra-2.1.3.jar:2.1.3] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/ cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/ cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[
https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387300#comment-14387300
]
Gary Ogden commented on CASSANDRA-8917:
---
I ran nodetool status on each node and had the exact same result:
{quote}
Datacenter: PRODDC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.6.71.204 122.39 GB 256 31.2% 93772457-9f70-42ea-89f2-a63d40d76703
RAC2
UN 10.6.71.205 123.49 GB 256 36.3% db0e2389-bbe5-43e4-b0e9-c99aff0449b8
RAC2
UN 10.6.71.198 122.45 GB 256 32.6% c0123329-3262-45a6-a6df-c3fe1b1b2978
RAC2
[gary@secasprddb01-2 ~]$ nodetool status company
Datacenter: PRODDC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.6.71.204 122.39 GB 256 100.0%
93772457-9f70-42ea-89f2-a63d40d76703 RAC2
UN 10.6.71.205 123.49 GB 256 100.0%
db0e2389-bbe5-43e4-b0e9-c99aff0449b8 RAC2
UN 10.6.71.198 122.45 GB 256 100.0%
c0123329-3262-45a6-a6df-c3fe1b1b2978 RAC2
{quote}
And when I run the select * from system.peers against each node, it only ever
shows the other 2 nodes. There's no extra old nodes in the list.
Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
-
Key: CASSANDRA-8917
URL: https://issues.apache.org/jira/browse/CASSANDRA-8917
Project: Cassandra
Issue Type: Bug
Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data
cassandra 1.1.1, cassandra java driver 2.0.9
Reporter: Gary Ogden
Fix For: 2.1.4
Attachments: b_output.log, jersey_error.log, node1-cassandra.yaml,
node1-system.log, node2-cassandra.yaml, node2-system.log,
node3-cassandra.yaml, node3-system.log
We have java apps running on glassfish that read/write to our 3 node cluster
running on 2.0.9.
we have the CL set to quorum for all reads and writes.
When we started to upgrade the first node and did the sstable upgrade on that
node, we started getting this error on reads and writes:
com.datastax.driver.core.exceptions.UnavailableException: Not enough replica
available for query at consistency QUORUM (2 required but only 1 alive)
How is that possible when we have 3 nodes total, and there was 2 that were up
and it's saying we can't get the required CL?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14385519#comment-14385519 ] Robert Stupp commented on CASSANDRA-8917: - Can you run `nodetool status` and `select * from system.peers;` against all three nodes (maybe there's an old, removed node still present) ? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Fix For: 2.1.4 Attachments: b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381768#comment-14381768 ] Gary Ogden commented on CASSANDRA-8917: --- We haven't attempted the upgrade again since we ran into this issue. Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Fix For: 2.1.4 Attachments: b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380848#comment-14380848 ] Philip Thompson commented on CASSANDRA-8917: I don't believe the number of seed nodes is related to the issue. Have you run into the problem again? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Fix For: 2.1.4 Attachments: b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369321#comment-14369321 ] Gary Ogden commented on CASSANDRA-8917: --- We only have one node designated as the seed node. Should we increase that to two? Would that be a potential cause of this issue? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-cassandra.yaml, node1-system.log, node2-cassandra.yaml, node2-system.log, node3-cassandra.yaml, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352845#comment-14352845 ] Gary Ogden commented on CASSANDRA-8917: --- It's set to: max_hint_window_in_ms: 1080 # 3 hours Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[
https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352059#comment-14352059
]
Robert Stupp commented on CASSANDRA-8917:
-
What I've seen from your logs:
* node3 went down at approx 19:01
* node3 was started at approx 19:11
* node3 went down at approx 19:14
* node3 was started at approx 19:41 with this neat message: {{Node
localhost/127.0.0.1 state jump to normal}} - looks like a configuration mistake
(I assume you don't want 127.0.0.1).
Another thing I've noticed is a lot of {{HintedHandoffMetrics.java (line 79)
/10.6.71.198 has 119756 dropped hints, because node is down past configured
hint window.}} messages appearing every some minutes for example on node1. Did
you reconfigure the {{max_hint_window_in_ms}} parameter in {{cassandra.yaml}} -
maybe too low?
Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
-
Key: CASSANDRA-8917
URL: https://issues.apache.org/jira/browse/CASSANDRA-8917
Project: Cassandra
Issue Type: Bug
Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data
cassandra 1.1.1, cassandra java driver 2.0.9
Reporter: Gary Ogden
Attachments: b_output.log, jersey_error.log, node1-system.log,
node2-system.log, node3-system.log
We have java apps running on glassfish that read/write to our 3 node cluster
running on 2.0.9.
we have the CL set to quorum for all reads and writes.
When we started to upgrade the first node and did the sstable upgrade on that
node, we started getting this error on reads and writes:
com.datastax.driver.core.exceptions.UnavailableException: Not enough replica
available for query at consistency QUORUM (2 required but only 1 alive)
How is that possible when we have 3 nodes total, and there was 2 that were up
and it's saying we can't get the required CL?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351253#comment-14351253 ] Gary Ogden commented on CASSANDRA-8917: --- That's when we decided to remove the node and revert back to 2.0.9. That's 3 hours after we started seeing the problems. If you look at the b_output.log you'll see the errors we well before our decision to remove the node, install 2.0.9 and then add the node to the cluster. Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[
https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351181#comment-14351181
]
Tyler Hobbs commented on CASSANDRA-8917:
The logs show that a removenode operation was run on node1 for node3:
{noformat}
./node1-system.log:13000: INFO [RMI TCP Connection(507592)-10.6.71.204]
2015-03-04 23:46:25,548 Gossiper.java (line 438) Removing host:
70c2f6c8-bd2a-4229-bf0f-9dd91c7ce823
./node1-system.log-13001- INFO [RMI TCP Connection(507592)-10.6.71.204]
2015-03-04 23:46:25,549 Gossiper.java (line 439) Sleeping for 3ms to ensure
/10.6.71.198 does not change
{noformat}
There are later logs that show the removal completing. That's almost surely
why you're getting that exception.
Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
-
Key: CASSANDRA-8917
URL: https://issues.apache.org/jira/browse/CASSANDRA-8917
Project: Cassandra
Issue Type: Bug
Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data
cassandra 1.1.1, cassandra java driver 2.0.9
Reporter: Gary Ogden
Attachments: b_output.log, jersey_error.log, node1-system.log,
node2-system.log, node3-system.log
We have java apps running on glassfish that read/write to our 3 node cluster
running on 2.0.9.
we have the CL set to quorum for all reads and writes.
When we started to upgrade the first node and did the sstable upgrade on that
node, we started getting this error on reads and writes:
com.datastax.driver.core.exceptions.UnavailableException: Not enough replica
available for query at consistency QUORUM (2 required but only 1 alive)
How is that possible when we have 3 nodes total, and there was 2 that were up
and it's saying we can't get the required CL?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351077#comment-14351077 ] Robert Stupp commented on CASSANDRA-8917: - Your node1 should have been working fine - but it might have been that you've lost one replica (if you're running at RF=2). One question: did you perform DDL statements during the upgrade? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351112#comment-14351112 ] Gary Ogden commented on CASSANDRA-8917: --- RF was = 3. We did not do and DDL statements during the upgrade. Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350416#comment-14350416 ] Gary Ogden commented on CASSANDRA-8917: --- We had one node defined as the seed node, which was node 1. When we turned off node 3 to update the packages, we didn't get the quorum issue. It was only when we turned on node3 with 2.1.3 and started the sstable upgrade did we start seeing the quorum issue. Is it possible that node1 was unavailable since it was streaming to node 3? And therefore only node2 was the only node available? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348876#comment-14348876 ] Gary Ogden commented on CASSANDRA-8917: --- I don't have the exact CQL for the queries since we're using spring data cassandra and it's generating them. I will have to turn on logging and capture those queries. Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden Attachments: b_output.log, jersey_error.log, node1-system.log, node2-system.log, node3-system.log We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348724#comment-14348724 ] Gary Ogden commented on CASSANDRA-8917: --- We never finished the upgrade, so I don't know. We aborted it. So I only know it happens during. Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348719#comment-14348719 ] Robert Stupp commented on CASSANDRA-8917: - Does that happen during the rolling upgrade or even after the update's been finished? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8917) Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348728#comment-14348728 ] Robert Stupp commented on CASSANDRA-8917: - Hm - do you have any more information like log files and CQL statements that you can provide? Upgrading from 2.0.9 to 2.1.3 with 3 nodes, CL = quorum causes exceptions - Key: CASSANDRA-8917 URL: https://issues.apache.org/jira/browse/CASSANDRA-8917 Project: Cassandra Issue Type: Bug Environment: C* 2.0.9, Centos 6.5, Java 1.7.0_72, spring data cassandra 1.1.1, cassandra java driver 2.0.9 Reporter: Gary Ogden We have java apps running on glassfish that read/write to our 3 node cluster running on 2.0.9. we have the CL set to quorum for all reads and writes. When we started to upgrade the first node and did the sstable upgrade on that node, we started getting this error on reads and writes: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive) How is that possible when we have 3 nodes total, and there was 2 that were up and it's saying we can't get the required CL? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
