Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine micha...@citypath.com wrote: Hi, After configuring the encryption on Cassandra.yaml I get this error when upgrading from 1.0.0 to 1.0.2 Attached the log file with the errors. https://issues.apache.org/jira/browse/CASSANDRA-3466 -Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Does this means that I have to wait to 1.0.3? -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 3:51 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 1:21 AM, Michael Vaknine micha...@citypath.com wrote: Hi, After configuring the encryption on Cassandra.yaml I get this error when upgrading from 1.0.0 to 1.0.2 Attached the log file with the errors. https://issues.apache.org/jira/browse/CASSANDRA-3466 -Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine micha...@citypath.com wrote: Does this means that I have to wait to 1.0.3? In the meantime you can just delete the hints and rely on read repair or antientropy repair if you're concerned about the consistency of your replicas. -Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.compareAndSet(AtomicReferenceFieldUpdater.java:227) at java.util.concurrent.ConcurrentSkipListMap.casHead(ConcurrentSkipListMap.jav a:359) at java.util.concurrent.ConcurrentSkipListMap.insertIndex(ConcurrentSkipListMap .java:968) at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java: 898) at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap .java:1893) at org.apache.cassandra.db.ThreadSafeSortedColumns.addColumn(ThreadSafeSortedCo lumns.java:88) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont ainer.java:122) at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCont ainer.java:117) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColum ns(SSTableNamesIterator.java:183) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableName sIterator.java:130) at org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNa mesIterator.java:72) at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(Nam esQueryFilter.java:60) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFil ter.java:78) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:113) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) 187179,2-995% Should I have deleted all the hints on all the clustr? Is 1.0.0 stable enough? I am checking this and don't want to upgrade again in a week or so. Thanks Michael -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 4:01 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 7:53 AM, Michael Vaknine micha...@citypath.com wrote: Does this means that I have to wait to 1.0.3? In the meantime you can just delete the hints and rely on read repair or antientropy repair if you're concerned about the consistency of your replicas. -Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine micha...@citypath.com wrote: Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) This is different an unrelated to hints. What jvm are you using? It looks like it just simply ran out of stack space, which is odd, but you can control that with the -Xss option if needed. -Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
I am running java version: java version 1.6.0_20 OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) I have tried to run with -Xss96k or -Xss256k or -Xss320 but it still give me the error TheNode has 8GB memory and %GB allocated to JAVA ERROR [MutationStage:104] 2011-11-15 08:55:59,408 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:104,5,main] java.lang.StackOverflowError at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:209) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:74) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) at org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:159) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) at org.apache.cassandra.db.RowMutation.applyUnsafe(RowMutation.java:260) at org.apache.cassandra.db.CollationController.collectTimeOrderedData(Collation Controller.java:159) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCont roller.java:62) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilySto re.java:1278) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1164) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore. java:1126) at org.apache.cassandra.db.Table.readCurrentIndexedColumns(Table.java:507) at org.apache.cassandra.db.Table.apply(Table.java:444) I ma getting WARN as well after changing JAVA WARN [MutationStage:44] 2011-11-15 08:51:57,479 Memtable.java (line 142) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Monday, November 14, 2011 7:53 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Mon, Nov 14, 2011 at 8:06 AM, Michael Vaknine micha...@citypath.com wrote: Well, I tried to delete the hints on the failed cluster but I could not start it I got other errors such as ERROR [MutationStage:34] 2011-11-14 15:37:43,813 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MutationStage:34,5,main] java.lang.StackOverflowError at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceField UpdaterImpl.updateCheck(AtomicReferenceFieldUpdater.java:216) This is different an unrelated to hints. What jvm are you using? It looks like it just simply ran out of stack space, which is odd, but you can control that with the -Xss option if needed. -Brandon
Upgrade Cassandra Cluster to 1.0.2
Hi, I would appreciate any help. I have a cluster of 4 servers with replication factor 3 version 1.0.0 The cluster was upgraded from 0.7.8. I am trying to upgrade to 1.0.2 and when I try to start the first upgraded server I get the following error ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.5.6.102,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou ndTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn ectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection .java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav a:93) ERROR [WRITE-/10.4.251.82] 2011-11-13 10:20:37,448 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.4.251.82,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou ndTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn ectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection .java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav a:93) ERROR [WRITE-/10.64.146.54] 2011-11-13 10:20:37,449 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.64.146.54,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou ndTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn ectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection .java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav a:93) when I run nodetool version I get ReleaseVersion: 1.0.2 When I run nodetool ring I get Address DC RackStatus State LoadOwns Token 127605887595351923798765477786913079296 10.5.6.102 datacenter1 rack1 Down Normal ? 25.00% 0 10.4.251.82 datacenter1 rack1 Down Normal ? 25.00% 42535295865117307932921825928971026432 10.64.146.54datacenter1 rack1 Down Normal ? 25.00% 85070591730234615865843651857942052864 10.104.121.246 datacenter1 rack1 Up Normal 9.55 GB 25.00% 127605887595351923798765477786913079296 The only server up is the one I have upgraded. When I run nodetool ring from a different server on the cluster I get this 10.5.6.102 datacenter1 rack1 Up Normal 22.56 GB 25.00% 0 10.4.251.82 datacenter1 rack1 Up Normal 18.85 GB 25.00% 42535295865117307932921825928971026432 10.64.146.54datacenter1 rack1 Up Normal 16.24 GB 25.00% 85070591730234615865843651857942052864 10.104.121.246 datacenter1 rack1 Down Normal 10.21 GB 25.00% 127605887595351923798765477786913079296 All up except for the upgraded server. Thanks Michael vaknine
Re: Upgrade Cassandra Cluster to 1.0.2
On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine micha...@citypath.com wrote: I am trying to upgrade to 1.0.2 and when I try to start the first upgraded server I get the following error ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.5.6.102,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(OutboundTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:93) You are probably missing the encryption options in your yaml. I noticed this problem as part of https://issues.apache.org/jira/browse/CASSANDRA-3045. -Brandon
RE: Upgrade Cassandra Cluster to 1.0.2
You are right this solved the problem. I do not understand why version 1.0.0 was not affected since I used the same configuration yaml file. Thank you. Michael Vaknine -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Sunday, November 13, 2011 4:48 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine micha...@citypath.com wrote: I am trying to upgrade to 1.0.2 and when I try to start the first upgraded server I get the following error ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.5.6.102,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou ndTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn ectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection .java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav a:93) You are probably missing the encryption options in your yaml. I noticed this problem as part of https://issues.apache.org/jira/browse/CASSANDRA-3045. -Brandon
Re: Upgrade Cassandra Cluster to 1.0.2
I believe https://issues.apache.org/jira/browse/CASSANDRA-2802 broke it. I've created https://issues.apache.org/jira/browse/CASSANDRA-3489 to address this separately. On Sun, Nov 13, 2011 at 9:37 AM, Michael Vaknine micha...@citypath.com wrote: You are right this solved the problem. I do not understand why version 1.0.0 was not affected since I used the same configuration yaml file. Thank you. Michael Vaknine -Original Message- From: Brandon Williams [mailto:dri...@gmail.com] Sent: Sunday, November 13, 2011 4:48 PM To: user@cassandra.apache.org Cc: cassandra-u...@incubator.apache.org Subject: Re: Upgrade Cassandra Cluster to 1.0.2 On Sun, Nov 13, 2011 at 4:35 AM, Michael Vaknine micha...@citypath.com wrote: I am trying to upgrade to 1.0.2 and when I try to start the first upgraded server I get the following error ERROR [WRITE-/10.5.6.102] 2011-11-13 10:20:37,447 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[WRITE-/10.5.6.102,5,main] java.lang.NullPointerException at org.apache.cassandra.net.OutboundTcpConnectionPool.isEncryptedChannel(Outbou ndTcpConnectionPool.java:93) at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConn ectionPool.java:77) at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection .java:209) at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav a:93) You are probably missing the encryption options in your yaml. I noticed this problem as part of https://issues.apache.org/jira/browse/CASSANDRA-3045. -Brandon
