Re: Cassandra node throws NPE on startup
Late reply, but I just got the same error restarting after upgrading from 0.7.2 to 0.7.5. I did a drain using nodetool on each node before I killed them and did the upgrade. Should all commitlogs have been cleaned up after a drain? I would think so, but they were not. Maybe there is a bug around not cleaning up certain stale commitlogs, which, when replayed causes the error. On Fri, Apr 29, 2011 at 2:02 PM, aaron morton aa...@thelastpickle.comwrote: I've not heard of it happening in a production system. The commit log and the schema have diverged, that does not normally happen. Aaron On 29 Apr 2011, at 20:54, Subscriber wrote: But isn't their a better way than to delete the commitlogs? What do I do when this happens in a productive system? Ciao Udo Am 29.04.2011 um 02:07 schrieb aaron morton: Thought you may have re-created the schema. Kill the process like that should be ok, let us know if you get the error again. Aaron On 29 Apr 2011, at 02:56, Subscriber wrote: Hi Aaron, what exactly do you mean? I restarted the cluster by calling bin/cassandra -p pid.file on all three nodes. The first node is the (only) seed. Udo Am 27.04.2011 um 23:28 schrieb Aaron Morton: What approach did you take to restarting the cluster? It looks like the keyspace name was changed and the log replay tried to write to the old one. Aaron On 28/04/2011, at 12:03 AM, Subscriber subscri...@zfabrik.de wrote: Hi again, some more remarks. I renamed the commitlog directory on the third node so that cassandra cannot see it on startup. Now the node starts fine. The problem seems to have something to do with the commitlogs... Best Regards Udo Am 27.04.2011 um 13:22 schrieb Subscriber: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo
Re: Cassandra node throws NPE on startup
It looks like this will happen if it tries to replay a mutation for a dropped keyspace (dropped CFs are handled correctly). Created https://issues.apache.org/jira/browse/CASSANDRA-2631 to fix. On Wed, Apr 27, 2011 at 6:22 AM, Subscriber subscri...@zfabrik.de wrote: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Cassandra node throws NPE on startup
But isn't their a better way than to delete the commitlogs? What do I do when this happens in a productive system? Ciao Udo Am 29.04.2011 um 02:07 schrieb aaron morton: Thought you may have re-created the schema. Kill the process like that should be ok, let us know if you get the error again. Aaron On 29 Apr 2011, at 02:56, Subscriber wrote: Hi Aaron, what exactly do you mean? I restarted the cluster by calling bin/cassandra -p pid.file on all three nodes. The first node is the (only) seed. Udo Am 27.04.2011 um 23:28 schrieb Aaron Morton: What approach did you take to restarting the cluster? It looks like the keyspace name was changed and the log replay tried to write to the old one. Aaron On 28/04/2011, at 12:03 AM, Subscriber subscri...@zfabrik.de wrote: Hi again, some more remarks. I renamed the commitlog directory on the third node so that cassandra cannot see it on startup. Now the node starts fine. The problem seems to have something to do with the commitlogs... Best Regards Udo Am 27.04.2011 um 13:22 schrieb Subscriber: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo
Re: Cassandra node throws NPE on startup
I've not heard of it happening in a production system. The commit log and the schema have diverged, that does not normally happen. Aaron On 29 Apr 2011, at 20:54, Subscriber wrote: But isn't their a better way than to delete the commitlogs? What do I do when this happens in a productive system? Ciao Udo Am 29.04.2011 um 02:07 schrieb aaron morton: Thought you may have re-created the schema. Kill the process like that should be ok, let us know if you get the error again. Aaron On 29 Apr 2011, at 02:56, Subscriber wrote: Hi Aaron, what exactly do you mean? I restarted the cluster by calling bin/cassandra -p pid.file on all three nodes. The first node is the (only) seed. Udo Am 27.04.2011 um 23:28 schrieb Aaron Morton: What approach did you take to restarting the cluster? It looks like the keyspace name was changed and the log replay tried to write to the old one. Aaron On 28/04/2011, at 12:03 AM, Subscriber subscri...@zfabrik.de wrote: Hi again, some more remarks. I renamed the commitlog directory on the third node so that cassandra cannot see it on startup. Now the node starts fine. The problem seems to have something to do with the commitlogs... Best Regards Udo Am 27.04.2011 um 13:22 schrieb Subscriber: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo
Re: Cassandra node throws NPE on startup
Hi Aaron, what exactly do you mean? I restarted the cluster by calling bin/cassandra -p pid.file on all three nodes. The first node is the (only) seed. Udo Am 27.04.2011 um 23:28 schrieb Aaron Morton: What approach did you take to restarting the cluster? It looks like the keyspace name was changed and the log replay tried to write to the old one. Aaron On 28/04/2011, at 12:03 AM, Subscriber subscri...@zfabrik.de wrote: Hi again, some more remarks. I renamed the commitlog directory on the third node so that cassandra cannot see it on startup. Now the node starts fine. The problem seems to have something to do with the commitlogs... Best Regards Udo Am 27.04.2011 um 13:22 schrieb Subscriber: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo
Re: Cassandra node throws NPE on startup
What approach did you take to restarting the cluster? It looks like the keyspace name was changed and the log replay tried to write to the old one. Aaron On 28/04/2011, at 12:03 AM, Subscriber subscri...@zfabrik.de wrote: Hi again, some more remarks. I renamed the commitlog directory on the third node so that cassandra cannot see it on startup. Now the node starts fine. The problem seems to have something to do with the commitlogs... Best Regards Udo Am 27.04.2011 um 13:22 schrieb Subscriber: Hi, I'm using Cassandra 0.7.4 on a three node cluster. The cluster was setup yesterday as a fresh installation (no upgrade). The cluster is installed beside a hadoop cluster (I want to discover how cassandra works together with hadoop's map/reduce feature). After loading some test data into the cassandra cluster (some text-files for a first wordcount m/r example ;-) I killed all nodes and restarted the cluster. Two nodes are starting fine but the third node doesn't start and throws a NullPointerException: INFO 12:44:42,557 Finished reading /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log ERROR 12:44:42,558 Exception encountered during startup. java.lang.NullPointerException at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) at org.apache.cassandra.db.Table.init(Table.java:221) at org.apache.cassandra.db.Table.open(Table.java:110) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:173) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) Exception encountered during startup. This looks pretty much the same as issue https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in resolution cannot reproduce. What can I do to make the node starting again? Thanks in advance and best regards Udo
