Re: Cassandra 2.1.18 - NPE during startup
Hi Thomas, The process you suggested to get around the issue should work with the system.keyspaces table. Make sure to backup the original *system.keyspaces* table files on the node that fails to start. Then, copy only the *system.keyspaces *table files from a working node into the *system/schema_keyspaces-...* folder of the node that fails to start. This method will only work for certain system tables as some of the data stored in the system tables will differ between nodes. Regards, Anthony On Thu, 28 Mar 2019 at 05:54, Steinmaurer, Thomas < thomas.steinmau...@dynatrace.com> wrote: > Hello, > > > > any ideas regarding below, cause it happened again on a different node. > > > > Thanks > > Thomas > > > > *From:* Steinmaurer, Thomas > *Sent:* Dienstag, 05. Februar 2019 23:03 > *To:* user@cassandra.apache.org > *Subject:* Cassandra 2.1.18 - NPE during startup > > > > Hello, > > > > at a particular customer location, we are seeing the following NPE during > startup with Cassandra 2.1.18. > > > > INFO [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475 > - Opening > /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130 > (256 bytes) > > ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception > encountered during startup > > org.apache.cassandra.io.FSReadError: java.lang.NullPointerException > > at > org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) > ~[apache-cassandra-2.1.18.jar:2.1.18] > > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) > [apache-cassandra-2.1.18.jar:2.1.18] > > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) > [apache-cassandra-2.1.18.jar:2.1.18] > > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) > [apache-cassandra-2.1.18.jar:2.1.18] > > Caused by: java.lang.NullPointerException: null > > at > org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664) > ~[apache-cassandra-2.1.18.jar:2.1.18] > > ... 3 common frames omitted > > > > I found https://issues.apache.org/jira/browse/CASSANDRA-10501 > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCASSANDRA-10501=02%7C01%7Cthomas.steinmaurer%40dynatrace.com%7C796c92ba8c4d4c94937208d68bb69f1d%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C636850013806426682=5Fx4An63c5R%2Faa2%2BfFXFCg7%2FDejG1V57st7UNtq9hrA%3D=0>, > but this should be fixed in 2.1.18. > > > > Is the above log stating that it is caused by a system keyspace related > SSTable? > > > > This is a 3 node setup with 2 others running fine. If system table related > and as LocalStrategy is used as replication strategy (to my knowledge), > perhaps simply copying over data for the schema_keyspaces table from > another node might fix it? > > > > Any help appreciated. > > > > Thanks. > > Thomas > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 >
RE: Cassandra 2.1.18 - NPE during startup
Hello, any ideas regarding below, cause it happened again on a different node. Thanks Thomas From: Steinmaurer, Thomas Sent: Dienstag, 05. Februar 2019 23:03 To: user@cassandra.apache.org Subject: Cassandra 2.1.18 - NPE during startup Hello, at a particular customer location, we are seeing the following NPE during startup with Cassandra 2.1.18. INFO [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475 - Opening /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130 (256 bytes) ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception encountered during startup org.apache.cassandra.io.FSReadError: java.lang.NullPointerException at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18] Caused by: java.lang.NullPointerException: null at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664) ~[apache-cassandra-2.1.18.jar:2.1.18] ... 3 common frames omitted I found https://issues.apache.org/jira/browse/CASSANDRA-10501<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCASSANDRA-10501=02%7C01%7Cthomas.steinmaurer%40dynatrace.com%7C796c92ba8c4d4c94937208d68bb69f1d%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C636850013806426682=5Fx4An63c5R%2Faa2%2BfFXFCg7%2FDejG1V57st7UNtq9hrA%3D=0>, but this should be fixed in 2.1.18. Is the above log stating that it is caused by a system keyspace related SSTable? This is a 3 node setup with 2 others running fine. If system table related and as LocalStrategy is used as replication strategy (to my knowledge), perhaps simply copying over data for the schema_keyspaces table from another node might fix it? Any help appreciated. Thanks. Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313 The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
Cassandra 2.1.18 - NPE during startup
Hello, at a particular customer location, we are seeing the following NPE during startup with Cassandra 2.1.18. INFO [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475 - Opening /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130 (256 bytes) ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception encountered during startup org.apache.cassandra.io.FSReadError: java.lang.NullPointerException at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18] Caused by: java.lang.NullPointerException: null at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664) ~[apache-cassandra-2.1.18.jar:2.1.18] ... 3 common frames omitted I found https://issues.apache.org/jira/browse/CASSANDRA-10501, but this should be fixed in 2.1.18. Is the above log stating that it is caused by a system keyspace related SSTable? This is a 3 node setup with 2 others running fine. If system table related and as LocalStrategy is used as replication strategy (to my knowledge), perhaps simply copying over data for the schema_keyspaces table from another node might fix it? Any help appreciated. Thanks. Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313
