Hey C*, During startup of a Cassandra 2.2.7 node, some part of the process fails. Here's a snippet of logs with the stack trace
TRACE [main] 2016-07-28 19:06:25,381 SliceQueryFilter.java:269 - collecting 6 of 2147483647: custom_table:ep_str:component_index:false:4@1441163636178000TRACE [main] 2016-07-28 19:06:25,381 SliceQueryFilter.java:269 - collecting 7 of 2147483647: custom_table:ep_str:index_options:false:4@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 8 of 2147483647: custom_table:ep_str:type:false:14@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 9 of 2147483647: custom_table:ep_str:validator:false:86@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 10 of 2147483647: custom_table:mbr_guid::false:0@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 11 of 2147483647: custom_table:mbr_guid:index_options:false:4@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 12 of 2147483647: custom_table:mbr_guid:type:false:13@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 SliceQueryFilter.java:269 - collecting 13 of 2147483647: custom_table:mbr_guid:validator:false:40@1441163636178000TRACE [main] 2016-07-28 19:06:25,382 FileCacheService.java:151 - Estimated memory usage is 123298 compared to actual usage 123298TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 0 of 2147483647: a_table_of_ours:ep_int::false:0@1441163637780000TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 1 of 2147483647: a_table_of_ours:ep_int:component_index:false:4@1441163637780000TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 2 of 2147483647: a_table_of_ours:ep_int:index_options:false:4@1441163637780000TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 3 of 2147483647: a_table_of_ours:ep_int:type:false:14@1441163637780000TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 4 of 2147483647: a_table_of_ours:ep_int:validator:false:87@1441163637780000TRACE [main] 2016-07-28 19:06:25,384 SliceQueryFilter.java:269 - collecting 5 of 2147483647: a_table_of_ours:ep_str::false:0@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 6 of 2147483647: a_table_of_ours:ep_str:component_index:false:4@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 7 of 2147483647: a_table_of_ours:ep_str:index_options:false:4@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 8 of 2147483647: a_table_of_ours:ep_str:type:false:7@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 9 of 2147483647: a_table_of_ours:ep_str:validator:false:40@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 10 of 2147483647: a_table_of_ours:mbr_guid::false:0@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 11 of 2147483647: a_table_of_ours:mbr_guid:index_options:false:4@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 12 of 2147483647: a_table_of_ours:mbr_guid:type:false:13@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 SliceQueryFilter.java:269 - collecting 13 of 2147483647: a_table_of_ours:mbr_guid:validator:false:40@1441163637780000TRACE [main] 2016-07-28 19:06:25,385 FileCacheService.java:151 - Estimated memory usage is 123298 compared to actual usage 123298TRACE [main] 2016-07-28 19:06:25,388 FileCacheService.java:151 - Estimated memory usage is 104405 compared to actual usage 104405TRACE [main] 2016-07-28 19:06:25,388 SSTableReader.java:1535 - Adding cache entry for KeyCacheKey(/home/var/cassandra/data/system/schema_columnfamilies/lb-4596-big, 746573745f736f746f) -> org.apache.cassandra.db.RowIndexEntry@58f2fce7TRACE [main] 2016-07-28 19:06:25,388 FileCacheService.java:151 - Estimated memory usage is 104405 compared to actual usage 104405TRACE [main] 2016-07-28 19:06:25,389 FileCacheService.java:151 - Estimated memory usage is 227703 compared to actual usage 227703TRACE [main] 2016-07-28 19:06:25,389 SSTableReader.java:1535 - Adding cache entry for KeyCacheKey(/home/var/cassandra/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/lb-829-big, 746573745f736f746f) -> org.apache.cassandra.db.RowIndexEntry@443e782cTRACE [main] 2016-07-28 19:06:25,389 FileCacheService.java:151 - Estimated memory usage is 227703 compared to actual usage 227703ERROR [main] 2016-07-28 19:06:25,395 CassandraDaemon.java:644 - Exception encountered during startupjava.lang.IllegalStateException: One row required, 0 foundat org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:77) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.schema.LegacySchemaTables.createKeyspaceFromSchemaPartition(LegacySchemaTables.java:758) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.schema.LegacySchemaTables.createKeyspaceFromSchemaPartitions(LegacySchemaTables.java:737) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.schema.LegacySchemaTables.readSchemaFromSystemTables(LegacySchemaTables.java:219) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:117) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:107) ~[apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:215) [apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:522) [apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT]at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:631) [apache-cassandra-2.2.7.jar:2.2.7-SNAPSHOT] Following some of this execution in the git repo, it seems we try to get the first row in the system.schema_keyspaces table but there is no such row. Am I reading this correctly? What could have caused this and what can I do to fix it? Thanks,Soto
