Re: Cassandra Daemon not coming up

2018-03-07 Thread Rahul Singh 
It’s possible that the schema supporting roles and users is corrupted. Do you 
have a backup of it? Another quick fix would be to potentially reset 
permissions on your data dirs and restart. You can also inspect using the 
offline Cassandra sstable reader to see if it’s unaffected.


--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 5, 2018, 6:26 PM -0500, mahesh rajamani <rajamani.mah...@gmail.com>, 
wrote:
> I did not add any user and disk space was fine.
>
>
>
> > On Tue, Feb 27, 2018, 11:33 Rahul Singh <rahul.xavier.si...@gmail.com> 
> > wrote:
> > > Were there any changes to the system such as permissions, etc. Did you 
> > > add users / change auth scheme?
> > >
> > > On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A <az1...@att.com>, wrote:
> > > > Can you check if you’ve enough disk space available ?
> > > > ~Asad
> > > >
> > > > From: mahesh rajamani [mailto:rajamani.mah...@gmail.com]
> > > > Sent: Tuesday, February 27, 2018 10:11 AM
> > > > To: user@cassandra.apache.org
> > > > Subject: Cassandra Daemon not coming up
> > > >
> > > > I am using Cassandra 3.0.9 version on a 12 node cluster. I have 
> > > > multiple node down after a restart. The cassandra VM is not coming up 
> > > > with an asset error as below. On running in debug mode it is failing 
> > > > while doing operation on " resource_role_permissons_index" in 
> > > > system_auth keyspace. Please let me know how to bring the cassandra 
> > > > running from this state.
> > > >
> > > > Logs from system.log
> > > >
> > > > INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.columns
> > > > INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.triggers
> > > > INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.dropped_columns
> > > > INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.views
> > > > INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.types
> > > > INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.functions
> > > > INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.aggregates
> > > > INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - 
> > > > Initializing system_schema.indexes
> > > > ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - 
> > > > Exception encountered during startup
> > > > java.lang.AssertionError: null
> > > >         at 
> > > > org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) 
> > > > ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) 
> > > > ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) 
> > > > ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
> > > >  ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) 
> > > > ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) 
> > > > ~[apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239)
> > > >  [apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
> > > >  [apache-cassandra-3.0.9.jar:3.0.9]
> > > >         at 
> > > > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696)
> > > >  [apache-cassandra-3.0.9.jar:3.0.9]
> > > >
> > > > --
> > > > Regards,
> > > > Mahesh Rajamani

Re: Cassandra Daemon not coming up

2018-03-05 Thread mahesh rajamani 
I did not add any user and disk space was fine.



On Tue, Feb 27, 2018, 11:33 Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> Were there any changes to the system such as permissions, etc. Did you add
> users / change auth scheme?
>
> On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A <az1...@att.com>, wrote:
>
> Can you check if you’ve enough disk space available ?
>
> ~Asad
>
>
>
> *From:* mahesh rajamani [mailto:rajamani.mah...@gmail.com]
> *Sent:* Tuesday, February 27, 2018 10:11 AM
> *To:* user@cassandra.apache.org
> *Subject:* Cassandra Daemon not coming up
>
>
>
> I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple
> node down after a restart. The cassandra VM is not coming up with an asset
> error as below. On running in debug mode it is failing while doing
> operation on " resource_role_permissons_index" in system_auth keyspace.
> Please let me know how to bring the cassandra running from this state.
>
>
>
> Logs from system.log
>
>
>
> INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 -
> Initializing system_schema.columns
>
>
> INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 -
> Initializing system_schema.triggers
>
>
> INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 -
> Initializing system_schema.dropped_columns
>
>
> INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 -
> Initializing system_schema.views
>
>
> INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 -
> Initializing system_schema.types
>
>
> INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 -
> Initializing system_schema.functions
>
>
> INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 -
> Initializing system_schema.aggregates
>
>
> INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 -
> Initializing system_schema.indexes
>
>
> ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception
> encountered during startup
>
>
> java.lang.AssertionError: null
>
>
>
> at
> org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239)
> [apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
> [apache-cassandra-3.0.9.jar:3.0.9]
>
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696)
> [apache-cassandra-3.0.9.jar:3.0.9]
>
>
>
> --
>
> Regards,
> Mahesh Rajamani
>
>

RE: Cassandra Daemon not coming up

2018-02-27 Thread Rahul Singh 
Were there any changes to the system such as permissions, etc. Did you add 
users / change auth scheme?

On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A <az1...@att.com>, wrote:
> Can you check if you’ve enough disk space available ?
> ~Asad
>
> From: mahesh rajamani [mailto:rajamani.mah...@gmail.com]
> Sent: Tuesday, February 27, 2018 10:11 AM
> To: user@cassandra.apache.org
> Subject: Cassandra Daemon not coming up
>
> I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node 
> down after a restart. The cassandra VM is not coming up with an asset error 
> as below. On running in debug mode it is failing while doing operation on " 
> resource_role_permissons_index" in system_auth keyspace. Please let me know 
> how to bring the cassandra running from this state.
>
> Logs from system.log
>
> INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - 
> Initializing system_schema.columns
> INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - 
> Initializing system_schema.triggers
> INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - 
> Initializing system_schema.dropped_columns
> INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - 
> Initializing system_schema.views
> INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - 
> Initializing system_schema.types
> INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - 
> Initializing system_schema.functions
> INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - 
> Initializing system_schema.aggregates
> INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - 
> Initializing system_schema.indexes
> ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception 
> encountered during startup
> java.lang.AssertionError: null
>         at 
> org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
>  ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) 
> ~[apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) 
> [apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
>  [apache-cassandra-3.0.9.jar:3.0.9]
>         at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) 
> [apache-cassandra-3.0.9.jar:3.0.9]
>
> --
> Regards,
> Mahesh Rajamani

RE: Cassandra Daemon not coming up

2018-02-27 Thread ZAIDI, ASAD A 
Can you check if you’ve enough disk space available ?
~Asad

From: mahesh rajamani [mailto:rajamani.mah...@gmail.com]
Sent: Tuesday, February 27, 2018 10:11 AM
To: user@cassandra.apache.org
Subject: Cassandra Daemon not coming up

I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node 
down after a restart. The cassandra VM is not coming up with an asset error as 
below. On running in debug mode it is failing while doing operation on " 
resource_role_permissons_index" in system_auth keyspace. Please let me know how 
to bring the cassandra running from this state.

Logs from system.log

INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - Initializing 
system_schema.columns
INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - Initializing 
system_schema.triggers
INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - Initializing 
system_schema.dropped_columns
INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - Initializing 
system_schema.views
INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - Initializing 
system_schema.types
INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - Initializing 
system_schema.functions
INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - Initializing 
system_schema.aggregates
INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - Initializing 
system_schema.indexes
ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception 
encountered during startup
java.lang.AssertionError: null
at 
org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
 ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) 
~[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) 
[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) 
[apache-cassandra-3.0.9.jar:3.0.9]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) 
[apache-cassandra-3.0.9.jar:3.0.9]

--
Regards,
Mahesh Rajamani

Cassandra Daemon not coming up

2018-02-27 Thread mahesh rajamani 
I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple
node down after a restart. The cassandra VM is not coming up with an asset
error as below. On running in debug mode it is failing while doing
operation on " resource_role_permissons_index" in system_auth keyspace.
Please let me know how to bring the cassandra running from this state.

Logs from system.log

INFO  [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 -
Initializing system_schema.columns

INFO  [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 -
Initializing system_schema.triggers

INFO  [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 -
Initializing system_schema.dropped_columns

INFO  [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 -
Initializing system_schema.views

INFO  [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 -
Initializing system_schema.types

INFO  [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 -
Initializing system_schema.functions

INFO  [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 -
Initializing system_schema.aggregates

INFO  [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 -
Initializing system_schema.indexes

ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception
encountered during startup

java.lang.AssertionError: null


at
org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856)
~[apache-cassandra-3.0.9.jar:3.0.9]

at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136)
~[apache-cassandra-3.0.9.jar:3.0.9]

at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
~[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239)
[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568)
[apache-cassandra-3.0.9.jar:3.0.9]

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696)
[apache-cassandra-3.0.9.jar:3.0.9]

-- 
Regards,
Mahesh Rajamani