[jira] [Commented] (CASSANDRA-14328) Invalid metadata has been detected for role
[
https://issues.apache.org/jira/browse/CASSANDRA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486617#comment-17486617
]
Jai Bheemsen Rao Dhanwada commented on CASSANDRA-14328:
---
[~tania.en...@quest.com] [~prnvjndl] were you able to figure out why the node
got into this state? it looks like this is a side effect of bootstrapping
process not streaming all the data
(https://issues.apache.org/jira/browse/CASSANDRA-14006). I see a similar issue
and just wondering if you have any more details and workaround on this issue?
> Invalid metadata has been detected for role
> ---
>
> Key: CASSANDRA-14328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14328
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/CQL
>Reporter: Pranav Jindal
>Priority: Normal
>
> Cassandra Version : 3.10
> One node was replaced and was successfully up and working but CQL-SH fails
> with error.
>
> CQL-SH error:
>
> {code:java}
> Connection error: ('Unable to connect to any servers', {'10.180.0.150':
> AuthenticationFailed('Failed to authenticate to 10.180.0.150: Error from
> server: code= [Server error] message="java.lang.RuntimeException: Invalid
> metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb"',)})
> {code}
>
> Cassandra server ERROR:
> {code:java}
> WARN [Native-Transport-Requests-1] 2018-03-20 13:37:17,894
> CassandraRoleManager.java:96 - An invalid value has been detected in the
> roles table for role utorjwcnruzzlzafxffgyqmlvkxiqcgb. If you are unable to
> login, you may need to disable authentication and confirm that values in that
> table are accurate
> ERROR [Native-Transport-Requests-1] 2018-03-20 13:37:17,895 Message.java:623
> - Unexpected exception during request; channel = [id: 0xdfc3604f,
> L:/10.180.0.150:9042 - R:/10.180.0.150:51668]
> java.lang.RuntimeException: Invalid metadata has been detected for role
> utorjwcnruzzlzafxffgyqmlvkxiqcgb
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.10.jar:3.10]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88)
> ~[apache-cassandra-3.10.jar:3.10]
> ... 16 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14328) Invalid metadata has been detected for role
[
https://issues.apache.org/jira/browse/CASSANDRA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938924#comment-16938924
]
Tania S Engel commented on CASSANDRA-14328:
---
I also had this issue happen on one of our nodes after it joined the Cassandra
cluster. I confirmed by using another role that worked and connecting with
cqlsh and running >use system_auth; select * from roles;
I could see that in the roles can_login column the role had null (rather than
true or false). This is the root cause of your exception.
In my case, joining looked as if it worked from our monitoring. nodetool
netstats did show Mode: Joining but also showed "not sending any streams".
nodetool netstats later was at Mode: normal. {color:#172b4d}nodetool status
showed it was an up node. {color}On closer inspection, the join had not
streamed. I did have debug.log and it showed the node bootstrapped but it never
logged "Creating new streaming plan for bootstrap.." so there was no streaming.
Shortly after the bad node bootstrap started, the debug.log on the good
existing seed node shows a socket closed and failure to connect. It did
reconnect, but perhaps that is the reason the streaming plan never commenced.
It is misleading that the bad node then continued to run and appear as if it
was bootstrapped.
I tried to fix the bad node by running >nodetool repair system_auth but that
did not work. I was able to fix the roles with >nodetool --full system_auth. I
was able to fix the remaining data by running a full repair on all tables.
> Invalid metadata has been detected for role
> ---
>
> Key: CASSANDRA-14328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14328
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/CQL
>Reporter: Pranav Jindal
>Priority: Normal
>
> Cassandra Version : 3.10
> One node was replaced and was successfully up and working but CQL-SH fails
> with error.
>
> CQL-SH error:
>
> {code:java}
> Connection error: ('Unable to connect to any servers', {'10.180.0.150':
> AuthenticationFailed('Failed to authenticate to 10.180.0.150: Error from
> server: code= [Server error] message="java.lang.RuntimeException: Invalid
> metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb"',)})
> {code}
>
> Cassandra server ERROR:
> {code:java}
> WARN [Native-Transport-Requests-1] 2018-03-20 13:37:17,894
> CassandraRoleManager.java:96 - An invalid value has been detected in the
> roles table for role utorjwcnruzzlzafxffgyqmlvkxiqcgb. If you are unable to
> login, you may need to disable authentication and confirm that values in that
> table are accurate
> ERROR [Native-Transport-Requests-1] 2018-03-20 13:37:17,895 Message.java:623
> - Unexpected exception during request; channel = [id: 0xdfc3604f,
> L:/10.180.0.150:9042 - R:/10.180.0.150:51668]
> java.lang.RuntimeException: Invalid metadata has been detected for role
> utorjwcnruzzlzafxffgyqmlvkxiqcgb
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.10.jar:3.10]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
[jira] [Commented] (CASSANDRA-14328) Invalid metadata has been detected for role
[
https://issues.apache.org/jira/browse/CASSANDRA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552714#comment-16552714
]
Kurt Greaves commented on CASSANDRA-14328:
--
[~prnvjndl] If this issue is still occurring can you send the output of:
{code}
select * from system_auth.roles where role = 'utorjwcnruzzlzafxffgyqmlvkxiqcgb'
{code}
You can remove any credentials if you see fit.
> Invalid metadata has been detected for role
> ---
>
> Key: CASSANDRA-14328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14328
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
>Reporter: Pranav Jindal
>Priority: Major
>
> Cassandra Version : 3.10
> One node was replaced and was successfully up and working but CQL-SH fails
> with error.
>
> CQL-SH error:
>
> {code:java}
> Connection error: ('Unable to connect to any servers', {'10.180.0.150':
> AuthenticationFailed('Failed to authenticate to 10.180.0.150: Error from
> server: code= [Server error] message="java.lang.RuntimeException: Invalid
> metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb"',)})
> {code}
>
> Cassandra server ERROR:
> {code:java}
> WARN [Native-Transport-Requests-1] 2018-03-20 13:37:17,894
> CassandraRoleManager.java:96 - An invalid value has been detected in the
> roles table for role utorjwcnruzzlzafxffgyqmlvkxiqcgb. If you are unable to
> login, you may need to disable authentication and confirm that values in that
> table are accurate
> ERROR [Native-Transport-Requests-1] 2018-03-20 13:37:17,895 Message.java:623
> - Unexpected exception during request; channel = [id: 0xdfc3604f,
> L:/10.180.0.150:9042 - R:/10.180.0.150:51668]
> java.lang.RuntimeException: Invalid metadata has been detected for role
> utorjwcnruzzlzafxffgyqmlvkxiqcgb
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.10.jar:3.10]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88)
> ~[apache-cassandra-3.10.jar:3.10]
> ... 16 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
