[
https://issues.apache.org/jira/browse/CASSANDRA-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088090#comment-17088090
]
Jai Bheemsen Rao Dhanwada commented on CASSANDRA-15449:
-------------------------------------------------------
Any pointers here?
Today, I saw an issue on a 3 node cluster where, I just started adding new
nodes (bootstrap) and see the issue.
In this case RF:3
Consistency for Read Queries: Local_QUORUM.
As I mentioned initially I don't see any exceptions or errors in the Cassandra
logs.
> Credentials out of sync after replacing the nodes
> -------------------------------------------------
>
> Key: CASSANDRA-15449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15449
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jai Bheemsen Rao Dhanwada
> Priority: Normal
> Attachments: Screen Shot 2019-12-12 at 11.13.52 AM.png
>
>
> Hello,
> We are seeing a strange issue where, after replacing multiple C* nodes from
> the clusters intermittently we see an issue where few nodes doesn't have any
> credentials and the client queries fail.
> Here are the sequence of steps
> 1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes
> in one DC.
> 2. The approach we took to replace the nodes is kill one node and launch a
> new node with {{-Dcassandra.replace_address=}} and proceed with next node
> once the node is bootstrapped and CQL is enabled.
> 3. This process works fine and all of a sudden, we started seeing our
> application started failing with the below errors in the logs
> {quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc
> has no SELECT permission on <table ks.cf> or any of its parents at
> com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59)
> at
> com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25)
> at
> {quote}
> 4. At this stage we see that 3 nodes in the cluster takes zero traffic, while
> rest of the nodes are serving ~100 requests. (attached the metrics)
> 5. We suspect some credentials sync issue and manually synced the
> credentials and restarted the nodes with 0 requests, which fixed the problem.
> Also, one few C* nodes we see below exception immediately after the bootstrap
> is completed and the process dies. is this contributing to the credentials
> issue?
> NOTE: The C* nodes with zero traffic and the nodes with the below exception
> are not the same.
> {quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 -
> Exception encountered during startup
> java.lang.AssertionError:
> org.apache.cassandra.exceptions.InvalidRequestException: Undefined name
> salted_hash in selection clause
> at
> org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.auth.Auth.setup(Auth.java:144)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
> [apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
> [apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
> [apache-cassandra-2.1.16.jar:2.1.16]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
> Undefined name salted_hash in selection clause
> at
> org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> at
> org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198)
> ~[apache-cassandra-2.1.16.jar:2.1.16]
> ... 7 common frames omitted
> {quote}
> Not sure why this is happening, is this a potential bug or any other pointers
> to fix the problem.
> C* Version: 2.1.16
> Client: Datastax Java Driver.
> system_auth RF: 3, dc-1:3 and dc-2:3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
