[
https://issues.apache.org/jira/browse/CASSANDRA-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937893#comment-15937893
]
Alex Petrov commented on CASSANDRA-13113:
-----------------------------------------
I've investigated a bit deeper. Although in my opinion it's kind of a
regression, even if it's not super-serious, but it has some user-facing
implications. I've ran {{bisect}} and narrowed it down to [this
commit|https://github.com/apache/cassandra/commit/c607d76413be81a0e125c5780e068d7ab7594612]
Checking logs reveals that before this commit, we had error messages in the
form of:
{code}
Error from server: code=0100 [Bad credentials] message="Error during
authentication of user cassandra :
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level QUORUM"
{code}
After, it's changed to
{code}
Error from server: code=0000 [Server error]
message="java.lang.RuntimeException:
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
consistency level QUORUM"
{code}
I've checked underlying code and it looks like Guava was doing some unwrapping
in case of runtime exceptions on [cache
loading|http://grepcode.com/file/repo1.maven.org/maven2/com.google.guava/guava/11.0/com/google/common/cache/LocalCache.java#2234]
(might be a wrong guava version but you get the idea). Previously, we had to
unwrap the {{UncheckedExecutionException}} in order to extract cause and [turn
it into authentication
exception|https://github.com/ifesdjeen/cassandra/commit/c607d76413be81a0e125c5780e068d7ab7594612#diff-ef1e335e8d51911f09bcc735b0632c5cL97],
in order to trigger a correct error code. Now, we don't have to since
exception isn't un/rewrapped.
The stack trace of the other exception that was happening and causing {{Server
error}} instead of {{Bad Credentials}} was
{code}
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:487)
[main/:na]
at
org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
[main/:na]
at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
[main/:na]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[main/:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_121]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[main/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
{code}
Consequently, I have removed guava-specific exception rewrapping. The other
places (JMX permissions cache, Credentials cache, Passwords cache and
Permissions cache) look fine, with an exception with Permission cache where we
do re-wrap an exception but that doesn't change bubbling/error code.
|[trunk|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:13367-trunk]|[dtest|https://cassci.datastax.com/job/ifesdjeen-13367-trunk-dtest/]|[testall|https://cassci.datastax.com/job/ifesdjeen-13367-trunk-testall/]|
I wanted to add that it might be not a very good style that we're using
exceptions for a control flow. We might want to think of another way to handle
such things in future, at least for the top-tier user facing return codes. Or
at least as it was mentioned in [the
comment|https://issues.apache.org/jira/browse/CASSANDRA-10855?focusedCommentId=15789267&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15789267],
we might want to cover such things with test for future (not necessarily unit
tests even).
> test failure in auth_test.TestAuth.system_auth_ks_is_alterable_test
> -------------------------------------------------------------------
>
> Key: CASSANDRA-13113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13113
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
> Reporter: Sean McCarthy
> Assignee: Alex Petrov
> Labels: dtest, test-failure
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1466/testReport/auth_test/TestAuth/system_auth_ks_is_alterable_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 358, in run
> self.tearDown()
> File "/home/automaton/cassandra-dtest/dtest.py", line 582, in tearDown
> raise AssertionError('Unexpected error in log, see stdout')
> {code}{code}
> Standard Output
> Unexpected error in node2 log, error:
> ERROR [Native-Transport-Requests-1] 2017-01-08 21:10:55,056 Message.java:623
> - Unexpected exception during request; channel = [id: 0xf39c6dae,
> L:/127.0.0.2:9042 - R:/127.0.0.1:43640]
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503)
> ~[main/:na]
> at
> org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310)
> ~[main/:na]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [main/:na]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level QUORUM
> at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1734)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1696)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1642)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1557)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:964)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:252)
> ~[main/:na]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:511)
> ~[main/:na]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:493)
> ~[main/:na]
> ... 13 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
