[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841705#comment-16841705
]
Per Otterström commented on CASSANDRA-15041:
--------------------------------------------
So, this is mostly a cosmetic issue in the sense that there is no way to
resolve the underlying problem - that not enough replicas are available to read
from the system_auth tables. Still, this generates error messages that cause
operators to jump, and Cassandra isn't behaving well towards clients when this
happens.
I've updated the auth_test.py to use consistent cache settings and created a
few new test cases to reproduce this issue in the scenario where not enough
replicas are available. They revile unwanted behavior with small variation on
different releases. In short:
* TC1 will trigger Cassandra to perform a background update of a cached
credentials/roles/permissions entries.
* TC2 will trigger authorization when cached entries have passed both
update-interval and validity (blocking update)
* TC3 will trigger authorization when cache is disabled.
Link to dtest
[patch|https://github.com/apache/cassandra-dtest/compare/master...eperott:cassandra-15041].
I'm expecting Cassandra to fail gracefully on TC1, possibly with a warning, but
no stack trace.
I'm expecting TC2 and TC3 to reject the request with exception to indicate
not-authorized|unavailable|timeout, not sure which makes most sense. In any
case, TC2 and TC3 should fail the same way and there should be no errors or
stack traces in the log.
Results on different releases:
* 4.0: TC2 fail
* 3.11: TC1, TC2, TC3 and existing test_login fail
* 3.0: TC1, TC2 and TC3 fail
* 2.2: TC1, TC2 and TC3 fail
4.0 behaves generally better since a similar ticket as this one was fixed in
CASSANDRA-13113. The reason TC2 fails is that the request actually will be
authorized, even though the cached entries should have timed out. From what I
can tell the cache is handing out stale entries.
The reason test_login fail on 3.11 branch is that we're caching credentials
since 3.4.
All new test cases fail on 3.11, 3.0 and 2.2 as reported in this ticket.
So far I've made no attempt to work on a fix for this. Before we dive into
that, I'd like to get some feedback on the dtests and my findings above.
TC2 and TC3 currently expect an UnavailableException and a message similar to
"Cannot achieve consistency level QUORUM" at the client side, simply because
this is the behavior in 4.0 branch. I feel this might confuse users a bit as
the exception and message is related to the internal lookup on the
system_auth.* table, rather than the actual query sent to the cluster. Would it
make more sense to throw back an UnauthorizedException?
[~beobal] and [~ifesdjeen], you were both much involved in CASSANDRA-13113.
What are your thoughts on this?
> UncheckedExecutionException if authentication/authorization query fails
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
> Reporter: Per Otterström
> Priority: Normal
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:283)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level QUORUM
> at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 47 common frames omitted
> {noformat}
> Also, if {{x_validity_in_ms}} > {{x_update_interval_in_ms}}, then the
> background update thread will fail in a similar way:
> {noformat}
> ERROR [PermissionsCacheRefresh:1] 2019-03-04 16:30:43,541
> CassandraDaemon.java:228 - Exception in thread
> Thread[PermissionsCacheRefresh:1,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:307)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthCache$1.lambda$reload$0(AuthCache.java:180)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level QUORUM
> at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 11 common frames omitted
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
