[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842130#comment-16842130
]
Sam Tunnicliffe commented on CASSANDRA-15041:
---------------------------------------------
Hey [~eperott].
I agree that leaking the details about the failure back to the user and
spamming the logs are not really desirable and probably quite confusing, so it
would be great to fix them both. I'm not totally convinced about changing the
response type as an auth failure signifies incorrect credentials or
insufficient privileges and so a client should not assume that retrying the
request is a valid strategy. On the other hand, a server error indicates an
issue outside the client's control and suggests that if the same request is
sent again, it may be successful. We could definitely improve the text of the
error response though to give a nicer message when this does occur.
{quote}4.0: TC2 fail
{quote}
This appears to be due to a conflict in the Caffeine cache (earlier branches
use Guava) between {{refreshAfterWrite}} and {{expireAfterWrite}}. Basically.
the refresh keeps trying and failing, which prevents the expiration happening.
I haven't been through the changelog to see exactly what the issue was, but
upgrading to Caffeine 2.7.0 seems to fix this.
> UncheckedExecutionException if authentication/authorization query fails
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
> Reporter: Per Otterström
> Priority: Normal
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:283)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level QUORUM
> at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 47 common frames omitted
> {noformat}
> Also, if {{x_validity_in_ms}} > {{x_update_interval_in_ms}}, then the
> background update thread will fail in a similar way:
> {noformat}
> ERROR [PermissionsCacheRefresh:1] 2019-03-04 16:30:43,541
> CassandraDaemon.java:228 - Exception in thread
> Thread[PermissionsCacheRefresh:1,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:307)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthCache$1.lambda$reload$0(AuthCache.java:180)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level QUORUM
> at
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 11 common frames omitted
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
