Yeah found that all had 3 replication factor and system_auth had 1 , chnaged to 3 now ..so was this issue due to system_auth replication facyor mismatch?
On Saturday, May 12, 2018, Hannu Kröger <hkro...@gmail.com> wrote: > Hi, > > Did you check replication strategy and amounts of replicas of system_auth > keyspace? > > Hannu > > Abdul Patel <abd786...@gmail.com> kirjoitti 12.5.2018 kello 5.21: > > No applicatiom isnt impacted ..no complains .. > Also its an 4 node cluster in lower non production and all are on same > version. > > On Friday, May 11, 2018, Jeff Jirsa <jji...@gmail.com> wrote: > >> The read is timing out - is the cluster healthy? Is it fully upgraded or >> mixed versions? Repeated isn’t great, but is the application impacted? >> >> -- >> Jeff Jirsa >> >> >> On May 12, 2018, at 6:17 AM, Abdul Patel <abd786...@gmail.com> wrote: >> >> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if >> this is repeatedly coming , whats solution for this? >> >> WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 >> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize >> #<User cas_sar_dit> for <table et_session.session> >> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 >> ErrorMessage.java:384 - Unexpected exception during request >> com.google.common.util.concurrent.UncheckedExecutionException: >> java.lang.RuntimeException: >> org.apache.cassandra.exceptions.ReadTimeoutException: >> Operation timed out - received only 0 responses. >> at >> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >> ~[guava-18.0.jar:na] >> at >> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >> ~[guava-18.0.jar:na] >> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at org.apache.cassandra.service.ClientState.checkPermissionOnRe >> sourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> >> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa <jji...@gmail.com> wrote: >> >>> That looks like Cassandra 3.10 not 3.11.2 >>> >>> It’s also just the auth cache failing to refresh - if it’s transient >>> it’s probably not a big deal. If it continues then there may be an issue >>> with the cache refresher. >>> >>> -- >>> Jeff Jirsa >>> >>> >>> On May 12, 2018, at 5:55 AM, Abdul Patel <abd786...@gmail.com> wrote: >>> >>> HI All, >>> >>> Seen below stack trace messages , in errorlog one day after upgrade. >>> one of the blogs said this might be due to old drivers, but not sure on >>> it. >>> >>> FYI : >>> >>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>> OutboundTcpConnection.java:510 - Handshaking version with / >>> 10.152.205.150 >>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / >>> 10.152.205.150 >>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: >>> Cannot achieve consistency level LOCAL_ONE >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at org.apache.cassandra.service.ClientState.checkPermissionOnRe >>> sourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) >>> [apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) >>> [apache-cassandra-3.10.jar:3.10] >>> at io.netty.channel.SimpleChannelInboundHandler.channelRead(Sim >>> pleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4. >>> 0.39.Final] >>> at io.netty.channel.AbstractChannelHandlerContext.invokeChannel >>> Read(AbstractChannelHandlerContext.java:366) >>> [netty-all-4.0.39.Final.jar:4.0.39.Final] >>> at io.netty.channel.AbstractChannelHandlerContext.access$600(Ab >>> stractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4. >>> 0.39.Final] >>> at io.netty.channel.AbstractChannelHandlerContext$7.run(Abstrac >>> tChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4. >>> 0.39.Final] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_51] >>> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorSe >>> rvice$FutureTask.run(AbstractLocalAwareExecutorService.java:162) >>> [apache-cassandra-3.10.jar:3.10] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] >>> Caused by: com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: >>> Cannot achieve consistency level LOCAL_ONE >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.PermissionsCache$$Lambda$269/275740217.apply(Unknown >>> Source) ~[na:na] >>> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) >>> ~[guava-18.0.jar:na] >>> ... 26 common frames omitted >>> Caused by: java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: >>> Cannot achieve consistency level LOCAL_ONE >>> at >>> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:513) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:481) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:285) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.RolesCache$$Lambda$280/1091297529.apply(Unknown >>> Source) ~[na:na] >>> >>> >>
