A couple additional things:
- Make sure that you ran repair on the system_auth keyspace on all nodes after changing the RF - If you are not often changing roles/permissions, you might look to increase permissions_validity_in_ms and roles_validity_in_ms so they are not being fetched all the time (especially with the internal Cassandra Authorizer/Authenticator). Sean Durity From: Jeff Jirsa <jji...@gmail.com> Sent: Saturday, May 12, 2018 9:21 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: Error after 3.1.0 to 3.11.2 upgrade RF of one means all auth requests go to the same node, so they’re more likely to time out if that host is overloaded or restarts Increasing it distributed the queries among more hosts -- Jeff Jirsa On May 12, 2018, at 6:11 AM, Abdul Patel <abd786...@gmail.com<mailto:abd786...@gmail.com>> wrote: Yeah found that all had 3 replication factor and system_auth had 1 , chnaged to 3 now ..so was this issue due to system_auth replication facyor mismatch? On Saturday, May 12, 2018, Hannu Kröger <hkro...@gmail.com<mailto:hkro...@gmail.com>> wrote: Hi, Did you check replication strategy and amounts of replicas of system_auth keyspace? Hannu Abdul Patel <abd786...@gmail.com<mailto:abd786...@gmail.com>> kirjoitti 12.5.2018 kello 5.21: No applicatiom isnt impacted ..no complains .. Also its an 4 node cluster in lower non production and all are on same version. On Friday, May 11, 2018, Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated isn’t great, but is the application impacted? -- Jeff Jirsa On May 12, 2018, at 6:17 AM, Abdul Patel <abd786...@gmail.com<mailto:abd786...@gmail.com>> wrote: Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is repeatedly coming , whats solution for this? WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize #<User cas_sar_dit> for <table et_session.session> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) ~[apache-cassandra-3.11.2.jar:3.11.2] On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>> wrote: That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa On May 12, 2018, at 5:55 AM, Abdul Patel <abd786...@gmail.com<mailto:abd786...@gmail.com>> wrote: HI All, Seen below stack trace messages , in errorlog one day after upgrade. one of the blogs said this might be due to old drivers, but not sure on it. FYI : INFO [HANDSHAKE-/10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=>] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to /10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=Y8XD0Q963uyonwtvyXypGAKBaC4gihfqvxDECM8v8hU&s=5W-dlzrZjStoaJ_FJmmLmqNGfnZFIAgZ-_B8hMtHLyw&e=> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.PermissionsCache$$Lambda$269/275740217.apply(Unknown Source) ~[na:na] at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na] ... 26 common frames omitted Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:513) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:481) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:285) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.RolesCache$$Lambda$280/1091297529.apply(Unknown Source) ~[na:na] ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
