RE: [EXTERNAL] Re: Error after 3.1.0 to 3.11.2 upgrade
A couple additional things: - Make sure that you ran repair on the system_auth keyspace on all nodes after changing the RF - If you are not often changing roles/permissions, you might look to increase permissions_validity_in_ms and roles_validity_in_ms so they are not being fetched all the time (especially with the internal Cassandra Authorizer/Authenticator). Sean Durity From: Jeff Jirsa Sent: Saturday, May 12, 2018 9:21 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Re: Error after 3.1.0 to 3.11.2 upgrade RF of one means all auth requests go to the same node, so they’re more likely to time out if that host is overloaded or restarts Increasing it distributed the queries among more hosts -- Jeff Jirsa On May 12, 2018, at 6:11 AM, Abdul Patel mailto:abd786...@gmail.com>> wrote: Yeah found that all had 3 replication factor and system_auth had 1 , chnaged to 3 now ..so was this issue due to system_auth replication facyor mismatch? On Saturday, May 12, 2018, Hannu Kröger mailto:hkro...@gmail.com>> wrote: Hi, Did you check replication strategy and amounts of replicas of system_auth keyspace? Hannu Abdul Patel mailto:abd786...@gmail.com>> kirjoitti 12.5.2018 kello 5.21: No applicatiom isnt impacted ..no complains .. Also its an 4 node cluster in lower non production and all are on same version. On Friday, May 11, 2018, Jeff Jirsa mailto:jji...@gmail.com>> wrote: The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated isn’t great, but is the application impacted? -- Jeff Jirsa On May 12, 2018, at 6:17 AM, Abdul Patel mailto:abd786...@gmail.com>> wrote: Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is repeatedly coming , whats solution for this? WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize # for ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) ~[apache-cassandra-3.11.2.jar:3.11.2] On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa mailto:jji...@gmail.com>> wrote: That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa On May 12, 2018, at 5:55 AM, Abdul Patel mailto:abd786...@gmail.com>> wrote: HI All, Seen below stack trace messages , in errorlog one day after upgrade. one of the blogs said this might be due to old drivers, but not sure on it. FYI : INFO [HANDSHAKE-/10.152.205.150<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.152.205.150&d=DwMFaQ&c=MtgQEAMQGqekjTjiA
Re: Error after 3.1.0 to 3.11.2 upgrade
RF of one means all auth requests go to the same node, so they’re more likely to time out if that host is overloaded or restarts Increasing it distributed the queries among more hosts -- Jeff Jirsa > On May 12, 2018, at 6:11 AM, Abdul Patel wrote: > > Yeah found that all had 3 replication factor and system_auth had 1 , chnaged > to 3 now ..so was this issue due to system_auth replication facyor mismatch? > >> On Saturday, May 12, 2018, Hannu Kröger wrote: >> Hi, >> >> Did you check replication strategy and amounts of replicas of system_auth >> keyspace? >> >> Hannu >> >>> Abdul Patel kirjoitti 12.5.2018 kello 5.21: >>> >>> No applicatiom isnt impacted ..no complains .. >>> Also its an 4 node cluster in lower non production and all are on same >>> version. >>> On Friday, May 11, 2018, Jeff Jirsa wrote: The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated isn’t great, but is the application impacted? -- Jeff Jirsa > On May 12, 2018, at 6:17 AM, Abdul Patel wrote: > > Seems its coming from 3.10, got bunch of them today for 3.11.2, so if > this is repeatedly coming , whats solution for this? > > WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 > CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize > # for > ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out > - received only 0 responses. > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) > ~[apache-cassandra-3.11.2.jar:3.11.2] > >> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: >> That looks like Cassandra 3.10 not 3.11.2 >> >> It’s also just the auth cache failing to refresh - if it’s transient >> it’s probably not a big deal. If it continues then there may be an issue >> with the cache refresher. >> >> -- >> Jeff Jirsa >> >> >>> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >>> >>> HI All, >>> >>> Seen below stack trace messages , in errorlog one day after upgrade. >>> one of the blogs said this might be due to old drivers, but not sure on >>> it. >>> >>> FYI : >>> >>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>> OutboundTcpConnection.java:510 - Handshaking version with >>> /10.152.205.150 >>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to
Re: Error after 3.1.0 to 3.11.2 upgrade
Yeah found that all had 3 replication factor and system_auth had 1 , chnaged to 3 now ..so was this issue due to system_auth replication facyor mismatch? On Saturday, May 12, 2018, Hannu Kröger wrote: > Hi, > > Did you check replication strategy and amounts of replicas of system_auth > keyspace? > > Hannu > > Abdul Patel kirjoitti 12.5.2018 kello 5.21: > > No applicatiom isnt impacted ..no complains .. > Also its an 4 node cluster in lower non production and all are on same > version. > > On Friday, May 11, 2018, Jeff Jirsa wrote: > >> The read is timing out - is the cluster healthy? Is it fully upgraded or >> mixed versions? Repeated isn’t great, but is the application impacted? >> >> -- >> Jeff Jirsa >> >> >> On May 12, 2018, at 6:17 AM, Abdul Patel wrote: >> >> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if >> this is repeatedly coming , whats solution for this? >> >> WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 >> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize >> # for >> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 >> ErrorMessage.java:384 - Unexpected exception during request >> com.google.common.util.concurrent.UncheckedExecutionException: >> java.lang.RuntimeException: >> org.apache.cassandra.exceptions.ReadTimeoutException: >> Operation timed out - received only 0 responses. >> at >> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >> ~[guava-18.0.jar:na] >> at >> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >> ~[guava-18.0.jar:na] >> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at org.apache.cassandra.service.ClientState.checkPermissionOnRe >> sourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> at >> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) >> ~[apache-cassandra-3.11.2.jar:3.11.2] >> >> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: >> >>> That looks like Cassandra 3.10 not 3.11.2 >>> >>> It’s also just the auth cache failing to refresh - if it’s transient >>> it’s probably not a big deal. If it continues then there may be an issue >>> with the cache refresher. >>> >>> -- >>> Jeff Jirsa >>> >>> >>> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >>> >>> HI All, >>> >>> Seen below stack trace messages , in errorlog one day after upgrade. >>> one of the blogs said this might be due to old drivers, but not sure on >>> it. >>> >>> FYI : >>> >>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>> OutboundTcpConnection.java:510 - Handshaking version with / >>> 10.152.205.150 >>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / >>> 10.152.205.150 >>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: >>> Cannot achieve consistency level LOCAL_ONE >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na
Re: Error after 3.1.0 to 3.11.2 upgrade
Hi, Did you check replication strategy and amounts of replicas of system_auth keyspace? Hannu > Abdul Patel kirjoitti 12.5.2018 kello 5.21: > > No applicatiom isnt impacted ..no complains .. > Also its an 4 node cluster in lower non production and all are on same > version. > >> On Friday, May 11, 2018, Jeff Jirsa wrote: >> The read is timing out - is the cluster healthy? Is it fully upgraded or >> mixed versions? Repeated isn’t great, but is the application impacted? >> >> -- >> Jeff Jirsa >> >> >>> On May 12, 2018, at 6:17 AM, Abdul Patel wrote: >>> >>> Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this >>> is repeatedly coming , whats solution for this? >>> >>> WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 >>> CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize >>> # for >>> ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - >>> received only 0 responses. >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> at >>> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) >>> ~[apache-cassandra-3.11.2.jar:3.11.2] >>> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on > it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to > /10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at > com.googl
Re: Error after 3.1.0 to 3.11.2 upgrade
No applicatiom isnt impacted ..no complains .. Also its an 4 node cluster in lower non production and all are on same version. On Friday, May 11, 2018, Jeff Jirsa wrote: > The read is timing out - is the cluster healthy? Is it fully upgraded or > mixed versions? Repeated isn’t great, but is the application impacted? > > -- > Jeff Jirsa > > > On May 12, 2018, at 6:17 AM, Abdul Patel wrote: > > Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this > is repeatedly coming , whats solution for this? > > WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 > CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize > # for > ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: > Operation timed out - received only 0 responses. > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.service.ClientState. > checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.cql3.statements.SelectStatement. > checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar: > 3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) > ~[apache-cassandra-3.11.2.jar:3.11.2] > > On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: > >> That looks like Cassandra 3.10 not 3.11.2 >> >> It’s also just the auth cache failing to refresh - if it’s transient it’s >> probably not a big deal. If it continues then there may be an issue with >> the cache refresher. >> >> -- >> Jeff Jirsa >> >> >> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >> >> HI All, >> >> Seen below stack trace messages , in errorlog one day after upgrade. >> one of the blogs said this might be due to old drivers, but not sure on >> it. >> >> FYI : >> >> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >> OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 >> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / >> 10.152.205.150 >> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >> ErrorMessage.java:384 - Unexpected exception during request >> com.google.common.util.concurrent.UncheckedExecutionException: >> com.google.common.util.concurrent.UncheckedExecutionException: >> java.lang.RuntimeException: >> org.apache.cassandra.exceptions.UnavailableException: >> Cannot achieve consistency level LOCAL_ONE >> at >> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >> ~[guava-18.0.jar:na] >> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >> ~[guava-18.0.jar:na] >> at >> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >> ~[guava-18.0.jar:na] >> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >> ~[apache-cassandra-3.10.jar:3.10]
Re: Error after 3.1.0 to 3.11.2 upgrade
The read is timing out - is the cluster healthy? Is it fully upgraded or mixed versions? Repeated isn’t great, but is the application impacted? -- Jeff Jirsa > On May 12, 2018, at 6:17 AM, Abdul Patel wrote: > > Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is > repeatedly coming , whats solution for this? > > WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 > CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize # cas_sar_dit> for > ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) > ~[apache-cassandra-3.11.2.jar:3.11.2] > >> On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: >> That looks like Cassandra 3.10 not 3.11.2 >> >> It’s also just the auth cache failing to refresh - if it’s transient it’s >> probably not a big deal. If it continues then there may be an issue with the >> cache refresher. >> >> -- >> Jeff Jirsa >> >> >>> On May 12, 2018, at 5:55 AM, Abdul Patel wrote: >>> >>> HI All, >>> >>> Seen below stack trace messages , in errorlog one day after upgrade. >>> one of the blogs said this might be due to old drivers, but not sure on it. >>> >>> FYI : >>> >>> INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 >>> OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 >>> DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 >>> 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to >>> /10.152.205.150 >>> ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 >>> ErrorMessage.java:384 - Unexpected exception during request >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> com.google.common.util.concurrent.UncheckedExecutionException: >>> java.lang.RuntimeException: >>> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve >>> consistency level LOCAL_ONE >>> at >>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) >>> ~[guava-18.0.jar:na] >>> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) >>> ~[guava-18.0.jar:na] >>> at >>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>> ~[guava-18.0.jar:na] >>> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) >>> ~[apache-cassandra-3.10.jar:3.10] >>> at >>> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(Authenti
Re: Error after 3.1.0 to 3.11.2 upgrade
Seems its coming from 3.10, got bunch of them today for 3.11.2, so if this is repeatedly coming , whats solution for this? WARN [Native-Transport-Requests-24] 2018-05-11 16:46:20,938 CassandraAuthorizer.java:96 - CassandraAuthorizer failed to authorize # for ERROR [Native-Transport-Requests-24] 2018-05-11 16:46:20,940 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:260) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:530) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:507) ~[apache-cassandra-3.11.2.jar:3.11.2] On Fri, May 11, 2018 at 8:30 PM, Jeff Jirsa wrote: > That looks like Cassandra 3.10 not 3.11.2 > > It’s also just the auth cache failing to refresh - if it’s transient it’s > probably not a big deal. If it continues then there may be an issue with > the cache refresher. > > -- > Jeff Jirsa > > > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / > 10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: > Cannot achieve consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) > ~[apache-cassandra-3.10.jar:3.10] > at org.apache.cassandra.service.ClientState. > checkPermissionOnResourceChain(ClientState.java:352) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassand
Re: Error after 3.1.0 to 3.11.2 upgrade
That looks like Cassandra 3.10 not 3.11.2 It’s also just the auth cache failing to refresh - if it’s transient it’s probably not a big deal. If it continues then there may be an issue with the cache refresher. -- Jeff Jirsa > On May 12, 2018, at 5:55 AM, Abdul Patel wrote: > > HI All, > > Seen below stack trace messages , in errorlog one day after upgrade. > one of the blogs said this might be due to old drivers, but not sure on it. > > FYI : > > INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 > OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 > DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 > 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to > /10.152.205.150 > ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) > ~[apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.10.jar:3.10] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.10.jar:3.10] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.39.Final.jar:4.0.39.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_51] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.10.jar:3.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > Caused by: com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level LOCAL_ONE > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] >
Error after 3.1.0 to 3.11.2 upgrade
HI All, Seen below stack trace messages , in errorlog one day after upgrade. one of the blogs said this might be due to old drivers, but not sure on it. FYI : INFO [HANDSHAKE-/10.152.205.150] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:510 - Handshaking version with /10.152.205.150 DEBUG [MessagingService-Outgoing-/10.152.205.150-Gossip] 2018-05-09 10:22:27,160 OutboundTcpConnection.java:482 - Done connecting to / 10.152.205.150 ERROR [Native-Transport-Requests-1] 2018-05-09 10:22:29,971 ErrorMessage.java:384 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10] at org.apa
