Hi, Cluster set up: 1 DC with 5 nodes (each node having 700GB data) 1 kespace with RF of 2 write CL is LOCAL_ONE read CL is LOCAL_QUORUM
One node was down for about 1 hour because of OOM issue. During the down period, all 4 other nodes report "Cannot achieve consistency level LOCAL_ONE" constantly until I brought up the dead node. My data seems lost during that down time. To me this could not happen because the write CL is LOCAL_ONE and only one node was dead. I encountered node down before because of OOM issue and I believe I didn't lose data because of the hinted handoff feature. One more thing, the dead node was added recently and the only difference is the other 4 nodes are behind an internal SLB(Service Load Balance) with VIP, and the new one not. Our application access Casssandra cluster by the SLB VIP. Any thoughts are appreciated. Best regards, -Simon System log: 57659 Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.Unavai lableException: Cannot achieve consistency level LOCAL_ONE 57660 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] 57661 at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] 57662 at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] 57663 at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] 57664 at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:70) ~[apache-cassandra-2.2.8.jar:2.2.8] 57665 at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-2.2.8.jar:2.2.8] 57666 at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-2.2.8.jar:2.2.8] 57667 at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:76) ~[apache-cassandra-2.2.8.jar:2.2.8] 57668 at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.2.8.jar:2.2.8] 57669 at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.2.8.jar:2.2.8] 57670 at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] 57671 at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] 57672 at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] 57673 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] 57674 ... 25 common frames omitted
