[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118625#comment-16118625
]
Aleksey Yeschenko commented on CASSANDRA-13043:
---
bq. I don't understand why the replica node (the node where this exception is
triggered) is checking for Consistency Level though, but I guess it might be a
corner case of how counters are handled.
The coordinator for the counter write - the initial point of contact - will
pick the leader first, the node will end up owning the write. The leader would
perform the blocking read before write, then propagate the applied change to
the rest of the replica set.
Even if the coordinator is a replica itself, it might still elect another one
as a leader - to balance the load.
Picking the leader is pretty simplistic. We pick a random node in the local DC,
if any. If none, we just go and pick the closest replica to the coordinator, as
measured by the snitch.
As for checking for consistency level, we do it twice. Once on the coordinator,
to allow a faster response if we know in advance that we cannot fulfil the CL,
and then again on the leader itself, as part of {{StorageProxy.performWrite()}}.
So, yes,
1) Availability check happens on the leader, but also on the coordinator,
before that.
2) The leader is getting the requests forwarded from the coordinator, not the
client, explaining why despite the node not yet listening for native protocol
clients can still trigger the UE.
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
>
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099896#comment-16099896
]
Stefano Ortolani commented on CASSANDRA-13043:
--
Thanks [~iamaleksey]. If need be, I can assist in testing patches.
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098619#comment-16098619
]
Aleksey Yeschenko commented on CASSANDRA-13043:
---
[~ostefano] Will try to find the time this week. Regarding commit log, you were
correct - it was a red herring. Since 2.1, commitlog only stores the result of
the counter update, and replay is strictly local, unaffected by the state of
the rest of the cluster.
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092977#comment-16092977
]
Stefano Ortolani commented on CASSANDRA-13043:
--
[~iamaleksey] did you manage to give it a look by any chance?
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PM
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087188#comment-16087188
]
Stefano Ortolani commented on CASSANDRA-13043:
--
Seems like the more connections the worse it is (which would make sense). Last
node restart produced > 30 Exceptions, which in turn cause 20 independent
queries at QUORUM level to fail :/
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
>
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063205#comment-16063205
]
Aleksey Yeschenko commented on CASSANDRA-13043:
---
I'll try to help out if you poke me again after July 10th; currently on
vacation, sorry.
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
>
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062950#comment-16062950
]
Stefano Ortolani commented on CASSANDRA-13043:
--
Based on my analysis the request is coming from another node (and not from a
client), apparently the coordinator. I don't understand why the replica node
(the node where this exception is triggered) is checking for Consistency Level
though, but I guess it might be a corner case of how counters are handled.
[~iamaleksey], I have been yold you might have some more insights here re how
counters are handled?
I guess an easy solution would be to delay requests from other nodes.
Would that make sense/being feasible?
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
>
[jira] [Commented] (CASSANDRA-13043) Unable to achieve CL while applying counters from commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-13043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048886#comment-16048886
]
Stefano Ortolani commented on CASSANDRA-13043:
--
Update: turned out the commitlog was a false flag. On all other nodes where I
found this issue I had no replayed mutations.
The missed QUORUMs were client requests being served before the node had full
visibility over the cluster.
> Unable to achieve CL while applying counters from commitlog
> ---
>
> Key: CASSANDRA-13043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13043
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Catalin Alexandru Zamfir
>
> In version 3.9 of Cassandra, we get the following exceptions on the
> system.log whenever booting an agent. They seem to grow in number with each
> reboot. Any idea where they come from or what can we do about them? Note that
> the cluster is healthy (has sufficient live nodes).
> {noformat}
> 2/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMINFO 10:39:47 Updating topology for /10.136.64.120
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-111,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_111]
> 12/14/2016 12:39:47 PMWARN 10:39:47 Uncaught exception on thread
> Thread[CounterMutationStage-118,5,main]: {}
> 12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException:
> Cannot achieve consistency level LOCAL_QUORUM
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> 12/14/2016 12:39:47 PMat
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.9.jar:3.9]
> 12/14/2016 12:39:47 PMat
>
