[
https://issues.apache.org/jira/browse/CASSANDRA-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537409#comment-15537409
]
Tyler Hobbs commented on CASSANDRA-12689:
-----------------------------------------
[~brstgt] thanks for the patch! I think you selected the best approach for
now. My main concern is that it's error prone (in terms of blocking callers
using the wrong method), but I'm not sure how feasible it is to avoid that
without a larger overhaul (e.g. CASSANDRA-10993).
Here's my feedback on the patch:
* I think the sleep in between lock acquisition attempts could be too long when
write timeouts are higher than the default. I would be in favor of switching
to a low, hard-coded timeout (maybe somewhere around 10ms), as it shouldn't be
common for the lock to be held longer than this. A longer sleep could
introduce unnecessary latency spikes under light contention.
* {{PaxosState.commit()}} needs to switch to {{applyNotDeferrable()}}
* Adding some javadocs to the various {{apply()}} methods to state when each
should be used would be good, and help to avoid accidental misuse in the future.
I also have a few code style nitpicks:
* For single-line {{if/else}} bodies, we don't use curly braces (see the
{{TEST_FAIL_MV_LOCKS_COUNT}} handling)
* The {{isDeferrable}} override can just be written as {{isDeferrable |=
TEST_FORCE_DEFERABLE_MUTATIONS}}
* The name {{TEST_FORCE_DEFERABLE_MUTATIONS}} was typo'd (should be
{{DEFERRABLE}})
* You can use {{Integer.getInteger("cassandra.test...")}} and
{{Boolean.getBoolean()}} when setting {{TEST_FAIL_MV_LOCKS_COUNT}} and
{{TEST_FORCE_DEFERRABLE_MUTATIONS}}
Regarding the new dtest, that looks like a very nice start. Would you mind
opening a pull request against the [dtest
repo|https://github.com/riptano/cassandra-dtest] referencing this ticket? I
can provide feedback for the test on the pull request.
Thanks again for taking the time to fix this.
> All MutationStage threads blocked, kills server
> -----------------------------------------------
>
> Key: CASSANDRA-12689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12689
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: Benjamin Roth
> Priority: Critical
>
> Under heavy load (e.g. due to repair during normal operations), a lot of
> NullPointerExceptions occur in MutationStage. Unfortunately, the log is not
> very chatty, trace is missing:
> {noformat}
> 2016-09-22T06:29:47+00:00 cas6 [MutationStage-1]
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService Uncaught
> exception on thread Thread[MutationStage-1,5,main]: {}
> 2016-09-22T06:29:47+00:00 cas6 #011java.lang.NullPointerException: null
> {noformat}
> Then, after some time, in most cases ALL threads in MutationStage pools are
> completely blocked. This leads to piling up pending tasks until server runs
> OOM and is completely unresponsive due to GC. Threads will NEVER unblock
> until server restart. Even if load goes completely down, all hints are
> paused, and no compaction or repair is running. Only restart helps.
> I can understand that pending tasks in MutationStage may pile up under heavy
> load, but tasks should be processed and dequeud after load goes down. This is
> definitively not the case. This looks more like a an unhandled exception
> leading to a stuck lock.
> Stack trace from jconsole, all Threads in MutationStage show same trace.
> {noformat}
> Name: MutationStage-48
> State: WAITING on java.util.concurrent.CompletableFuture$Signaller@fcc8266
> Total blocked: 137 Total waited: 138.513
> {noformat}
> Stack trace:
> {noformat}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
> org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
> org.apache.cassandra.hints.Hint.apply(Hint.java:96)
> org.apache.cassandra.hints.HintVerbHandler.doVerb(HintVerbHandler.java:91)
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
