[
https://issues.apache.org/jira/browse/CASSANDRA-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300403#comment-15300403
]
Tyler Hobbs commented on CASSANDRA-11891:
-----------------------------------------
For any operators that hit this: another workaround is to temporarily raise
{{write_request_timeout_in_ms}} in cassandra.yaml.
> WriteTimeout during commit log replay due to MV lock
> ----------------------------------------------------
>
> Key: CASSANDRA-11891
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11891
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle, Local Write-Read Paths
> Reporter: Tyler Hobbs
> Priority: Critical
> Fix For: 3.0.x, 3.x
>
>
> During commit log replay, if there are materialized views, it's possible for
> contention on the MV lock to cause a {{WriteTimeoutException}}. This makes
> commit log replay fail, which of course prevents the node from starting up.
> This generally means that the operator has to move the commitlog segments to
> avoid replay.
> Here's a stacktrace of this happening on 3.0.5:
> {noformat}
> ERROR [main] 2016-05-25 15:10:31,120 CassandraDaemon.java:692 - Exception
> encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:624)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:511)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:406)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:153)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
> [apache-cassandra-3.0.5.jar:3.0.5]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.get(AbstractLocalAwareExecutorService.java:200)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> ... 9 common frames omitted
> Suppressed: java.util.concurrent.ExecutionException:
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out -
> received only 0 responses.
> ... 11 common frames omitted
> Caused by: org.apache.cassandra.exceptions.WriteTimeoutException:
> Operation timed out - received only 0 responses.
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:431)
> at
> org.apache.cassandra.db.Keyspace.lambda$apply$62(Keyspace.java:443)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should ignore the {{write_rpc_timeout}} setting while acquiring MV locks
> if we're on the commitlog replay path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
