[
https://issues.apache.org/jira/browse/CASSANDRA-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624607#comment-14624607
]
Amit Khare commented on CASSANDRA-9525:
---------------------------------------
We are facing the same issue in 2.1.3 version as well. The node remains in the
hang state and does not serve any writes going forward. Also, causes memtable
writes to be blocked.
> Commitlog allocation failure doesn't stop the entire node.
> ----------------------------------------------------------
>
> Key: CASSANDRA-9525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jeremiah Jordan
> Fix For: 2.1.x
>
>
> If there is an error opening a commit log segment, the thread dies, but it
> doesn't stop the whole node.
> Got the following on a node:
> {noformat}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-05-29 17:06:45,722 CommitLog.java:397 -
> Failed managing commit log segments. Commit disk failure policy is stop;
> terminating thread
> org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException:
> /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:177)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:397)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:394)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [cassandra-all-2.1.5.jar:2.1.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.io.FileNotFoundException:
> /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied)
> at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_45]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> ~[na:1.8.0_45]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:155)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> ... 5 common frames omitted
> {noformat}
> And the node stayed kind of up, didn't notice something wrong until the node
> died OOM because some threads were dead and others weren't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
