[
https://issues.apache.org/jira/browse/CASSANDRA-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624607#comment-14624607
]
Amit Khare edited comment on CASSANDRA-9525 at 7/13/15 1:02 PM:
----------------------------------------------------------------
We are facing the same issue in 2.1.3 version as well. The node remains in the
hang state and does not serve any writes going forward. Also, causes memtable
writes to be blocked.
Also, due to this, there were consistency issues seen because the reads were
made on the stale data of the write-dead node.
was (Author: adkhare):
We are facing the same issue in 2.1.3 version as well. The node remains in the
hang state and does not serve any writes going forward. Also, causes memtable
writes to be blocked.
> Commitlog allocation failure doesn't stop the entire node.
> ----------------------------------------------------------
>
> Key: CASSANDRA-9525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jeremiah Jordan
> Fix For: 2.1.x
>
>
> If there is an error opening a commit log segment, the thread dies, but it
> doesn't stop the whole node.
> Got the following on a node:
> {noformat}
> ERROR [COMMIT-LOG-ALLOCATOR] 2015-05-29 17:06:45,722 CommitLog.java:397 -
> Failed managing commit log segments. Commit disk failure policy is stop;
> terminating thread
> org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException:
> /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:177)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:397)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:394)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [cassandra-all-2.1.5.jar:2.1.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.io.FileNotFoundException:
> /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied)
> at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_45]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> ~[na:1.8.0_45]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:155)
> ~[cassandra-all-2.1.5.jar:2.1.5]
> ... 5 common frames omitted
> {noformat}
> And the node stayed kind of up, didn't notice something wrong until the node
> died OOM because some threads were dead and others weren't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
