[
https://issues.apache.org/jira/browse/CASSANDRA-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-11448:
-----------------------------------------
Description:
Currently when you run OOS, this happens:
{noformat}
ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047
CassandraDaemon.java:229 - Exception in thread
Thread[MemtableFlushWriter:8561,5,main] java.lang.RuntimeException:
Insufficient disk space to write 48 bytes
at
org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_66]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
{noformat}
Now your flush writer is dead and postflush tasks build up forever. Instead we
should throw FSWE and trigger the failure policy.
was:
Currently when you run OOS, this happens:
{noformat}
ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047
CassandraDaemon.java:229 - Exception in thread
Thread[MemtableFlushWriter:8561,5,main] java.lang.RuntimeException:
Insufficient disk space to write 48 bytes
at
org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120)
~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_66]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
{noformat}
Now your flush is dead and postflush tasks build up forever. Instead we should
throw FSWE and trigger the failure policy.
> Running OOS should trigger the disk failure policy
> --------------------------------------------------
>
> Key: CASSANDRA-11448
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11448
> Project: Cassandra
> Issue Type: Bug
> Reporter: Brandon Williams
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> Currently when you run OOS, this happens:
> {noformat}
> ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047
> CassandraDaemon.java:229 - Exception in thread
> Thread[MemtableFlushWriter:8561,5,main] java.lang.RuntimeException:
> Insufficient disk space to write 48 bytes
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.1.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> {noformat}
> Now your flush writer is dead and postflush tasks build up forever. Instead
> we should throw FSWE and trigger the failure policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
