[
https://issues.apache.org/jira/browse/CASSANDRA-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360313#comment-14360313
]
Brandon Williams commented on CASSANDRA-8964:
---------------------------------------------
You may just not have enough file handles, 6442 isn't exactly a lot for LCS.
The debian packaging, for instance, sets it to unlimited.
> SSTable count rises during compactions and max open files exceeded
> ------------------------------------------------------------------
>
> Key: CASSANDRA-8964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8964
> Project: Cassandra
> Issue Type: Bug
> Environment: Apache Cassandra 2.1.2
> Centos 6
> AWS EC2
> i2.2xlarge
> Reporter: Anthony Fisk
> Priority: Critical
> Attachments: lsof_with_tmp.txt, lsof_without_tmp.txt,
> nodetool_cfstats.zip
>
>
> LCS compaction was not able to keep up with the prolonged insert load on one
> of our tables called log, resulting in 2,185 SSTables for that table and
> 1,779 pending compactions all together during a test we were running.
> We stopped our load, unthrottled compaction throughput, increased the
> concurrent compactors from 2 to 8, and let it compact the SSTables.
> All was going well until the number of SSTables count for our log table got
> down to around 97, then began rising again until it had reached 758 SSTables
> 1.5 hours later... (we've been recording the cfstats output every half hour,
> [attached|^nodetool_cfstats.zip])
> Eventually we exceeded the number of open files:
> {code}
> ERROR [MemtableFlushWriter:286] 2015-03-12 13:44:36,748
> CassandraDaemon.java:153 - Exception in thread
> Thread[MemtableFlushWriter:286,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
> (Too many open files)
> at
> org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:75)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:552)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:134)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:390)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:329)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:313)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1037)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
> Caused by: java.io.FileNotFoundException:
> /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
> (Too many open files)
> at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_51]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
> ~[na:1.7.0_51]
> at
> org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:71)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ... 14 common frames omitted
> ERROR [MemtableFlushWriter:286] 2015-03-12 13:44:36,750
> JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
> forcefully due to:
> java.io.FileNotFoundException:
> /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-6618-Index.db
> (Too many open files)
> at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_51]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
> ~[na:1.7.0_51]
> at
> org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:71)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:552)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:134)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:390)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:329)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:313)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1037)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
> {code}
> I restarted Cassandra and the number of SSTables reduced to 81 before
> increasing again... Looking at lsof output, it seemed that the file handles
> for the log table were also continually increasing. It seemed that roughly
> half of them [were for "tmp" files|^lsof_with_tmp.txt] and the other half
> [were for the directory of the table|^lsof_without_tmp.txt]. Also of note, it
> seemed we only had one compactor running on the log table, rather than the 8
> that we had seen before after changing the configuration and restarting.
> Cassandra eventually crashed again with the same error when it reached 6,442
> open files (seen in lsof). Without Cassandra running our system has 1,126
> open files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
