[ https://issues.apache.org/jira/browse/CASSANDRA-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2794. --------------------------------------- Resolution: Duplicate Fix Version/s: (was: 0.7.7) scrub leaking file handles was fixed for 0.7.7 in CASSANDRA-2669 > Scrub fails on system with blocked compaction with > java.io.FileNotFoundException / Too many open files > ------------------------------------------------------------------------------------------------------ > > Key: CASSANDRA-2794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2794 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.7.6 > Environment: Ubuntu > Reporter: Dominic Williams > > A node is suffering from CASSANDRA-2793. Scrub is run to try and fix the > problem. Although ulimit shows unlimited file handles allowed scrub > eventually fails with java.io.FileNotFoundException / Too many open files: > INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 CompactionManager.java > (line 511) Scrubbing > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db') > INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 CompactionManager.java > (line 652) Scrub of > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db') > complete: 275 rows in new sstable and 0 empty (tombstoned) rows dropped > INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 CompactionManager.java > (line 511) Scrubbing > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db') > INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 CompactionManager.java > (line 652) Scrub of > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db') > complete: 297 rows in new sstable and 0 empty (tombstoned) rows dropped > INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 CompactionManager.java > (line 511) Scrubbing > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db') > INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java > (line 652) Scrub of > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db') > complete: 347 rows in new sstable and 0 empty (tombstoned) rows dropped > INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java > (line 511) Scrubbing > SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db') > ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 > AbstractCassandraDaemon.java (line 114) Fatal exception in thread > Thread[CompactionExecutor:1,1,main] > java.io.FileNotFoundException: > /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data.db (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113) > at > org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78) > at > org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2243) > at > org.apache.cassandra.db.CompactionManager.maybeCreateWriter(CompactionManager.java:794) > at > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:534) > at > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) > at > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > This error eventually "takes over": > ERROR [ReadStage:328] 2011-06-17 00:46:38,634 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[ReadStage:328,5,main] > java.io.IOError: java.io.FileNotFoundException: > /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too > many open files) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553) > at > org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442) > at > org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.FileNotFoundException: > /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74) > ... 8 more > ERROR [ReadStage:343] 2011-06-17 00:46:48,624 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[ReadStage:343,5,main] > java.io.IOError: java.io.FileNotFoundException: > /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too > many open files) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78) > at > org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553) > at > org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442) > at > org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.FileNotFoundException: > /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74) > ... 8 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira