[
https://issues.apache.org/jira/browse/CASSANDRA-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903996#comment-13903996
]
Andreas Schnitzerling commented on CASSANDRA-6283:
--------------------------------------------------
I updated 2 of 8 nodes to 2.0.5-rel, with finalizer patch and LEAK-logging.
Result: After Start a lot of LEAK-finalizer Errors like:
{panel:title=start-up node}
ERROR [Finalizer] 2014-02-18 12:53:42,388 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
ERROR [Finalizer] 2014-02-18 12:53:42,388 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\system\schema_keyspaces\system-schema_keyspaces-jb-433-Data.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1362)
at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1147)
at
org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1599)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1718)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1656)
at
org.apache.cassandra.db.SystemKeyspace.serializedSchema(SystemKeyspace.java:767)
at
org.apache.cassandra.db.DefsTables.loadFromKeyspace(DefsTables.java:121)
at
org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:525)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:242)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
ERROR [Finalizer] 2014-02-18 12:53:42,388 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\system\local\system-local-jb-168-Data.db allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
at
org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:57)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1560)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:232)
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:250)
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:58)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:255)
at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:526)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:233)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
{panel}
After Starting >>nodetool repair -par nieste<< I got again a lot of such
messages ("nieste" is my cql3 KS):
{panel:title=with -par}
ERROR [Finalizer] 2014-02-18 13:00:43,961 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\nieste\nfiles\nieste-nfiles-jb-1150-Index.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:103)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:90)
at
org.apache.cassandra.io.util.BufferedPoolingSegmentedFile.createReader(BufferedPoolingSegmentedFile.java:45)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
at
org.apache.cassandra.io.util.SegmentedFile$SegmentIterator.next(SegmentedFile.java:162)
at
org.apache.cassandra.io.util.SegmentedFile$SegmentIterator.next(SegmentedFile.java:143)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:936)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:871)
at
org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:783)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1186)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1174)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:252)
at
org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:888)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:787)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR [Finalizer] 2014-02-18 13:00:44,371 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\nieste\niesteinverters\nieste-niesteinverters-jb-1492-Index.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:103)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:95)
at
org.apache.cassandra.io.sstable.SSTableReader.openIndexReader(SSTableReader.java:1369)
at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:97)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1190)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1174)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
at
org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:888)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:787)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{panel}
On the neighbour-node I got the same errors:
{panel:title=neighbour}
ERROR [Finalizer] 2014-02-18 13:03:18,203 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\nieste\niesteinverters\nieste-niesteinverters-jb-3271-Data.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:34)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:48)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:96)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1190)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1174)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
at
org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:888)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:787)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR [Finalizer] 2014-02-18 13:03:27,543 RandomAccessReader.java (line 394)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
D:\Programme\cassandra\data\nieste\niesteplants\nieste-niesteplants-jb-7-Index.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:63)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:103)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:90)
at
org.apache.cassandra.io.util.BufferedPoolingSegmentedFile.createReader(BufferedPoolingSegmentedFile.java:45)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
at
org.apache.cassandra.io.util.SegmentedFile$SegmentIterator.next(SegmentedFile.java:162)
at
org.apache.cassandra.io.util.SegmentedFile$SegmentIterator.next(SegmentedFile.java:143)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:936)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:871)
at
org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:783)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1186)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1174)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:252)
at
org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:888)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:787)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{panel}
Repair succeed. So only lost filehandles. The strings "delete" or
"FSWriteError" don't appear in any of these logs (w/-par).
> Windows 7 data files keept open / can't be deleted after compaction.
> --------------------------------------------------------------------
>
> Key: CASSANDRA-6283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows 7 (32) / Java 1.7.0.45
> Reporter: Andreas Schnitzerling
> Assignee: Joshua McKenzie
> Labels: compaction
> Fix For: 2.0.6
>
> Attachments: leakdetect.patch, screenshot-1.jpg, system.log
>
>
> Files cannot be deleted, patch CASSANDRA-5383 (Win7 deleting problem) doesn't
> help on Win-7 on Cassandra 2.0.2. Even 2.1 Snapshot is not running. The cause
> is: Opened file handles seem to be lost and not closed properly. Win 7
> blames, that another process is still using the file (but its obviously
> cassandra). Only restart of the server makes the files deleted. But after
> heavy using (changes) of tables, there are about 24K files in the data folder
> (instead of 35 after every restart) and Cassandra crashes. I experiminted and
> I found out, that a finalizer fixes the problem. So after GC the files will
> be deleted (not optimal, but working fine). It runs now 2 days continously
> without problem. Possible fix/test:
> I wrote the following finalizer at the end of class
> org.apache.cassandra.io.util.RandomAccessReader:
> {code:title=RandomAccessReader.java|borderStyle=solid}
> @Override
> protected void finalize() throws Throwable {
> deallocate();
> super.finalize();
> }
> {code}
> Can somebody test / develop / patch it? Thx.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
