[
https://issues.apache.org/jira/browse/CASSANDRA-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903340#comment-13903340
]
Joshua McKenzie edited comment on CASSANDRA-6283 at 2/17/14 4:11 PM:
---------------------------------------------------------------------
I've reproduced leaked file handles on repair in a lab on 2.0.5
w/leakfinalizer.patch. Nodes start up without issue - I'm not seeing any LEAK
or File-Not-Found on regular init, and the LEAK aren't showing up until repair
kicks off. W/the finalizer patch repair runs through to completion. Andreas -
have you had a chance to try out 2.0.5 w/the patch yet?
lastly - the leaks I'm seeing look like they're all isolated to a single case -
streaming data outbound during the repair process:
{code:title=error|borderStyle=solid}
ERROR [Finalizer] 2014-02-17 09:21:52,922 RandomAccessReader.java (line 399)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
C:\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-jb-41-CRC.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:66)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:106)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:98)
at
org.apache.cassandra.io.util.DataIntegrityMetadata$ChecksumValidator.<init>(DataIntegrityMetadata.java:53)
at
org.apache.cassandra.io.util.DataIntegrityMetadata.checksumValidator(DataIntegrityMetadata.java:40)
at
org.apache.cassandra.streaming.StreamWriter.write(StreamWriter.java:76)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:383)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:355)
at java.lang.Thread.run(Thread.java:744)
{code}
Andreas - could you confirm whether or not this matches what you're seeing in
your environment? I'm curious if this is a dangling file handle like we've
seen in other related tickets or if this is perhaps a race condition on access
Windows is intolerant of.
was (Author: joshuamckenzie):
I've reproduced leaked file handles on repair in a lab on 2.0.5
w/leakfinalizer.patch. Nodes start up without issue - I'm not seeing any LEAK
or File-Not-Found on regular init, and the LEAK aren't showing up until repair
kicks off. W/the finalizer patch repair runs through to completion. Andreas -
have you had a chance to try out 2.0.5 w/the patch yet?
lastly - the leaks I'm seeing look like they're all isolated to a single case -
streaming data outbound during the repair process:
ERROR [Finalizer] 2014-02-17 09:21:52,922 RandomAccessReader.java (line 399)
LEAK finalizer had to clean up
java.lang.Exception: RAR for
C:\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-jb-41-CRC.db
allocated
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:66)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:106)
at
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:98)
at
org.apache.cassandra.io.util.DataIntegrityMetadata$ChecksumValidator.<init>(DataIntegrityMetadata.java:53)
at
org.apache.cassandra.io.util.DataIntegrityMetadata.checksumValidator(DataIntegrityMetadata.java:40)
at
org.apache.cassandra.streaming.StreamWriter.write(StreamWriter.java:76)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:383)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:355)
at java.lang.Thread.run(Thread.java:744)
Andreas - could you confirm whether or not this matches what you're seeing in
your environment? I'm curious if this is a dangling file handle like we've
seen in other related tickets or if this is perhaps a race condition on access
Windows is intolerant of.
> Windows 7 data files keept open / can't be deleted after compaction.
> --------------------------------------------------------------------
>
> Key: CASSANDRA-6283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows 7 (32) / Java 1.7.0.45
> Reporter: Andreas Schnitzerling
> Assignee: Joshua McKenzie
> Labels: compaction
> Fix For: 2.0.6
>
> Attachments: leakdetect.patch, screenshot-1.jpg, system.log
>
>
> Files cannot be deleted, patch CASSANDRA-5383 (Win7 deleting problem) doesn't
> help on Win-7 on Cassandra 2.0.2. Even 2.1 Snapshot is not running. The cause
> is: Opened file handles seem to be lost and not closed properly. Win 7
> blames, that another process is still using the file (but its obviously
> cassandra). Only restart of the server makes the files deleted. But after
> heavy using (changes) of tables, there are about 24K files in the data folder
> (instead of 35 after every restart) and Cassandra crashes. I experiminted and
> I found out, that a finalizer fixes the problem. So after GC the files will
> be deleted (not optimal, but working fine). It runs now 2 days continously
> without problem. Possible fix/test:
> I wrote the following finalizer at the end of class
> org.apache.cassandra.io.util.RandomAccessReader:
> {code:title=RandomAccessReader.java|borderStyle=solid}
> @Override
> protected void finalize() throws Throwable {
> deallocate();
> super.finalize();
> }
> {code}
> Can somebody test / develop / patch it? Thx.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
