[
https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013590#comment-14013590
]
Dawid Weiss edited comment on LUCENE-5716 at 6/1/14 6:47 PM:
-------------------------------------------------------------
I spoke with Robert at Buzzwords about it a bit. It'd be interesting to capture
file handle leaks from tests (unclosed sockets, unclosed files). There are
essentially no java built-in mechanisms to do it. One could use the operating
system's tools such as lsof, but this would measure ALL file handles, including
handles of open jar files, etc. It's also an interesting puzzle to see if one
can do it from Java in a portable (?) way.
There are a few options.
1) Override bootclasspath for a particular release of the JVM and modify system
classes to allow resource tracking. A similar "solution" kind of exists already
in FileInputStream:
{code}
public int read() throws IOException {
Object traceContext = IoTrace.fileReadBegin(path);
int b = 0;
try {
b = read0();
} finally {
IoTrace.fileReadEnd(traceContext, b == -1 ? 0 : 1);
}
return b;
}
{code}
the obscure IoTrace class unfortunately only tracks file writes, not its
open/close status. A similar hack could be applied to capture constructors and
close calls though.
2) Use bytecode transformation and capture all places where interesting objects
are created/ closed. AspectJ is closest to my heart since it allows fast
prototyping and is essentially just a bytecode manipulation framework. I've
written a simple aspect tracing FileInputStream usage, here's the code.
https://github.com/dweiss/fhandle-tracing
The aspect itself:
https://github.com/dweiss/fhandle-tracing/blob/master/src/main/aspect/com/carrotsearch/aspects/FileInputStreamTracker.aj
There is a JUnit test in there and if you run mvn test, you can see that it's
actually working quite nice. Not everything can be easily addressed (for
example, it's difficult to advise close in classes that inherit from FIS but
don't override this method), but alternative solutions to the problem also
exist (capture all close calls, capture all constructors of
FileInputStream+.new). Doable and should work for 99% of the use cases I think.
3) use jvmti instrumentation agents to essentially provide the same as above. I
don't think it is functionally any different from (2) and I like aspectj's
syntax for fast hacking. The only add-on value might be (didn't check) if we
could redefine and reload the definitions of core Java classes
(FileInputStream, for example) so that we can instrument constructors and
methods of these base classes directly. This would be quite nice because then
we'd be able to make it portable across all JVMs and it should work for all the
code, including standard library itself.
4) Use btrace and dump a log, then do post-mortem analysis for unclosed files.
Doable, although a bit hairy.
https://kenai.com/projects/btrace/sources/hg/content/samples/FileTracker.java
was (Author: dweiss):
I spoke with Robert at Buzzwords about it a bit. It'd be interesting to capture
file handle leaks from tests (unclosed sockets, unclosed files). There are
essentially no java built-in mechanisms to do it. One could use the operating
system's tools such as lsof, but this would measure ALL file handles, including
handles of open jar files, etc. It's also an interesting puzzle to see if one
can do it from Java in a portable (?) way.
There are a few options.
1) Override bootclasspath for a particular release of the JVM and modify system
classes to allow resource tracking. A similar "solution" kind of exists already
in FileInputStream:
{code}
public int read() throws IOException {
Object traceContext = IoTrace.fileReadBegin(path);
int b = 0;
try {
b = read0();
} finally {
IoTrace.fileReadEnd(traceContext, b == -1 ? 0 : 1);
}
return b;
}
{code}
the obscure IoTrace class unfortunately only tracks file writes, not its
open/close status. A similar hack could be applied to capture constructors and
close calls though.
2) Use bytecode transformation and capture all places where interesting objects
are created/ closed. AspectJ is closest to my heart since it allows fast
prototyping and is essentially just a bytecode manipulation framework. I've
written a simple aspect tracing FileInputStream usage, here's the code.
https://github.com/dweiss/fhandle-tracing
The aspect itself:
https://github.com/dweiss/fhandle-tracing/blob/master/src/main/aspect/com/carrotsearch/aspects/FileInputStreamTracker.aj
There is a JUnit test in there and if you run mvn test, you can see that it's
actually working quite nice. Not everything can be easily addressed (for
example, it's difficult to advise close in classes that inherit from FIS but
don't override this method), but alternative solutions to the problem also
exist (capture all close calls, capture all constructors of
FileInputStream+.new). Doable and should work for 99% of the use cases I think.
3) use jvmti instrumentation agents to essentially provide the same as above. I
don't think it is functionally any different from (2) and I like aspectj's
syntax for fast hacking. The only add-on value might be (didn't check) if we
could redefine and reload the definitions of core Java classes
(FileInputStream, for example) so that we can instrument constructors and
methods of these base classes directly. This would be quite nice because then
we'd be able to make it portable across all JVMs and it should work for all the
code, including standard library itself.
I'll keep experimenting with (2) for now as it's the low-hanging fruit (already
works), but if somebody wants to inspect (3) it'd be quite interesting (I may
do it too, time permitting).
> Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
> -------------------------------------------------------------------------
>
> Key: LUCENE-5716
> URL: https://issues.apache.org/jira/browse/LUCENE-5716
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
