[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069388#comment-15069388 ] Dawid Weiss commented on LUCENE-5716: - This is just a note to self. I requested an open source license for the most recent YourKit profiler (graciously donated by the company -- yay!). There is a new feature I haven't seen before -- probes, exactly in the shape and form I always imagined them to be (I have been a fan of aspectj, but this is done via agents, so it allows modifications of system classes I believe). Very cool. This could be used to track other resource leaks (sockets, etc.). If combined with JUnit rules, it could even detect such leaks at class or method level. https://www.yourkit.com/docs/java/help/probes.jsp https://www.yourkit.com/docs/java/help/event_method_call.jsp > Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). > - > > Key: LUCENE-5716 > URL: https://issues.apache.org/jira/browse/LUCENE-5716 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111057#comment-14111057 ] Ryan Ernst commented on LUCENE-5716: [~dweiss] Any reason this was not put into trunk/4x? Is there more work to do? Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111258#comment-14111258 ] Dawid Weiss commented on LUCENE-5716: - Yeah, this patch is a proof of concept more than a fully fledged solution. It only covers wrapping FileInputStream, doesn't include output streams, NIO or sockets. It also introduces aspectj dependency which may be an annoyance to many people -- I personally like it, but I can understand why others may hate such a piece of bytecode voodoo... Also note my comment above: Option (3) is feasible, I just checked (same project as above, jvmti branch). This is, I think, a much more reliable way to hack into the JVM since it'd allow you to wrap, track (and reload) the core system classes, not just pointcuts (bytecode calls) where these classes are used. It is a more difficult way, but it's also more rewarding in the end (or so I tend to believe). Sadly, I won't be able to work on it in any near future -- too much stuff on my plate right now. If you want to take over, go ahead! Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[
https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019608#comment-14019608
]
Dawid Weiss commented on LUCENE-5716:
-
{code}
[junit4] Assumption #1: 'nightly' test group is disabled (@Nightly())
[junit4] 1 Tracked 76 objects, 76 closed, 0 open.
[junit4] 1 FH:
FileInputStream(C:\Work\lucene-solr-svn\LUCENE-5716\lucene\build\core\test\J0\.\temp\sort3028791570
611301125partition) [closed]
[junit4] 1 FH:
FileInputStream(C:\Work\lucene-solr-svn\LUCENE-5716\lucene\build\core\test\J0\.\temp\sort6926018466
304216348partition) [closed]
[junit4] 1 FH:
FileInputStream(C:\Work\lucene-solr-svn\LUCENE-5716\lucene\build\core\test\J0\.\temp\sort6765538901
416981889partition) [closed]
...
{code}
A list of all open/closed file handles is for debug/ showcase purposes only.
Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
-
Key: LUCENE-5716
URL: https://issues.apache.org/jira/browse/LUCENE-5716
Project: Lucene - Core
Issue Type: Improvement
Reporter: Dawid Weiss
Assignee: Dawid Weiss
Priority: Minor
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[
https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019607#comment-14019607
]
Dawid Weiss commented on LUCENE-5716:
-
You can get a feeling of what it works like by switching to LUCENE-5716 and
running:
{code}
cd lucene
ant test-core -Dtests.slow=false -Dtestcase=TestOfflineSorter
{code}
Try opening a FileInputStream to something (out of the temporary folders) and
leaving it unclosed.
Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
-
Key: LUCENE-5716
URL: https://issues.apache.org/jira/browse/LUCENE-5716
Project: Lucene - Core
Issue Type: Improvement
Reporter: Dawid Weiss
Assignee: Dawid Weiss
Priority: Minor
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019679#comment-14019679 ] Michael McCandless commented on LUCENE-5716: This looks really awesome! Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019820#comment-14019820 ] Shalin Shekhar Mangar commented on LUCENE-5716: --- Very cool! Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018701#comment-14018701 ] ASF subversion and git services commented on LUCENE-5716: - Commit 1600614 from [~dawidweiss] in branch 'dev/branches/LUCENE-5716' [ https://svn.apache.org/r1600614 ] Branch for LUCENE-5716 Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[
https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013590#comment-14013590
]
Dawid Weiss commented on LUCENE-5716:
-
I spoke with Robert at Buzzwords about it a bit. It'd be interesting to capture
file handle leaks from tests (unclosed sockets, unclosed files). There are
essentially no java built-in mechanisms to do it. One could use the operating
system's tools such as lsof, but this would measure ALL file handles, including
handles of open jar files, etc. It's also an interesting puzzle to see if one
can do it from Java in a portable (?) way.
There are a few options.
1) Override bootclasspath for a particular release of the JVM and modify system
classes to allow resource tracking. A similar solution kind of exists already
in FileInputStream:
{code}
public int read() throws IOException {
Object traceContext = IoTrace.fileReadBegin(path);
int b = 0;
try {
b = read0();
} finally {
IoTrace.fileReadEnd(traceContext, b == -1 ? 0 : 1);
}
return b;
}
{code}
the obscure IoTrace class unfortunately only tracks file writes, not its
open/close status. A similar hack could be applied to capture constructors and
close calls though.
2) Use bytecode transformation and capture all places where interesting objects
are created/ closed. AspectJ is closest to my heart since it allows fast
prototyping and is essentially just a bytecode manipulation framework. I've
written a simple aspect tracing FileInputStream usage, here's the code.
https://github.com/dweiss/fhandle-tracing
The aspect itself:
https://github.com/dweiss/fhandle-tracing/blob/master/src/main/aspect/com/carrotsearch/aspects/FileInputStreamTracker.aj
There is a JUnit test in there and if you run mvn test, you can see that it's
actually working quite nice. Not everything can be easily addressed (for
example, it's difficult to advise close in classes that inherit from FIS but
don't override this method), but alternative solutions to the problem also
exist (capture all close calls, capture all constructors of
FileInputStream+.new). Doable and should work for 99% of the use cases I think.
3) use jvmti instrumentation agents to essentially provide the same
instrumentation as above. I don't think it is functionally any different from
(2) and I like aspect's syntax for fast hacking. The only exception might be
(didn't check) if we could redefine and reload the definitions of core Java
classes (FileInputStream, for example) so that we can instrument constructors
and methods of these base classes directly. This would be quite nice because
then we'd be able to make it portable across all JVMs and it should work for
all code fragments, including standard library itself.
I'll keep experimenting with (2) for now as it's the low-hanging fruit (already
works), but if somebody wants to inspect (3) it'd be quite interesting.
Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
-
Key: LUCENE-5716
URL: https://issues.apache.org/jira/browse/LUCENE-5716
Project: Lucene - Core
Issue Type: Improvement
Reporter: Dawid Weiss
Assignee: Dawid Weiss
Priority: Minor
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013604#comment-14013604 ] Shalin Shekhar Mangar commented on LUCENE-5716: --- In a past job, I have used a custom protocol factory for http and https connections and tracked open/close, bytes read/written using a thread-local to make sure that webapps aren't leaking connections. Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013606#comment-14013606 ] Dawid Weiss commented on LUCENE-5716: - Thanks Shalin. What I'm aiming at here is much more low-level (and should be more accurate). We'll see how it goes. Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5716) Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly).
[ https://issues.apache.org/jira/browse/LUCENE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014246#comment-14014246 ] Dawid Weiss commented on LUCENE-5716: - Option (3) is feasible, I just checked (same project as above, jvmti branch). Even though FileInputStream is already loaded, it can be redefined with (relative) ease. If we used asm we could just inject some simple before/after hooks that would call to some normal (not-asmified) code. Track file handle leaks (FileDescriptor, NIO Path SPI and Socket mostly). - Key: LUCENE-5716 URL: https://issues.apache.org/jira/browse/LUCENE-5716 Project: Lucene - Core Issue Type: Improvement Reporter: Dawid Weiss Assignee: Dawid Weiss Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
