[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216796#comment-14216796 ] Ryan Boettcher commented on VFS-142: I see, in our case we are explicitly closing all the FileContent instances so this fix should work for us, thanks! Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Assignee: Bernd Eckenfels Fix For: 2.1 Attachments: fix_ThreadData-clear.patch, threadlocal_setNull_vs_remove_bug.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215249#comment-14215249 ] Bernd Eckenfels commented on VFS-142: - Hm, I don't see really a difference. Both methods will clean the value and keep the key (ThreadLocal instance) (weakly references). The Key does not contain references to VFS objects. In fact remove() seems to be a bit more efficient as it would not create the ma or insert the key if not present (anymore). Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch, threadlocal_setNull_vs_remove_bug.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215263#comment-14215263 ] Bernd Eckenfels commented on VFS-142: - Ah stupid me... I reverse-read the patch. Yes remove is better than set :) (But I don't think it explains a leak). Will commit it. Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch, threadlocal_setNull_vs_remove_bug.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215295#comment-14215295 ] Bernd Eckenfels commented on VFS-142: - Commited (at least I tried to, it reported Rev 1640242 but it is not visible. I sent a mal to dev@commons about it and will commit again once resolved) Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch, threadlocal_setNull_vs_remove_bug.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998741#comment-13998741 ] Gary Gregory commented on VFS-142: -- This is all quite alarming. Which behavior can be attributed to the 2.0 and trunk version of the code? Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998429#comment-13998429 ] Adam Heath commented on VFS-142: Our current deployment application does not make use of commons-vfs anymore; we now use more off-the-shelf(but still opensource) tools, and don't maintain as much internal forks of software as we used to. After this original bug was filed, I kept finding more and more issues running commons-vfs in a long-running server environment; from memory, including, but not limited to: * long running thread pools(as you mention) * when nearly out of memory, the app server calls into vfs to read a file, the vfs extensions we wrote then need to parse xml, this then attempts to load a class(which was loaded previously, but was thrown away due to the OOM that will happen shortly), which then calls into a classloader, which is also backed by the very same top-level vfs instance; deadlock ensues. This is really more of a bug in java.lang.Classloader.loadClass, because it is synchronized. * much use of synchronized in the code base * not a bug per se, but an ugliness; at the time, there was no generics in the api. Due to these, and other issues, I basically ended up just using the vfs interfaces, and none of the base support classes. All were rewritten, including the Manager. Final result was generics, no synchronized(uses java.util.concurrent(aka non-blocking), no dead locks, and no mem-leaks. Btw, the primary filesystem implementation we had was an 'overlay', which is similar to linux union; directories could overlay other directories, getChildren() would be merge, it had support for whiteouts. There was also support for mount-points at any level. The metadata for whiteouts was stored as xml as .cowstate.xml(the original name was COWFileSystem, for copy-on-write). We used this to have a large, shared BASE, with multiple overlayed CHILDREN. If a child modified a file, the base was up-copied. There was also a 'FlatAttribute' file system; getAttribute(name) was mapped to 'file/path@/name'; this made storing metadata for vfs files easy, and integrated well with svn and git revision control. Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993082#comment-13993082 ] Ryan Boettcher commented on VFS-142: Just to add to what Adam is saying; over here in my group's environment we are currently using vfs in dropwizard app (uses thread pool for connections so the threads stick around basically forever) to access a filesystem which (for various technical reasons) is several levels deep and has potentially thousands of subdirs and/or files under each dir in the tree, so every leaked filesystem in our case leaves well over 100MB of unrecoverable memory linked to nothing but the thread. With potentially each filesystem leaked multiple times in every pooled thread what this means, in practice (ie. our production environment), is this leak eats up about 4GB of server memory at least twice a week, which is very annoying. We're currently working around the problem by threading all our file operations, but this is very undesirable and we don't want to keep this workaround in production for long. Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (VFS-142) Clear ThreadData after all streams are closed, fixes a memory leak
[ https://issues.apache.org/jira/browse/VFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977315#comment-13977315 ] Ryan Boettcher commented on VFS-142: ThreadLocal actually stores it's data in the Thread itself, I don't think your patch covers the whole problem. Even when the ThreadLocal content is set to null it may not remove the reference in the Thread. Clear ThreadData after all streams are closed, fixes a memory leak -- Key: VFS-142 URL: https://issues.apache.org/jira/browse/VFS-142 Project: Commons VFS Issue Type: Bug Affects Versions: 1.1 Reporter: Adam Heath Attachments: fix_ThreadData-clear.patch After all streams are closed in FileContentThreadData, clear the ThreadLocal. -- This message was sent by Atlassian JIRA (v6.2#6252)
