Re: [Resin-interest] Memory leak - in Resin?
My initial testing indicates that the leak indeed has been fixed in 4.0! I'll have to get back on this once I have our real application up and running in 4.0 /Mattias Mattias Jiderhamn wrote (2009-04-02 11:23): > Scott Ferguson wrote (2009-03-30 18:20): > >> On Mar 30, 2009, at 3:54 AM, Mattias Jiderhamn wrote: >> >> >>> After drawing the conclusion below, it isn't very far away to realize >>> it probably has to do with some static (i.e. class loader specific) >>> member of EnvironmentClassLoader. >>> And just as I thought, the heart of the problem is >>> private static EnvironmentLocal> >>> _addLoaderListeners >>> >>> More specifically, if line 234 of com.caucho.server.resin.Resin is >>> commented out, the leak is removed (assuming my >>> HttpRequest._invocation patch is also applied) >>>// Environment.addChildLoaderListener(new WebBeansAddLoaderListener()) >>> >>> Scott, you said the WebBeans caches were not to blame because they >>> are classloader local. Well, it seems something about this >>> classloader locality isn't working the way it should, doesn't it...? >>> Do you have enough info to try to fix this in a future (hopefully >>> soon to be released) 3.1 release? >>> >> In the early work with 4.0, I'd cleaned a number of dead links >> (primarily WebBeans, but also some JMX). I believe the current 4.0 >> still has a few left, so I'll need to run a final pass at the end of >> the release cycle. >> > Are you saying this won't be fixed in the 3.1 branch? I'm assuming the > 4.0 production release is still months ahead? > Could we provide you any additional information to change your mind...? > > ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Scott Ferguson wrote (2009-03-30 18:20): > > On Mar 30, 2009, at 3:54 AM, Mattias Jiderhamn wrote: > >> After drawing the conclusion below, it isn't very far away to realize >> it probably has to do with some static (i.e. class loader specific) >> member of EnvironmentClassLoader. >> And just as I thought, the heart of the problem is >> private static EnvironmentLocal> >> _addLoaderListeners >> >> More specifically, if line 234 of com.caucho.server.resin.Resin is >> commented out, the leak is removed (assuming my >> HttpRequest._invocation patch is also applied) >>// Environment.addChildLoaderListener(new WebBeansAddLoaderListener()) >> >> Scott, you said the WebBeans caches were not to blame because they >> are classloader local. Well, it seems something about this >> classloader locality isn't working the way it should, doesn't it...? >> Do you have enough info to try to fix this in a future (hopefully >> soon to be released) 3.1 release? > > In the early work with 4.0, I'd cleaned a number of dead links > (primarily WebBeans, but also some JMX). I believe the current 4.0 > still has a few left, so I'll need to run a final pass at the end of > the release cycle. Are you saying this won't be fixed in the 3.1 branch? I'm assuming the 4.0 production release is still months ahead? Could we provide you any additional information to change your mind...? -- ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Hi, I did some more tests today, as, like you, I suspected that the listener pointing to the Groovy classes where quite suspicious but ran out of time. I repeated the tests substituting the groovy jar file with Xalan and the leak is gone. I performed the same tests and re-initialised the application several times. After each initialisation, the non-heap memory would grow, then go back to a stable number, and the number of instances of EnvironmentClassLoader would also grow and then schrink. To confirm, I re-added the Groovy.jar file and there it was: after each context restart, one extra EnvironmentClassLoader could not be GC-ed and the non-Heap memory would keep growing with the aprox. amount of WEB-INF/lib size. That's the problem with these kind of leaks, the container classloaders always look suspicious because they are always involved in the process and they are the ones that hold the bigger space. But while they can be the problem, just a single problem with one of the libraries and the EnvironmentClassLoader cannot be GC-ed effectively. Well, that means that Groovy is also on the list of buggy libraries for such kind of environment, along with some JDBC drivers, like HSQLDB that starts a thread that is never stopped unless you add some custom-driver-specific code, and with some jakarta commons that do not clean ThreadLocal references appropriately. Not sure what it can be done, apart of notifying the library developers so they can fix those bugs, but it really is a global problem as there are way too many libraries not ready for a throw-away classloader environment :(. S! D. S'està citant Mattias Jiderhamn : > After drawing the conclusion below, it isn't very far away to realize it > probably has to do with some static (i.e. class loader specific) member > of EnvironmentClassLoader. > And just as I thought, the heart of the problem is > private static EnvironmentLocal> > _addLoaderListeners > > More specifically, if line 234 of com.caucho.server.resin.Resin is > commented out, the leak is removed (assuming my HttpRequest._invocation > patch is also applied) >// Environment.addChildLoaderListener(new WebBeansAddLoaderListener()) > > Scott, you said the WebBeans caches were not to blame because they are > classloader local. Well, it seems something about this classloader > locality isn't working the way it should, doesn't it...? > Do you have enough info to try to fix this in a future (hopefully soon > to be released) 3.1 release? > > Maybe we are now seeing the end to all these memory leaks!!! > > /Mattias > > > Mattias Jiderhamn wrote (2009-03-30 11:40): >> Just tried a new idea: I subclassed EnvironmentClassLoader within the >> Resin sources and compiled into resin.jar. There is still a leak (in >> contrast to when I subclassed EnvironmentClassLoader within my own >> application). >> >> Conclusion: The classloader that loads the (Environment)ClassLoader >> class is somehow of importance. >> >> /Mattias >> >> Mattias Jiderhamn wrote (2009-03-30 08:40): >>> Daniel, your findings confirms EnvironmentClassLoader as the prime >>> suspect (my own tests are a bit more detailed in an earlier post, >>> http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html) >>> >>> Regarding your GC root paths for this class: >>> 1. I haven't noticed any uses of java.util.prefs - could this be part >>> of any of your added JARs (like groovy)? >>> 2. This is the error I have found within Resin that is still to be >>> fixed. As mentioned earlier, there is a quick and dirty patch for >>> Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid >>> this leak. However even with the patch applied I'm still having leaks. >>> 3. This obviously seems like a groovy problem, probably a static >>> initialization triggered at class load. >>> >>> So would you mind giving this another try with groovy removed and >>> with a patched version of Resin? (I can provide a patched resin.jar >>> if that helps) >>> >>> Cheers, >>> Mattias Jiderhamn >>> >>> Daniel Lopez wrote (2009-03-29 14:52): Hi, Using that .war as a sample application, I did some more tests and found some things that might be interesting: I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, they simply live there. Important data: total size for WEB-INF/lib: 10.3 MB. I then deployed the application, start it and here's some data I gathered after accessing the application a couple of times and hitting "Force Garbage Collector" until the memory is stable: Heap Memory used: 31MB Instances of com.caucho.loader.EnvironmentClassLoader: 1 I then re-save web.xml so the context is reloaded, access and force GC until memory is stable again: Heap Memory used: 41MB Instances of com.caucho.loader.EnvironmentClassLoader: 2
Re: [Resin-interest] Memory leak - in Resin?
On Mar 30, 2009, at 3:54 AM, Mattias Jiderhamn wrote: After drawing the conclusion below, it isn't very far away to realize it probably has to do with some static (i.e. class loader specific) member of EnvironmentClassLoader. And just as I thought, the heart of the problem is private static EnvironmentLocal> _addLoaderListeners More specifically, if line 234 of com.caucho.server.resin.Resin is commented out, the leak is removed (assuming my HttpRequest._invocation patch is also applied) // Environment.addChildLoaderListener(new WebBeansAddLoaderListener()) Scott, you said the WebBeans caches were not to blame because they are classloader local. Well, it seems something about this classloader locality isn't working the way it should, doesn't it...? Do you have enough info to try to fix this in a future (hopefully soon to be released) 3.1 release? In the early work with 4.0, I'd cleaned a number of dead links (primarily WebBeans, but also some JMX). I believe the current 4.0 still has a few left, so I'll need to run a final pass at the end of the release cycle. -- Scott Maybe we are now seeing the end to all these memory leaks!!! /Mattias Mattias Jiderhamn wrote (2009-03-30 11:40): Just tried a new idea: I subclassed EnvironmentClassLoader within the Resin sources and compiled into resin.jar. There is still a leak (in contrast to when I subclassed EnvironmentClassLoader within my own application). Conclusion: The classloader that loads the (Environment)ClassLoader class is somehow of importance. /Mattias Mattias Jiderhamn wrote (2009-03-30 08:40): Daniel, your findings confirms EnvironmentClassLoader as the prime suspect (my own tests are a bit more detailed in an earlier post, http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html) Regarding your GC root paths for this class: 1. I haven't noticed any uses of java.util.prefs - could this be part of any of your added JARs (like groovy)? 2. This is the error I have found within Resin that is still to be fixed. As mentioned earlier, there is a quick and dirty patch for Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid this leak. However even with the patch applied I'm still having leaks. 3. This obviously seems like a groovy problem, probably a static initialization triggered at class load. So would you mind giving this another try with groovy removed and with a patched version of Resin? (I can provide a patched resin.jar if that helps) Cheers, Mattias Jiderhamn Daniel Lopez wrote (2009-03-29 14:52): Hi, Using that .war as a sample application, I did some more tests and found some things that might be interesting: I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, they simply live there. Important data: total size for WEB-INF/lib: 10.3 MB. I then deployed the application, start it and here's some data I gathered after accessing the application a couple of times and hitting "Force Garbage Collector" until the memory is stable: Heap Memory used: 31MB Instances of com.caucho.loader.EnvironmentClassLoader: 1 I then re-save web.xml so the context is reloaded, access and force GC until memory is stable again: Heap Memory used: 41MB Instances of com.caucho.loader.EnvironmentClassLoader: 2 Once again: Heap Memory used: 51MB Instances of com.caucho.loader.EnvironmentClassLoader: 3 Focusing on the last memory snapshot the 3 EnvironmentClassLoader come from (shortest GC roots): 1.- contextClassLoader of java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, Thread] 2.- _classLoader of com.caucho.server.dispatch.Invocation -> _invocation of com.caucho.server.http.HttpRequest [Stack Local] -> 3.- of org.codehaus.groovy.tools.shell.util.Preferences $1 -> of org.codehaus.groovy.tools.shell.util.Preferences $1 -> [1] of java.util.prefs.PreferenceChangeListener[3] -> prefListeners of java.util.prefs.WindowsPreferences -> STORE of org.codehaus.groovy.tools.shell.util.Preferences -> [771] of java.lang.Object[2560] -> elementData of java.util.Vector -> classes of com.caucho.loader.EnvironmentClassLoader -> contextClassLoader of java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, Thread] They are ordered using order of appearance (in the first memory SS, one like 1 is present, after the first restart one like the 2 is present). And looking at one of the suspected classes to be replicated, org.codehaus.groovy.tools.shell.util.Preferences, you can see that indeed there are 3 instances of that class, each one loaded by a different EnvironmentClassLoader. So, I would say there is indeed a leak and the jackpot would be to find out why the EnvironmentClassLoader instances are not being cleaned. Without knowing how Resin is internally supposed to work and which loaders ar
Re: [Resin-interest] Memory leak - in Resin?
After drawing the conclusion below, it isn't very far away to realize it probably has to do with some static (i.e. class loader specific) member of EnvironmentClassLoader. And just as I thought, the heart of the problem is private static EnvironmentLocal> _addLoaderListeners More specifically, if line 234 of com.caucho.server.resin.Resin is commented out, the leak is removed (assuming my HttpRequest._invocation patch is also applied) // Environment.addChildLoaderListener(new WebBeansAddLoaderListener()) Scott, you said the WebBeans caches were not to blame because they are classloader local. Well, it seems something about this classloader locality isn't working the way it should, doesn't it...? Do you have enough info to try to fix this in a future (hopefully soon to be released) 3.1 release? Maybe we are now seeing the end to all these memory leaks!!! /Mattias Mattias Jiderhamn wrote (2009-03-30 11:40): > Just tried a new idea: I subclassed EnvironmentClassLoader within the > Resin sources and compiled into resin.jar. There is still a leak (in > contrast to when I subclassed EnvironmentClassLoader within my own > application). > > Conclusion: The classloader that loads the (Environment)ClassLoader > class is somehow of importance. > > /Mattias > > Mattias Jiderhamn wrote (2009-03-30 08:40): >> Daniel, your findings confirms EnvironmentClassLoader as the prime >> suspect (my own tests are a bit more detailed in an earlier post, >> http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html) >> >> Regarding your GC root paths for this class: >> 1. I haven't noticed any uses of java.util.prefs - could this be part >> of any of your added JARs (like groovy)? >> 2. This is the error I have found within Resin that is still to be >> fixed. As mentioned earlier, there is a quick and dirty patch for >> Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid >> this leak. However even with the patch applied I'm still having leaks. >> 3. This obviously seems like a groovy problem, probably a static >> initialization triggered at class load. >> >> So would you mind giving this another try with groovy removed and >> with a patched version of Resin? (I can provide a patched resin.jar >> if that helps) >> >> Cheers, >> Mattias Jiderhamn >> >> Daniel Lopez wrote (2009-03-29 14:52): >>> Hi, >>> >>> Using that .war as a sample application, I did some more tests and >>> found some things that might be interesting: >>> >>> I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and >>> scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, >>> they simply live there. Important data: total size for WEB-INF/lib: >>> 10.3 MB. >>> >>> I then deployed the application, start it and here's some data I >>> gathered after accessing the application a couple of times and hitting >>> "Force Garbage Collector" until the memory is stable: >>> >>> Heap Memory used: 31MB >>> Instances of com.caucho.loader.EnvironmentClassLoader: 1 >>> >>> I then re-save web.xml so the context is reloaded, access and force GC >>> until memory is stable again: >>> >>> Heap Memory used: 41MB >>> Instances of com.caucho.loader.EnvironmentClassLoader: 2 >>> >>> Once again: >>> >>> Heap Memory used: 51MB >>> Instances of com.caucho.loader.EnvironmentClassLoader: 3 >>> >>> Focusing on the last memory snapshot the 3 EnvironmentClassLoader come >>> from (shortest GC roots): >>> 1.- contextClassLoader of >>> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, >>> Thread] >>> 2.- _classLoader of com.caucho.server.dispatch.Invocation -> >>> _invocation of com.caucho.server.http.HttpRequest [Stack Local] -> >>> 3.- of org.codehaus.groovy.tools.shell.util.Preferences$1 -> >>> of org.codehaus.groovy.tools.shell.util.Preferences$1 -> >>> [1] of java.util.prefs.PreferenceChangeListener[3] -> >>> prefListeners of java.util.prefs.WindowsPreferences -> >>> STORE of org.codehaus.groovy.tools.shell.util.Preferences -> >>> [771] of java.lang.Object[2560] -> >>> elementData of java.util.Vector -> >>> classes of com.caucho.loader.EnvironmentClassLoader -> >>> contextClassLoader of >>> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, >>> Thread] >>> >>> They are ordered using order of appearance (in the first memory SS, >>> one like 1 is present, after the first restart one like the 2 is >>> present). >>> >>> And looking at one of the suspected classes to be replicated, >>> org.codehaus.groovy.tools.shell.util.Preferences, you can see that >>> indeed there are 3 instances of that class, each one loaded by a >>> different EnvironmentClassLoader. >>> >>> So, I would say there is indeed a leak and the jackpot would be to >>> find out why the EnvironmentClassLoader instances are not being >>> cleaned. Without knowing how Resin is internally supposed to work and >>> which loaders are supposed not to be there, it'
Re: [Resin-interest] Memory leak - in Resin?
Just tried a new idea: I subclassed EnvironmentClassLoader within the Resin sources and compiled into resin.jar. There is still a leak (in contrast to when I subclassed EnvironmentClassLoader within my own application). Conclusion: The classloader that loads the (Environment)ClassLoader class is somehow of importance. /Mattias Mattias Jiderhamn wrote (2009-03-30 08:40): > Daniel, your findings confirms EnvironmentClassLoader as the prime > suspect (my own tests are a bit more detailed in an earlier post, > http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html) > > Regarding your GC root paths for this class: > 1. I haven't noticed any uses of java.util.prefs - could this be part > of any of your added JARs (like groovy)? > 2. This is the error I have found within Resin that is still to be > fixed. As mentioned earlier, there is a quick and dirty patch for > Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid > this leak. However even with the patch applied I'm still having leaks. > 3. This obviously seems like a groovy problem, probably a static > initialization triggered at class load. > > So would you mind giving this another try with groovy removed and with > a patched version of Resin? (I can provide a patched resin.jar if that > helps) > > Cheers, > Mattias Jiderhamn > > Daniel Lopez wrote (2009-03-29 14:52): >> Hi, >> >> Using that .war as a sample application, I did some more tests and >> found some things that might be interesting: >> >> I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and >> scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, >> they simply live there. Important data: total size for WEB-INF/lib: >> 10.3 MB. >> >> I then deployed the application, start it and here's some data I >> gathered after accessing the application a couple of times and hitting >> "Force Garbage Collector" until the memory is stable: >> >> Heap Memory used: 31MB >> Instances of com.caucho.loader.EnvironmentClassLoader: 1 >> >> I then re-save web.xml so the context is reloaded, access and force GC >> until memory is stable again: >> >> Heap Memory used: 41MB >> Instances of com.caucho.loader.EnvironmentClassLoader: 2 >> >> Once again: >> >> Heap Memory used: 51MB >> Instances of com.caucho.loader.EnvironmentClassLoader: 3 >> >> Focusing on the last memory snapshot the 3 EnvironmentClassLoader come >> from (shortest GC roots): >> 1.- contextClassLoader of >> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, >> Thread] >> 2.- _classLoader of com.caucho.server.dispatch.Invocation -> >> _invocation of com.caucho.server.http.HttpRequest [Stack Local] -> >> 3.- of org.codehaus.groovy.tools.shell.util.Preferences$1 -> >> of org.codehaus.groovy.tools.shell.util.Preferences$1 -> >> [1] of java.util.prefs.PreferenceChangeListener[3] -> >> prefListeners of java.util.prefs.WindowsPreferences -> >> STORE of org.codehaus.groovy.tools.shell.util.Preferences -> >> [771] of java.lang.Object[2560] -> >> elementData of java.util.Vector -> >> classes of com.caucho.loader.EnvironmentClassLoader -> >> contextClassLoader of >> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, >> Thread] >> >> They are ordered using order of appearance (in the first memory SS, >> one like 1 is present, after the first restart one like the 2 is >> present). >> >> And looking at one of the suspected classes to be replicated, >> org.codehaus.groovy.tools.shell.util.Preferences, you can see that >> indeed there are 3 instances of that class, each one loaded by a >> different EnvironmentClassLoader. >> >> So, I would say there is indeed a leak and the jackpot would be to >> find out why the EnvironmentClassLoader instances are not being >> cleaned. Without knowing how Resin is internally supposed to work and >> which loaders are supposed not to be there, it's difficult to know. As >> the classes inside the lib directory are not even used at all, one >> would suspect is a Resin internal issue, but seeing Preferences >> classes and ThreadLocal in the mix makes the whole thing quite >> suspicious :). >> >> Any idea or test to find out more? >> D. >> >> >>> Seems to me nothing has changed in this regard in the 4.0 snapshot. I >>> did my leak test as previously described, see below, and no classes were >>> unloaded. >>> >>> - Dowload http://jiderhamn.se/leak.war >>> - Add some JARs to the WEB-INF/lib directory; preferrably a couple of >>> large ones like spring.jar and hibernate.jar (don't use Resin JARs though). >>> - Drop the WAR in a clean installation of Resin >>> - Hit http://...:nn/leak (once is enough) >>> - Force a redeploy by either deleting the webapps/leak dir or touch:ing >>> leak.war >>> - Hit http://...:nn/leak again >>> - Repeat the last two steps for as long as you'd like >>> >>> /Mattias >>> >> >> > __
Re: [Resin-interest] Memory leak - in Resin?
Daniel, your findings confirms EnvironmentClassLoader as the prime suspect (my own tests are a bit more detailed in an earlier post, http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html) Regarding your GC root paths for this class: 1. I haven't noticed any uses of java.util.prefs - could this be part of any of your added JARs (like groovy)? 2. This is the error I have found within Resin that is still to be fixed. As mentioned earlier, there is a quick and dirty patch for Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid this leak. However even with the patch applied I'm still having leaks. 3. This obviously seems like a groovy problem, probably a static initialization triggered at class load. So would you mind giving this another try with groovy removed and with a patched version of Resin? (I can provide a patched resin.jar if that helps) Cheers, Mattias Jiderhamn Daniel Lopez wrote (2009-03-29 14:52): > Hi, > > Using that .war as a sample application, I did some more tests and > found some things that might be interesting: > > I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and > scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, > they simply live there. Important data: total size for WEB-INF/lib: > 10.3 MB. > > I then deployed the application, start it and here's some data I > gathered after accessing the application a couple of times and hitting > "Force Garbage Collector" until the memory is stable: > > Heap Memory used: 31MB > Instances of com.caucho.loader.EnvironmentClassLoader: 1 > > I then re-save web.xml so the context is reloaded, access and force GC > until memory is stable again: > > Heap Memory used: 41MB > Instances of com.caucho.loader.EnvironmentClassLoader: 2 > > Once again: > > Heap Memory used: 51MB > Instances of com.caucho.loader.EnvironmentClassLoader: 3 > > Focusing on the last memory snapshot the 3 EnvironmentClassLoader come > from (shortest GC roots): > 1.- contextClassLoader of > java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, > Thread] > 2.- _classLoader of com.caucho.server.dispatch.Invocation -> > _invocation of com.caucho.server.http.HttpRequest [Stack Local] -> > 3.- of org.codehaus.groovy.tools.shell.util.Preferences$1 -> > of org.codehaus.groovy.tools.shell.util.Preferences$1 -> > [1] of java.util.prefs.PreferenceChangeListener[3] -> > prefListeners of java.util.prefs.WindowsPreferences -> > STORE of org.codehaus.groovy.tools.shell.util.Preferences -> > [771] of java.lang.Object[2560] -> > elementData of java.util.Vector -> > classes of com.caucho.loader.EnvironmentClassLoader -> > contextClassLoader of > java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, > Thread] > > They are ordered using order of appearance (in the first memory SS, > one like 1 is present, after the first restart one like the 2 is > present). > > And looking at one of the suspected classes to be replicated, > org.codehaus.groovy.tools.shell.util.Preferences, you can see that > indeed there are 3 instances of that class, each one loaded by a > different EnvironmentClassLoader. > > So, I would say there is indeed a leak and the jackpot would be to > find out why the EnvironmentClassLoader instances are not being > cleaned. Without knowing how Resin is internally supposed to work and > which loaders are supposed not to be there, it's difficult to know. As > the classes inside the lib directory are not even used at all, one > would suspect is a Resin internal issue, but seeing Preferences > classes and ThreadLocal in the mix makes the whole thing quite > suspicious :). > > Any idea or test to find out more? > D. > > >> Seems to me nothing has changed in this regard in the 4.0 snapshot. I >> did my leak test as previously described, see below, and no classes were >> unloaded. >> >> - Dowload http://jiderhamn.se/leak.war >> - Add some JARs to the WEB-INF/lib directory; preferrably a couple of >> large ones like spring.jar and hibernate.jar (don't use Resin JARs though). >> - Drop the WAR in a clean installation of Resin >> - Hit http://...:nn/leak (once is enough) >> - Force a redeploy by either deleting the webapps/leak dir or touch:ing >> leak.war >> - Hit http://...:nn/leak again >> - Repeat the last two steps for as long as you'd like >> >> /Mattias >> > > -- ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Hi, Using that .war as a sample application, I did some more tests and found some things that might be interesting: I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them, they simply live there. Important data: total size for WEB-INF/lib: 10.3 MB. I then deployed the application, start it and here's some data I gathered after accessing the application a couple of times and hitting "Force Garbage Collector" until the memory is stable: Heap Memory used: 31MB Instances of com.caucho.loader.EnvironmentClassLoader: 1 I then re-save web.xml so the context is reloaded, access and force GC until memory is stable again: Heap Memory used: 41MB Instances of com.caucho.loader.EnvironmentClassLoader: 2 Once again: Heap Memory used: 51MB Instances of com.caucho.loader.EnvironmentClassLoader: 3 Focusing on the last memory snapshot the 3 EnvironmentClassLoader come from (shortest GC roots): 1.- contextClassLoader of java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, Thread] 2.- _classLoader of com.caucho.server.dispatch.Invocation -> _invocation of com.caucho.server.http.HttpRequest [Stack Local] -> 3.- of org.codehaus.groovy.tools.shell.util.Preferences$1 -> of org.codehaus.groovy.tools.shell.util.Preferences$1 -> [1] of java.util.prefs.PreferenceChangeListener[3] -> prefListeners of java.util.prefs.WindowsPreferences -> STORE of org.codehaus.groovy.tools.shell.util.Preferences -> [771] of java.lang.Object[2560] -> elementData of java.util.Vector -> classes of com.caucho.loader.EnvironmentClassLoader -> contextClassLoader of java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local, Thread] They are ordered using order of appearance (in the first memory SS, one like 1 is present, after the first restart one like the 2 is present). And looking at one of the suspected classes to be replicated, org.codehaus.groovy.tools.shell.util.Preferences, you can see that indeed there are 3 instances of that class, each one loaded by a different EnvironmentClassLoader. So, I would say there is indeed a leak and the jackpot would be to find out why the EnvironmentClassLoader instances are not being cleaned. Without knowing how Resin is internally supposed to work and which loaders are supposed not to be there, it's difficult to know. As the classes inside the lib directory are not even used at all, one would suspect is a Resin internal issue, but seeing Preferences classes and ThreadLocal in the mix makes the whole thing quite suspicious :). Any idea or test to find out more? D. > Seems to me nothing has changed in this regard in the 4.0 snapshot. I > did my leak test as previously described, see below, and no classes were > unloaded. > > - Dowload http://jiderhamn.se/leak.war > - Add some JARs to the WEB-INF/lib directory; preferrably a couple of > large ones like spring.jar and hibernate.jar (don't use Resin JARs though). > - Drop the WAR in a clean installation of Resin > - Hit http://...:nn/leak (once is enough) > - Force a redeploy by either deleting the webapps/leak dir or touch:ing > leak.war > - Hit http://...:nn/leak again > - Repeat the last two steps for as long as you'd like > > /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
I'm glad to see there's someone else with these problems. I've had problems with leaks since 3.0. I rely on resin to re-load my webapp every time I rebuild a class, and it will do that a few times before getting a PermGen exception, at which point I have to kill and restart resin. I'm on 3.2.1 now, but won't be moving to 4.0 for a bit yet. On Mar 25, 2009, at 07:35:32, Mattias Jiderhamn wrote: > Scott Ferguson wrote (2009-01-09 17:08): >> >> On Jan 9, 2009, at 12:22 AM, Mattias Jiderhamn wrote: >> >> >>> Scott Ferguson wrote (2008-11-26 16:53): >>> Thanks. Right now, our code base is a bit stuck due to the WebBeans/OSGi upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I can put up a snapshot I can take a look. >>> Scott Ferguson wrote (2008-12-29 20:41): >>> I've just made an early Resin 4.0 snapshot available. >>> Does this mean we can assume you will be able to look more closely >>> at >>> the memory leaks in 3.1 sometime soon...? >>> >> >> I was hoping you could verify the 4.0 memory fixes before looking at >> 3.1 Because several of those changes were more involved than I'm >> comfortable with for 3.1, I'd prefer to make sure they work in 4.0 >> first. >> > Seems to me nothing has changed in this regard in the 4.0 snapshot. > I did my leak test as previously described, see below, and no > classes were unloaded. > > - Dowload http://jiderhamn.se/leak.war > - Add some JARs to the WEB-INF/lib directory; preferrably a couple > of large ones like spring.jar and hibernate.jar (don't use Resin > JARs though). > - Drop the WAR in a clean installation of Resin > - Hit http://...:nn/leak (once is enough) > - Force a redeploy by either deleting the webapps/leak dir or > touch:ing leak.war > - Hit http://...:nn/leak again > - Repeat the last two steps for as long as you'd like > > /Mattias > > > > > > Now, here are the things really bugging me: > 1. If I keep redeploying over and over, I will eventually get > closer > and closer to the Perm Gen Max (in some instances, I have seen the > following behaivour instead turn up when Used reaches Init if Init > is > large enough). Then suddenly the unloadedClassCount is increased, > but > not with all the unused classes - only about the amount of one > redeployment. Redeploy again, and it will increase another step. > Meanwhile, the loadedClassCount remains pretty stable, since we > are > loading as many new classes as are unloaded. It's as if there > was a > FIFO queue/LRU cache of classloaders, so that the oldest one is > garbage collected once there is not enough space for a new one. > However, after a while there is (assumably) not enough space to > create the new classloader before the old one is garbage > collected, > and I get OutOfMemoryError somewhere in the middle. Sometimes I am > actually able to recover from this error by waiting for the GC > to do > it's job and then just try again. > > 2. Now I attached YourKit, looking for dangling classloaders as of > the inital post. I found none. In fact, the Classes without > Instances > inspection only shows the classes in the added JARs from the last > redeployment, so when tracing back to GC root, it goes via the > current EnvironmentClassLoader which is correct. There are also no > excessive instances of EnvironmentClassLoader. Hmm... Now wait a > minute. Look at the total number of java.lang.Class objects. It > does > not match with the totalLoadedClassCount. In fact, the total > number > of classes found by YourKit is about the same as the > totalLoadedClassCount on the very first hit of the application, > before any redeployments. So from YourKits point of view, there is > no > classloader leak! But why then isn't the PermGen space reclaimed. > This led me to wonder if there was some kind of JVM bug. (As a > side > note, I have yet to try with some other profiler) > > 3. So, I modified the test application (see commented out code in > MyServlet.java) to load the classes of the JARs in a regular > java.net.URLClassLoader which is then immediately thrown away. No > leak. loadedClassCount is immediately decreased (and > unloadedClassCount increased), as is the Used Perm Gen. That is, > it > behave the way we want the redeployment to behave. > Ok, lets take one step down in the classloader hierachy and load > all > the classes via a disposable com.caucho.loader.DynamicClassLoader. > No > leak. > So, what if I load them with a > com.caucho.loader.EnvironmentClassLoader which is then destroy()ed > and left for garbage collection. The leak is back! > EnvironmentClassLoader has now applied to become prime suspect. In > order to track things down I subclassed EnvironmentClassLoader >
Re: [Resin-interest] Memory leak - in Resin?
Scott Ferguson wrote (2009-01-09 17:08): > On Jan 9, 2009, at 12:22 AM, Mattias Jiderhamn wrote: > > >> Scott Ferguson wrote (2008-11-26 16:53): >> >>> Thanks. >>> >>> Right now, our code base is a bit stuck due to the WebBeans/OSGi >>> upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I >>> can put up a snapshot I can take a look. >>> >> Scott Ferguson wrote (2008-12-29 20:41): >> >>> I've just made an early Resin 4.0 snapshot available. >>> >> Does this mean we can assume you will be able to look more closely at >> the memory leaks in 3.1 sometime soon...? >> > > I was hoping you could verify the 4.0 memory fixes before looking at > 3.1 Because several of those changes were more involved than I'm > comfortable with for 3.1, I'd prefer to make sure they work in 4.0 > first. > Seems to me nothing has changed in this regard in the 4.0 snapshot. I did my leak test as previously described, see below, and no classes were unloaded. - Dowload http://jiderhamn.se/leak.war - Add some JARs to the WEB-INF/lib directory; preferrably a couple of large ones like spring.jar and hibernate.jar (don't use Resin JARs though). - Drop the WAR in a clean installation of Resin - Hit http://...:nn/leak (once is enough) - Force a redeploy by either deleting the webapps/leak dir or touch:ing leak.war - Hit http://...:nn/leak again - Repeat the last two steps for as long as you'd like /Mattias Now, here are the things really bugging me: 1. If I keep redeploying over and over, I will eventually get closer and closer to the Perm Gen Max (in some instances, I have seen the following behaivour instead turn up when Used reaches Init if Init is large enough). Then suddenly the unloadedClassCount is increased, but not with all the unused classes - only about the amount of one redeployment. Redeploy again, and it will increase another step. Meanwhile, the loadedClassCount remains pretty stable, since we are loading as many new classes as are unloaded. It's as if there was a FIFO queue/LRU cache of classloaders, so that the oldest one is garbage collected once there is not enough space for a new one. However, after a while there is (assumably) not enough space to create the new classloader before the old one is garbage collected, and I get OutOfMemoryError somewhere in the middle. Sometimes I am actually able to recover from this error by waiting for the GC to do it's job and then just try again. 2. Now I attached YourKit, looking for dangling classloaders as of the inital post. I found none. In fact, the Classes without Instances inspection only shows the classes in the added JARs from the last redeployment, so when tracing back to GC root, it goes via the current EnvironmentClassLoader which is correct. There are also no excessive instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at the total number of java.lang.Class objects. It does not match with the totalLoadedClassCount. In fact, the total number of classes found by YourKit is about the same as the totalLoadedClassCount on the very first hit of the application, before any redeployments. So from YourKits point of view, there is no classloader leak! But why then isn't the PermGen space reclaimed. This led me to wonder if there was some kind of JVM bug. (As a side note, I have yet to try with some other profiler) 3. So, I modified the test application (see commented out code in MyServlet.java) to load the classes of the JARs in a regular java.net.URLClassLoader which is then immediately thrown away. No leak. loadedClassCount is immediately decreased (and unloadedClassCount increased), as is the Used Perm Gen. That is, it behave the way we want the redeployment to behave. Ok, lets take one step down in the classloader hierachy and load all the classes via a disposable com.caucho.loader.DynamicClassLoader. No leak. So, what if I load them with a com.caucho.loader.EnvironmentClassLoader which is then destroy()ed and left for garbage collection. The leak is back! EnvironmentClassLoader has now applied to become prime suspect. In order to track things down I subclassed EnvironmentClassLoader inside my application and planned to make changes there. Well ... before any changes, the leak is nowhere to be found. What is going on here? Anything that might help my sanity would be appreciated. /Mattias Jiderhamn Mattias Jiderhamn wrote (2008-11-05 06:43): > In support of this latest theory is the fact that YourKit shows two > GC roots for the HttpRequest. Apart from the > com.caucho.server.port.TcpConnection._request reference, the > request > is a root i
Re: [Resin-interest] Memory leak - in Resin?
Bumping this and also bug 0002963, as we're still having memory issues in production that we can't trace back to our application, causing PermGen to build with every request and eventual OutOfMem/restart. Hoping issues in this thread can be resolved and pushed into the 3.1.8 trunk. Our app is a heavy hessian user. -tyson Scott Ferguson wrote: > > > On Jan 9, 2009, at 12:22 AM, Mattias Jiderhamn wrote: > >> Scott Ferguson wrote (2008-11-26 16:53): >>> >>> Thanks. >>> >>> Right now, our code base is a bit stuck due to the WebBeans/OSGi >>> upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I >>> can put up a snapshot I can take a look. >> >> Scott Ferguson wrote (2008-12-29 20:41): >>> I've just made an early Resin 4.0 snapshot available. >> >> Does this mean we can assume you will be able to look more closely at >> the memory leaks in 3.1 sometime soon...? > > I was hoping you could verify the 4.0 memory fixes before looking at > 3.1 Because several of those changes were more involved than I'm > comfortable with for 3.1, I'd prefer to make sure they work in 4.0 > first. > > -- Scott > >> >> >> /Mattias >> >> >> >>> >>> -- Scott Now, here are the things really bugging me: 1. If I keep redeploying over and over, I will eventually get closer and closer to the Perm Gen Max (in some instances, I have seen the following behaivour instead turn up when Used reaches Init if Init is large enough). Then suddenly the unloadedClassCount is increased, but not with all the unused classes - only about the amount of one redeployment. Redeploy again, and it will increase another step. Meanwhile, the loadedClassCount remains pretty stable, since we are loading as many new classes as are unloaded. It's as if there was a FIFO queue/LRU cache of classloaders, so that the oldest one is garbage collected once there is not enough space for a new one. However, after a while there is (assumably) not enough space to create the new classloader before the old one is garbage collected, and I get OutOfMemoryError somewhere in the middle. Sometimes I am actually able to recover from this error by waiting for the GC to do it's job and then just try again. 2. Now I attached YourKit, looking for dangling classloaders as of the inital post. I found none. In fact, the Classes without Instances inspection only shows the classes in the added JARs from the last redeployment, so when tracing back to GC root, it goes via the current EnvironmentClassLoader which is correct. There are also no excessive instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at the total number of java.lang.Class objects. It does not match with the totalLoadedClassCount. In fact, the total number of classes found by YourKit is about the same as the totalLoadedClassCount on the very first hit of the application, before any redeployments. So from YourKits point of view, there is no classloader leak! But why then isn't the PermGen space reclaimed. This led me to wonder if there was some kind of JVM bug. (As a side note, I have yet to try with some other profiler) 3. So, I modified the test application (see commented out code in MyServlet.java) to load the classes of the JARs in a regular java.net.URLClassLoader which is then immediately thrown away. No leak. loadedClassCount is immediately decreased (and unloadedClassCount increased), as is the Used Perm Gen. That is, it behave the way we want the redeployment to behave. Ok, lets take one step down in the classloader hierachy and load all the classes via a disposable com.caucho.loader.DynamicClassLoader. No leak. So, what if I load them with a com.caucho.loader.EnvironmentClassLoader which is then destroy()ed and left for garbage collection. The leak is back! EnvironmentClassLoader has now applied to become prime suspect. In order to track things down I subclassed EnvironmentClassLoader inside my application and planned to make changes there. Well ... before any changes, the leak is nowhere to be found. What is going on here? Anything that might help my sanity would be appreciated. /Mattias Jiderhamn Mattias Jiderhamn wrote (2008-11-05 06:43): > In support of this latest theory is the fact that YourKit shows two > GC roots for the HttpRequest. Apart from the > com.caucho.server.port.TcpConnection._request reference, the > request > is a root itself by being on the stack of a thread > (http--8080-...). > This could indicate a thread currently waiting in the > com.caucho.server.port.TcpConnection.run() method, where the > ServerRequest of the parent is also a local variable. > > Even if the requ
Re: [Resin-interest] Memory leak - in Resin?
On Jan 9, 2009, at 12:22 AM, Mattias Jiderhamn wrote: > Scott Ferguson wrote (2008-11-26 16:53): >> >> Thanks. >> >> Right now, our code base is a bit stuck due to the WebBeans/OSGi >> upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I >> can put up a snapshot I can take a look. > > Scott Ferguson wrote (2008-12-29 20:41): >> I've just made an early Resin 4.0 snapshot available. > > Does this mean we can assume you will be able to look more closely at > the memory leaks in 3.1 sometime soon...? I was hoping you could verify the 4.0 memory fixes before looking at 3.1 Because several of those changes were more involved than I'm comfortable with for 3.1, I'd prefer to make sure they work in 4.0 first. -- Scott > > > /Mattias > > > >> >> -- Scott >>> >>> >>> Now, here are the things really bugging me: >>> 1. If I keep redeploying over and over, I will eventually get closer >>> and closer to the Perm Gen Max (in some instances, I have seen the >>> following behaivour instead turn up when Used reaches Init if Init >>> is >>> large enough). Then suddenly the unloadedClassCount is increased, >>> but >>> not with all the unused classes - only about the amount of one >>> redeployment. Redeploy again, and it will increase another step. >>> Meanwhile, the loadedClassCount remains pretty stable, since we are >>> loading as many new classes as are unloaded. It's as if there was a >>> FIFO queue/LRU cache of classloaders, so that the oldest one is >>> garbage collected once there is not enough space for a new one. >>> However, after a while there is (assumably) not enough space to >>> create the new classloader before the old one is garbage collected, >>> and I get OutOfMemoryError somewhere in the middle. Sometimes I am >>> actually able to recover from this error by waiting for the GC to do >>> it's job and then just try again. >>> >>> 2. Now I attached YourKit, looking for dangling classloaders as of >>> the inital post. I found none. In fact, the Classes without >>> Instances >>> inspection only shows the classes in the added JARs from the last >>> redeployment, so when tracing back to GC root, it goes via the >>> current EnvironmentClassLoader which is correct. There are also no >>> excessive instances of EnvironmentClassLoader. Hmm... Now wait a >>> minute. Look at the total number of java.lang.Class objects. It does >>> not match with the totalLoadedClassCount. In fact, the total number >>> of classes found by YourKit is about the same as the >>> totalLoadedClassCount on the very first hit of the application, >>> before any redeployments. So from YourKits point of view, there is >>> no >>> classloader leak! But why then isn't the PermGen space reclaimed. >>> This led me to wonder if there was some kind of JVM bug. (As a side >>> note, I have yet to try with some other profiler) >>> >>> 3. So, I modified the test application (see commented out code in >>> MyServlet.java) to load the classes of the JARs in a regular >>> java.net.URLClassLoader which is then immediately thrown away. No >>> leak. loadedClassCount is immediately decreased (and >>> unloadedClassCount increased), as is the Used Perm Gen. That is, it >>> behave the way we want the redeployment to behave. >>> Ok, lets take one step down in the classloader hierachy and load all >>> the classes via a disposable com.caucho.loader.DynamicClassLoader. >>> No >>> leak. >>> So, what if I load them with a >>> com.caucho.loader.EnvironmentClassLoader which is then destroy()ed >>> and left for garbage collection. The leak is back! >>> EnvironmentClassLoader has now applied to become prime suspect. In >>> order to track things down I subclassed EnvironmentClassLoader >>> inside >>> my application and planned to make changes there. Well ... before >>> any >>> changes, the leak is nowhere to be found. >>> >>> What is going on here? >>> Anything that might help my sanity would be appreciated. >>> >>> /Mattias Jiderhamn >>> >>> >>> Mattias Jiderhamn wrote (2008-11-05 06:43): In support of this latest theory is the fact that YourKit shows two GC roots for the HttpRequest. Apart from the com.caucho.server.port.TcpConnection._request reference, the request is a root itself by being on the stack of a thread (http--8080-...). This could indicate a thread currently waiting in the com.caucho.server.port.TcpConnection.run() method, where the ServerRequest of the parent is also a local variable. Even if the request is reused, I believe the Invocation or the ClassLoader of the invocation needs to be reset somehow, to release the webapp classloader. I can help try out a proposed fix to see if it solves the problem (= feel free to mail me off list). I might even give it a shot myself. /Mattias >>> >> > > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com
Re: [Resin-interest] Memory leak - in Resin?
I trick that I use to trouble shoot perm gen memory leak involves using jconsole and , jmap, and jhat. I would attach to resin with jconsole, then reload the webapp a few times to trigger the perm gen leak. That I will stop the webapp completely, and then go to the memory tab to trigger some full GC. What I found is that I would need to trigger a few full GC back to back in order for the garbage collector to actually clean up the perm gen. At this point I take a heap dump with jmap and look at the heap dump with jhat. The webapp had been stopped and perm gen had been clean so if there is no leak, classes loaded by the webapp classlloader should not be in the heap dump anymore. So if you still see them it mean there is a perm gen leak. In jhat the detail page for any leaked classes has a link to the classloader which loaded that c lass. Follow that link and you will see a link for the reference chains from rootset (exclude weak ref) for the leaked classloader. That page will show you where the leak is. Bill On Fri, Jan 9, 2009 at 3:22 AM, Mattias Jiderhamn wrote: > Scott Ferguson wrote (2008-11-26 16:53): > > > > On Nov 24, 2008, at 11:08 PM, Mattias Jiderhamn wrote: > > > >> I'm still battling this PermGen leak and frankly I'm really starting > >> to doubt that I know what I'm doing anymore. I'd be very happy if > >> anyone would care to explain that to me... > >> > >> Since my last post Scott and I have discussed potential class loader > >> leaks and some of them have been fixed in the 3.1.8 release. It seems > >> there is (at least) one leak that didn't get fixed in 3.1.8. I have > >> made a quick and dirty patch to avoid that leak. If anyone would care > >> to try, the patch (which includes a few other things probably fixed > >> in 3.1.8 already) can be found here: > >> http://jiderhamn.se/resin-leak.patch > >> > >> However, even with that patch, it seems there is still some kind of > >> PermGen leak that eventually leads to OutOfMemoryError. I have > >> created a small application with the sole purpose of detecting these > >> leaks. If anyone would care to try, it can be found here (sources > >> included): http://jiderhamn.se/leak.war > >> You will need to add some JARs to the WEB-INF/lib directory; > >> preferrably a couple of large ones like spring.jar and hibernate.jar > >> (don't use Resin JARs though). > >> Then just drop the WAR in a clean installation of Resin 3.1.8 > >> (preferrably patched with the patch above). > >> Hit http://...:nn/leak (once is enough) > >> Force a redeploy by either deleting the webapps/leak dir or touch:ing > >> leak.war > >> Hit http://...:nn/leak again > >> Repeat the last two steps for as long as you'd like > >> > >> What you should see - or at least what I see on one Linux machine and > >> one Windows machine - is the (ClassLoadingMXBean) loadedClassCount > >> and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while > >> the unloadedClassCount remains pretty stable) for every redeploy, > >> which indicates a classloader leak. But I just can't find that leak. > > > > Thanks. > > > > Right now, our code base is a bit stuck due to the WebBeans/OSGi > > upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I > > can put up a snapshot I can take a look. > > Scott Ferguson wrote (2008-12-29 20:41): > > I've just made an early Resin 4.0 snapshot available. > > Does this mean we can assume you will be able to look more closely at > the memory leaks in 3.1 sometime soon...? > > /Mattias > > > > > > > -- Scott > >> > >> > >> Now, here are the things really bugging me: > >> 1. If I keep redeploying over and over, I will eventually get closer > >> and closer to the Perm Gen Max (in some instances, I have seen the > >> following behaivour instead turn up when Used reaches Init if Init is > >> large enough). Then suddenly the unloadedClassCount is increased, but > >> not with all the unused classes - only about the amount of one > >> redeployment. Redeploy again, and it will increase another step. > >> Meanwhile, the loadedClassCount remains pretty stable, since we are > >> loading as many new classes as are unloaded. It's as if there was a > >> FIFO queue/LRU cache of classloaders, so that the oldest one is > >> garbage collected once there is not enough space for a new one. > >> However, after a while there is (assumably) not enough space to > >> create the new classloader before the old one is garbage collected, > >> and I get OutOfMemoryError somewhere in the middle. Sometimes I am > >> actually able to recover from this error by waiting for the GC to do > >> it's job and then just try again. > >> > >> 2. Now I attached YourKit, looking for dangling classloaders as of > >> the inital post. I found none. In fact, the Classes without Instances > >> inspection only shows the classes in the added JARs from the last > >> redeployment, so when tracing back to GC root, it goes via the > >> current EnvironmentClassLoader which is correct. There are
Re: [Resin-interest] Memory leak - in Resin?
Scott Ferguson wrote (2008-11-26 16:53): > > On Nov 24, 2008, at 11:08 PM, Mattias Jiderhamn wrote: > >> I'm still battling this PermGen leak and frankly I'm really starting >> to doubt that I know what I'm doing anymore. I'd be very happy if >> anyone would care to explain that to me... >> >> Since my last post Scott and I have discussed potential class loader >> leaks and some of them have been fixed in the 3.1.8 release. It seems >> there is (at least) one leak that didn't get fixed in 3.1.8. I have >> made a quick and dirty patch to avoid that leak. If anyone would care >> to try, the patch (which includes a few other things probably fixed >> in 3.1.8 already) can be found here: >> http://jiderhamn.se/resin-leak.patch >> >> However, even with that patch, it seems there is still some kind of >> PermGen leak that eventually leads to OutOfMemoryError. I have >> created a small application with the sole purpose of detecting these >> leaks. If anyone would care to try, it can be found here (sources >> included): http://jiderhamn.se/leak.war >> You will need to add some JARs to the WEB-INF/lib directory; >> preferrably a couple of large ones like spring.jar and hibernate.jar >> (don't use Resin JARs though). >> Then just drop the WAR in a clean installation of Resin 3.1.8 >> (preferrably patched with the patch above). >> Hit http://...:nn/leak (once is enough) >> Force a redeploy by either deleting the webapps/leak dir or touch:ing >> leak.war >> Hit http://...:nn/leak again >> Repeat the last two steps for as long as you'd like >> >> What you should see - or at least what I see on one Linux machine and >> one Windows machine - is the (ClassLoadingMXBean) loadedClassCount >> and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while >> the unloadedClassCount remains pretty stable) for every redeploy, >> which indicates a classloader leak. But I just can't find that leak. > > Thanks. > > Right now, our code base is a bit stuck due to the WebBeans/OSGi > upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I > can put up a snapshot I can take a look. Scott Ferguson wrote (2008-12-29 20:41): > I've just made an early Resin 4.0 snapshot available. Does this mean we can assume you will be able to look more closely at the memory leaks in 3.1 sometime soon...? /Mattias > > -- Scott >> >> >> Now, here are the things really bugging me: >> 1. If I keep redeploying over and over, I will eventually get closer >> and closer to the Perm Gen Max (in some instances, I have seen the >> following behaivour instead turn up when Used reaches Init if Init is >> large enough). Then suddenly the unloadedClassCount is increased, but >> not with all the unused classes - only about the amount of one >> redeployment. Redeploy again, and it will increase another step. >> Meanwhile, the loadedClassCount remains pretty stable, since we are >> loading as many new classes as are unloaded. It's as if there was a >> FIFO queue/LRU cache of classloaders, so that the oldest one is >> garbage collected once there is not enough space for a new one. >> However, after a while there is (assumably) not enough space to >> create the new classloader before the old one is garbage collected, >> and I get OutOfMemoryError somewhere in the middle. Sometimes I am >> actually able to recover from this error by waiting for the GC to do >> it's job and then just try again. >> >> 2. Now I attached YourKit, looking for dangling classloaders as of >> the inital post. I found none. In fact, the Classes without Instances >> inspection only shows the classes in the added JARs from the last >> redeployment, so when tracing back to GC root, it goes via the >> current EnvironmentClassLoader which is correct. There are also no >> excessive instances of EnvironmentClassLoader. Hmm... Now wait a >> minute. Look at the total number of java.lang.Class objects. It does >> not match with the totalLoadedClassCount. In fact, the total number >> of classes found by YourKit is about the same as the >> totalLoadedClassCount on the very first hit of the application, >> before any redeployments. So from YourKits point of view, there is no >> classloader leak! But why then isn't the PermGen space reclaimed. >> This led me to wonder if there was some kind of JVM bug. (As a side >> note, I have yet to try with some other profiler) >> >> 3. So, I modified the test application (see commented out code in >> MyServlet.java) to load the classes of the JARs in a regular >> java.net.URLClassLoader which is then immediately thrown away. No >> leak. loadedClassCount is immediately decreased (and >> unloadedClassCount increased), as is the Used Perm Gen. That is, it >> behave the way we want the redeployment to behave. >> Ok, lets take one step down in the classloader hierachy and load all >> the classes via a disposable com.caucho.loader.DynamicClassLoader. No >> leak. >> So, what if I load them with a >> com.caucho.loader.EnvironmentClassLoader which is then d
Re: [Resin-interest] Memory leak - in Resin?
On Nov 24, 2008, at 11:08 PM, Mattias Jiderhamn wrote: I'm still battling this PermGen leak and frankly I'm really starting to doubt that I know what I'm doing anymore. I'd be very happy if anyone would care to explain that to me... Since my last post Scott and I have discussed potential class loader leaks and some of them have been fixed in the 3.1.8 release. It seems there is (at least) one leak that didn't get fixed in 3.1.8. I have made a quick and dirty patch to avoid that leak. If anyone would care to try, the patch (which includes a few other things probably fixed in 3.1.8 already) can be found here: http://jiderhamn.se/resin-leak.patch However, even with that patch, it seems there is still some kind of PermGen leak that eventually leads to OutOfMemoryError. I have created a small application with the sole purpose of detecting these leaks. If anyone would care to try, it can be found here (sources included): http://jiderhamn.se/leak.war You will need to add some JARs to the WEB-INF/lib directory; preferrably a couple of large ones like spring.jar and hibernate.jar (don't use Resin JARs though). Then just drop the WAR in a clean installation of Resin 3.1.8 (preferrably patched with the patch above). Hit http://...:nn/leak (once is enough) Force a redeploy by either deleting the webapps/leak dir or touch:ing leak.war Hit http://...:nn/leak again Repeat the last two steps for as long as you'd like What you should see - or at least what I see on one Linux machine and one Windows machine - is the (ClassLoadingMXBean) loadedClassCount and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while the unloadedClassCount remains pretty stable) for every redeploy, which indicates a classloader leak. But I just can't find that leak. Thanks. Right now, our code base is a bit stuck due to the WebBeans/OSGi upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I can put up a snapshot I can take a look. -- Scott Now, here are the things really bugging me: 1. If I keep redeploying over and over, I will eventually get closer and closer to the Perm Gen Max (in some instances, I have seen the following behaivour instead turn up when Used reaches Init if Init is large enough). Then suddenly the unloadedClassCount is increased, but not with all the unused classes - only about the amount of one redeployment. Redeploy again, and it will increase another step. Meanwhile, the loadedClassCount remains pretty stable, since we are loading as many new classes as are unloaded. It's as if there was a FIFO queue/LRU cache of classloaders, so that the oldest one is garbage collected once there is not enough space for a new one. However, after a while there is (assumably) not enough space to create the new classloader before the old one is garbage collected, and I get OutOfMemoryError somewhere in the middle. Sometimes I am actually able to recover from this error by waiting for the GC to do it's job and then just try again. 2. Now I attached YourKit, looking for dangling classloaders as of the inital post. I found none. In fact, the Classes without Instances inspection only shows the classes in the added JARs from the last redeployment, so when tracing back to GC root, it goes via the current EnvironmentClassLoader which is correct. There are also no excessive instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at the total number of java.lang.Class objects. It does not match with the totalLoadedClassCount. In fact, the total number of classes found by YourKit is about the same as the totalLoadedClassCount on the very first hit of the application, before any redeployments. So from YourKits point of view, there is no classloader leak! But why then isn't the PermGen space reclaimed. This led me to wonder if there was some kind of JVM bug. (As a side note, I have yet to try with some other profiler) 3. So, I modified the test application (see commented out code in MyServlet.java) to load the classes of the JARs in a regular java.net.URLClassLoader which is then immediately thrown away. No leak. loadedClassCount is immediately decreased (and unloadedClassCount increased), as is the Used Perm Gen. That is, it behave the way we want the redeployment to behave. Ok, lets take one step down in the classloader hierachy and load all the classes via a disposable com.caucho.loader.DynamicClassLoader. No leak. So, what if I load them with a com.caucho.loader.EnvironmentClassLoader which is then destroy()ed and left for garbage collection. The leak is back! EnvironmentClassLoader has now applied to become prime suspect. In order to track things down I subclassed EnvironmentClassLoader inside my application and planned to make changes there. Well ... before any changes, the leak is nowhere to be found. What is going on here? Anything that might help my sanity would
Re: [Resin-interest] Memory leak - in Resin?
I'm still battling this PermGen leak and frankly I'm really starting to doubt that I know what I'm doing anymore. I'd be very happy if anyone would care to explain that to me... Since my last post Scott and I have discussed potential class loader leaks and some of them have been fixed in the 3.1.8 release. It seems there is (at least) one leak that didn't get fixed in 3.1.8. I have made a quick and dirty patch to avoid that leak. If anyone would care to try, the patch (which includes a few other things probably fixed in 3.1.8 already) can be found here: http://jiderhamn.se/resin-leak.patch However, even with that patch, it seems there is still some kind of PermGen leak that eventually leads to OutOfMemoryError. I have created a small application with the sole purpose of detecting these leaks. If anyone would care to try, it can be found here (sources included): http://jiderhamn.se/leak.war You will need to add some JARs to the WEB-INF/lib directory; preferrably a couple of large ones like spring.jar and hibernate.jar (don't use Resin JARs though). Then just drop the WAR in a clean installation of Resin 3.1.8 (preferrably patched with the patch above). Hit http://...:nn/leak (once is enough) Force a redeploy by either deleting the webapps/leak dir or touch:ing leak.war Hit http://...:nn/leak again Repeat the last two steps for as long as you'd like What you should see - or at least what I see on one Linux machine and one Windows machine - is the (ClassLoadingMXBean) loadedClassCount and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while the unloadedClassCount remains pretty stable) for every redeploy, which indicates a classloader leak. But I just can't find that leak. Now, here are the things really bugging me: 1. If I keep redeploying over and over, I will eventually get closer and closer to the Perm Gen Max (in some instances, I have seen the following behaivour instead turn up when Used reaches Init if Init is large enough). Then suddenly the unloadedClassCount is increased, but not with all the unused classes - only about the amount of one redeployment. Redeploy again, and it will increase another step. Meanwhile, the loadedClassCount remains pretty stable, since we are loading as many new classes as are unloaded. It's as if there was a FIFO queue/LRU cache of classloaders, so that the oldest one is garbage collected once there is not enough space for a new one. However, after a while there is (assumably) not enough space to create the new classloader before the old one is garbage collected, and I get OutOfMemoryError somewhere in the middle. Sometimes I am actually able to recover from this error by waiting for the GC to do it's job and then just try again. 2. Now I attached YourKit, looking for dangling classloaders as of the inital post. I found none. In fact, the Classes without Instances inspection only shows the classes in the added JARs from the last redeployment, so when tracing back to GC root, it goes via the current EnvironmentClassLoader which is correct. There are also no excessive instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at the total number of java.lang.Class objects. It does not match with the totalLoadedClassCount. In fact, the total number of classes found by YourKit is about the same as the totalLoadedClassCount on the very first hit of the application, before any redeployments. So from YourKits point of view, there is no classloader leak! But why then isn't the PermGen space reclaimed. This led me to wonder if there was some kind of JVM bug. (As a side note, I have yet to try with some other profiler) 3. So, I modified the test application (see commented out code in MyServlet.java) to load the classes of the JARs in a regular java.net.URLClassLoader which is then immediately thrown away. No leak. loadedClassCount is immediately decreased (and unloadedClassCount increased), as is the Used Perm Gen. That is, it behave the way we want the redeployment to behave. Ok, lets take one step down in the classloader hierachy and load all the classes via a disposable com.caucho.loader.DynamicClassLoader. No leak. So, what if I load them with a com.caucho.loader.EnvironmentClassLoader which is then destroy()ed and left for garbage collection. The leak is back! EnvironmentClassLoader has now applied to become prime suspect. In order to track things down I subclassed EnvironmentClassLoader inside my application and planned to make changes there. Well ... before any changes, the leak is nowhere to be found. What is going on here? Anything that might help my sanity would be appreciated. /Mattias Jiderhamn Mattias Jiderhamn wrote (2008-11-05 06:43): > In support of this latest theory is the fact that YourKit shows two GC > roots for the HttpRequest. Apart from the > com.caucho.server.port.TcpConnection._request reference, the request > is a root itself by being on the stack of a thread (http--8080-...). > This could indicate a thread currently wai
Re: [Resin-interest] Memory leak - in Resin?
In support of this latest theory is the fact that YourKit shows two GC roots for the HttpRequest. Apart from the com.caucho.server.port.TcpConnection._request reference, the request is a root itself by being on the stack of a thread (http--8080-...). This could indicate a thread currently waiting in the com.caucho.server.port.TcpConnection.run() method, where the ServerRequest of the parent is also a local variable. Even if the request is reused, I believe the Invocation or the ClassLoader of the invocation needs to be reset somehow, to release the webapp classloader. I can help try out a proposed fix to see if it solves the problem (= feel free to mail me off list). I might even give it a shot myself. /Mattias Mattias Jiderhamn wrote (2008-11-04 23:28): > Scott Ferguson wrote (2008-11-04 23:16): > >> On Nov 4, 2008, at 2:10 PM, Mattias Jiderhamn wrote: >> >> >>> Scott wrote: >>> Did you build from resin/branches/3.1 or the trunk? >>> 3.1 branch. >>> >>> Now I'm browsing around the >>> com.caucho.server.port.TcpConnection._request which is never >>> explicitly release, but I realize it's getting a bit too late on this >>> side of the Atlantic ocean so I better continue tomorrow unless you >>> find the source of the problem until then. >>> >> That field is never released. The TcpConnection, Request and Response >> triple are reused together. >> > Wouldn't that be the problem then? > As long as there is one such triplet that has not been reused since the > webapp was reloaded, the request will point to (the Invocatoin that will > point to) the classloader of the old version of the app? > > (That could also explain why the problem has seemed to appear more often > on low load test sites than on production instances.) > > /Mattias > ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Scott Ferguson wrote (2008-11-04 23:16): > > On Nov 4, 2008, at 2:10 PM, Mattias Jiderhamn wrote: > >> Scott wrote: >>> Did you build from resin/branches/3.1 or the trunk? >> 3.1 branch. >> >> Now I'm browsing around the >> com.caucho.server.port.TcpConnection._request which is never >> explicitly release, but I realize it's getting a bit too late on this >> side of the Atlantic ocean so I better continue tomorrow unless you >> find the source of the problem until then. > > That field is never released. The TcpConnection, Request and Response > triple are reused together. Wouldn't that be the problem then? As long as there is one such triplet that has not been reused since the webapp was reloaded, the request will point to (the Invocatoin that will point to) the classloader of the old version of the app? (That could also explain why the problem has seemed to appear more often on low load test sites than on production instances.) /Mattias ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
On Nov 4, 2008, at 2:10 PM, Mattias Jiderhamn wrote: Scott wrote: Did you build from resin/branches/3.1 or the trunk? 3.1 branch. Now I'm browsing around the com.caucho.server.port.TcpConnection._request which is never explicitly release, but I realize it's getting a bit too late on this side of the Atlantic ocean so I better continue tomorrow unless you find the source of the problem until then. That field is never released. The TcpConnection, Request and Response triple are reused together. -- Scott /Mattias Mattias Jiderhamn wrote (2008-11-04 22:24): I noticed this bug was marked as resolved, so I tried reloading our webapp running on Resin compiled from SVN. Same problem. Familiarizing myself a bit further with YourKit and comparing with the sources, I believe both com.caucho.jca.UserTransactionProxy._threadTransaction and com.caucho.transaction.TransactionManagerImpl._threadTransaction should be cleared properly. Although I have not yet found the path to the HttpRequest. I could send you my YourKit snapshot if that helps in tracking this down. /Mattias Scott Ferguson wrote (2008-10-30 16:11): On Oct 30, 2008, at 6:39 AM, Mattias Jiderhamn wrote: Although embarrassed to admit it, we have had long standing problems with PermGen memory leaks. We have gotten used to restarting the server every time we redeploy, to avoid OutOfMemoryError. Since we would like to make use of - or at least evaluate - some of the new Resin features like WAR versioning, I thought I’d give fixing this another shot. I've added this as http://bugs.caucho.com/view.php?id=3031 That ThreadLocal needs to be cleared in a finally block when the request completes. I'll need to check on that. -- Scott ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Scott wrote: > Did you build from resin/branches/3.1 or the trunk? 3.1 branch. Now I'm browsing around the com.caucho.server.port.TcpConnection._request which is never explicitly release, but I realize it's getting a bit too late on this side of the Atlantic ocean so I better continue tomorrow unless you find the source of the problem until then. /Mattias Mattias Jiderhamn wrote (2008-11-04 22:24): > I noticed this bug was marked as resolved, so I tried reloading our > webapp running on Resin compiled from SVN. Same problem. > > Familiarizing myself a bit further with YourKit and comparing with the > sources, I believe both > com.caucho.jca.UserTransactionProxy._threadTransaction and > com.caucho.transaction.TransactionManagerImpl._threadTransaction > should be cleared properly. > Although I have not yet found the path to the HttpRequest. I could > send you my YourKit snapshot if that helps in tracking this down. > > /Mattias > > Scott Ferguson wrote (2008-10-30 16:11): >> On Oct 30, 2008, at 6:39 AM, Mattias Jiderhamn wrote: >> >> >>> Although embarrassed to admit it, we have had long standing problems >>> with PermGen memory leaks. We have gotten used to restarting the >>> server >>> every time we redeploy, to avoid OutOfMemoryError. Since we would like >>> to make use of - or at least evaluate - some of the new Resin features >>> like WAR versioning, I thought I’d give fixing this another shot. >>> >> >> I've added this as http://bugs.caucho.com/view.php?id=3031 >> >> That ThreadLocal needs to be cleared in a finally block when the >> request completes. I'll need to check on that. >> >> -- Scott >> ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
On Nov 4, 2008, at 1:24 PM, Mattias Jiderhamn wrote: I noticed this bug was marked as resolved, so I tried reloading our webapp running on Resin compiled from SVN. Same problem. Familiarizing myself a bit further with YourKit and comparing with the sources, I believe both com.caucho.jca.UserTransactionProxy._threadTransaction and com.caucho.transaction.TransactionManagerImpl._threadTransaction should be cleared properly. Although I have not yet found the path to the HttpRequest. I could send you my YourKit snapshot if that helps in tracking this down. Did you build from resin/branches/3.1 or the trunk? The trunk changes aren't checked in yet. The fix is only for the SecurityContext clearing. The comet changes had messed up the automatic clearing of that ThreadLocal. -- Scott /Mattias Scott Ferguson wrote (2008-10-30 16:11): On Oct 30, 2008, at 6:39 AM, Mattias Jiderhamn wrote: Although embarrassed to admit it, we have had long standing problems with PermGen memory leaks. We have gotten used to restarting the server every time we redeploy, to avoid OutOfMemoryError. Since we would like to make use of - or at least evaluate - some of the new Resin features like WAR versioning, I thought I’d give fixing this another shot. I've added this as http://bugs.caucho.com/view.php?id=3031 That ThreadLocal needs to be cleared in a finally block when the request completes. I'll need to check on that. -- Scott In com.caucho.server.dispatch.ServletInvocation there is a static ThreadLocal holding references to the current (original) request (while dispatching?). This means that as long as the executing (and dispatching) thread remains in Resins thread pool, but is not used for another dispatch, a handle to the original com.caucho.server.http.HttpRequest is held. In the request, there is a handle to a com.caucho.server.dispatch.Invocation, which in turn has a handle to - the application classloader! (And there is your PermGen leak) Isn’t the fact that the ThreadLocal is never cleared an obvious memory leak? Or why haven’t this been fixed even though Scott seems to have noticed this (see http://bugs.caucho.com/bug_view_advanced_page.php?bug_id=2883)? If the leak actually is in Resin, why isn’t anybody else seeing this??? Are we doing something unusual? (The only thing I could think of was ServletRequest implementations inside the application being dispatched, however I can't find any such implementations in use) /Mattias Jiderhamn ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
I noticed this bug was marked as resolved, so I tried reloading our webapp running on Resin compiled from SVN. Same problem. Familiarizing myself a bit further with YourKit and comparing with the sources, I believe both com.caucho.jca.UserTransactionProxy._threadTransaction and com.caucho.transaction.TransactionManagerImpl._threadTransaction should be cleared properly. Although I have not yet found the path to the HttpRequest. I could send you my YourKit snapshot if that helps in tracking this down. /Mattias Scott Ferguson wrote (2008-10-30 16:11): > On Oct 30, 2008, at 6:39 AM, Mattias Jiderhamn wrote: > > >> Although embarrassed to admit it, we have had long standing problems >> with PermGen memory leaks. We have gotten used to restarting the >> server >> every time we redeploy, to avoid OutOfMemoryError. Since we would like >> to make use of - or at least evaluate - some of the new Resin features >> like WAR versioning, I thought I’d give fixing this another shot. >> > > I've added this as http://bugs.caucho.com/view.php?id=3031 > > That ThreadLocal needs to be cleared in a finally block when the > request completes. I'll need to check on that. > > -- Scott > > >> >> In com.caucho.server.dispatch.ServletInvocation there is a static >> ThreadLocal holding references to the current (original) request >> (while >> dispatching?). This means that as long as the executing (and >> dispatching) thread remains in Resins thread pool, but is not used for >> another dispatch, a handle to the original >> com.caucho.server.http.HttpRequest is held. In the request, there is a >> handle to a com.caucho.server.dispatch.Invocation, which in turn has a >> handle to - the application classloader! (And there is your PermGen >> leak) >> >> Isn’t the fact that the ThreadLocal is never cleared an obvious memory >> leak? Or why haven’t this been fixed even though Scott seems to have >> noticed this (see >> http://bugs.caucho.com/bug_view_advanced_page.php?bug_id=2883)? >> If the leak actually is in Resin, why isn’t anybody else seeing >> this??? >> Are we doing something unusual? (The only thing I could think of was >> ServletRequest implementations inside the application being >> dispatched, >> however I can't find any such implementations in use) >> >> /Mattias Jiderhamn >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest >> > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest > > ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
On Oct 30, 2008, at 6:39 AM, Mattias Jiderhamn wrote: > Although embarrassed to admit it, we have had long standing problems > with PermGen memory leaks. We have gotten used to restarting the > server > every time we redeploy, to avoid OutOfMemoryError. Since we would like > to make use of - or at least evaluate - some of the new Resin features > like WAR versioning, I thought I’d give fixing this another shot. I've added this as http://bugs.caucho.com/view.php?id=3031 That ThreadLocal needs to be cleared in a finally block when the request completes. I'll need to check on that. -- Scott > > > I have already been through (and fixed a few of) the usual suspects; > JDBC drivers, log factories, ThreadLocals, explicit ClassLoader > references etc. There has also been at least one bug within Resin > causing leaks (bug 951 in Mantis). > > A while back I tried tracking this down using jhat, but since I have > limited experience with this, time did not allow me to get to the > bottom > of it. Today I found some more time, and also a tip on how to use > YourKit to track this down > (http://ampedandwired.com/2008/05/05/finding-permgen-memory-leaks-with-yourkit/ > for those interested). This lead me to something interesting, which I > don’t quite understand. > > If I interpret the YourKit memory snapshot correctly, here is what is > happening (from a Resin 3.1.6 perspective): > > In com.caucho.server.dispatch.ServletInvocation there is a static > ThreadLocal holding references to the current (original) request > (while > dispatching?). This means that as long as the executing (and > dispatching) thread remains in Resins thread pool, but is not used for > another dispatch, a handle to the original > com.caucho.server.http.HttpRequest is held. In the request, there is a > handle to a com.caucho.server.dispatch.Invocation, which in turn has a > handle to - the application classloader! (And there is your PermGen > leak) > > Isn’t the fact that the ThreadLocal is never cleared an obvious memory > leak? Or why haven’t this been fixed even though Scott seems to have > noticed this (see > http://bugs.caucho.com/bug_view_advanced_page.php?bug_id=2883)? > If the leak actually is in Resin, why isn’t anybody else seeing > this??? > Are we doing something unusual? (The only thing I could think of was > ServletRequest implementations inside the application being > dispatched, > however I can't find any such implementations in use) > > /Mattias Jiderhamn > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
We are also on REsin 3.1.6. We restart the server each deploy because of behavior exactly described below. Possibly unrelated but also strange issue we see is abnormally long re-start times, on a 2 server cluster where the process is: kill server 1 deploy war 1 restart server 1... wait long time for it to restart then repeat for server 2 On Thu, Oct 30, 2008 at 8:39 AM, Mattias Jiderhamn < [EMAIL PROTECTED]> wrote: > Although embarrassed to admit it, we have had long standing problems > with PermGen memory leaks. We have gotten used to restarting the server > every time we redeploy, to avoid OutOfMemoryError. Since we would like > to make use of - or at least evaluate - some of the new Resin features > like WAR versioning, I thought I'd give fixing this another shot. > > I have already been through (and fixed a few of) the usual suspects; > JDBC drivers, log factories, ThreadLocals, explicit ClassLoader > references etc. There has also been at least one bug within Resin > causing leaks (bug 951 in Mantis). > > A while back I tried tracking this down using jhat, but since I have > limited experience with this, time did not allow me to get to the bottom > of it. Today I found some more time, and also a tip on how to use > YourKit to track this down > ( > http://ampedandwired.com/2008/05/05/finding-permgen-memory-leaks-with-yourkit/ > for those interested). This lead me to something interesting, which I > don't quite understand. > > If I interpret the YourKit memory snapshot correctly, here is what is > happening (from a Resin 3.1.6 perspective): > > In com.caucho.server.dispatch.ServletInvocation there is a static > ThreadLocal holding references to the current (original) request (while > dispatching?). This means that as long as the executing (and > dispatching) thread remains in Resins thread pool, but is not used for > another dispatch, a handle to the original > com.caucho.server.http.HttpRequest is held. In the request, there is a > handle to a com.caucho.server.dispatch.Invocation, which in turn has a > handle to - the application classloader! (And there is your PermGen leak) > > Isn't the fact that the ThreadLocal is never cleared an obvious memory > leak? Or why haven't this been fixed even though Scott seems to have > noticed this (see > http://bugs.caucho.com/bug_view_advanced_page.php?bug_id=2883)? > If the leak actually is in Resin, why isn't anybody else seeing this??? > Are we doing something unusual? (The only thing I could think of was > ServletRequest implementations inside the application being dispatched, > however I can't find any such implementations in use) > > /Mattias Jiderhamn > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest > ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Memory leak - in Resin?
Do you have a lot of jsp's? We don't deploy war files, but I do notice that when we have any old jsp's laying around that have errors in them, resin attempts to recompile them every time it starts, and this slows down the start time considerably. So, if you have jsp's in your war, you might want to consider compiling them before you create your war file. Not sure if that's even possible, but I bet it would reduce your start times considerably, at the expense of increasing the size of your war file, of course. Rob On Oct 30, 2008, at 07:37, Sandeep Ghael wrote: We are also on REsin 3.1.6. We restart the server each deploy because of behavior exactly described below. Possibly unrelated but also strange issue we see is abnormally long re-start times, on a 2 server cluster where the process is: kill server 1 deploy war 1 restart server 1... wait long time for it to restart then repeat for server 2 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
[Resin-interest] Memory leak - in Resin?
I've got a similar experience, but not enough evidence to name a root cause. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
[Resin-interest] Memory leak - in Resin?
Although embarrassed to admit it, we have had long standing problems with PermGen memory leaks. We have gotten used to restarting the server every time we redeploy, to avoid OutOfMemoryError. Since we would like to make use of - or at least evaluate - some of the new Resin features like WAR versioning, I thought I’d give fixing this another shot. I have already been through (and fixed a few of) the usual suspects; JDBC drivers, log factories, ThreadLocals, explicit ClassLoader references etc. There has also been at least one bug within Resin causing leaks (bug 951 in Mantis). A while back I tried tracking this down using jhat, but since I have limited experience with this, time did not allow me to get to the bottom of it. Today I found some more time, and also a tip on how to use YourKit to track this down (http://ampedandwired.com/2008/05/05/finding-permgen-memory-leaks-with-yourkit/ for those interested). This lead me to something interesting, which I don’t quite understand. If I interpret the YourKit memory snapshot correctly, here is what is happening (from a Resin 3.1.6 perspective): In com.caucho.server.dispatch.ServletInvocation there is a static ThreadLocal holding references to the current (original) request (while dispatching?). This means that as long as the executing (and dispatching) thread remains in Resins thread pool, but is not used for another dispatch, a handle to the original com.caucho.server.http.HttpRequest is held. In the request, there is a handle to a com.caucho.server.dispatch.Invocation, which in turn has a handle to - the application classloader! (And there is your PermGen leak) Isn’t the fact that the ThreadLocal is never cleared an obvious memory leak? Or why haven’t this been fixed even though Scott seems to have noticed this (see http://bugs.caucho.com/bug_view_advanced_page.php?bug_id=2883)? If the leak actually is in Resin, why isn’t anybody else seeing this??? Are we doing something unusual? (The only thing I could think of was ServletRequest implementations inside the application being dispatched, however I can't find any such implementations in use) /Mattias Jiderhamn ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
