Re: [Resin-interest] Locking the EnvironmentClassLoader
Hi Scott and Daniel, Thanks for the prompt replies, and sorry for my slow one. It might be possible to try 3.1.5 in the future. Currently, since we upped the memory and changed the collector the app is behaving extremely well, so I don't feel an urgent need to look into it. We'll definitely take on board your advice about 3.1.5 and future versions having improvements to the speed and sychronisation-scope of class loading. Daniel's suggestion that there may have just been too many garbage collections going on makes sense to me, but the GC logs don't really show enough GC going on to make this seem like the complete solution. I hope to do some decent load testing of these settings in isolation in the future. Thanks again for your advice, Don On 03/04/2008, at 5:07 AM, Scott Ferguson wrote: > > On Apr 1, 2008, at 11:26 PM, Don Willis wrote: > >> Hi all, >> >> We have an application running on Resin 3.0.23. Lately it has been >> slowing down to a crawl on a regular basis. Thread dumps when it's >> slow show lots of threads (~30 to 100) waiting to lock the >> EnvironmentClassLoader. Like this one: > > If it's possible, can you try to reproduce this in 3.1.5? There are a > good number of changes to speed up the classloading, so it would be > useful to know if it's already resolved, or if you're seeing a new > issue. > > -- Scott > >> >> >> "resin-tcp-connection-j2ee.confluence.atlassian.com:8080-843" daemon >> prio=10 tid=0x2aab391d1000 nid=0x22a7 waiting for monitor entry >> [0x50f13000..0x50f15b90] >>java.lang.Thread.State: BLOCKED (on object monitor) >> at com.caucho.loader.DynamicClassLoader.loadClass >> (DynamicClassLoader.java:1044) >> - waiting to lock <0x2aaac72eaf18> (a >> com.caucho.loader.EnvironmentClassLoader) >> at com.caucho.loader.DynamicClassLoader.loadClass >> (DynamicClassLoader.java:1021) >> >> >> The causes for loading classes are wide and varied, and they are >> succeeding. That is, in successive thread dumps the thread locking >> the EnvironmentClassLoader changes, but many threads are still >> waiting to lock it 20 seconds later. My belief is that the Class >> Loading is just taking way too long, and we're doing too much of it. >> While I do think the app is a bit demanding of the ClassLoader, I >> will, predictably, assure you that we haven't encountered this >> bottleneck on other App Servers (or even any other deployments to >> Resin as far as I've heard.) Something that may be quite peculiar is >> that the app does a lot of requests to load classes that aren't >> there. These seem to be particularly slow. >> >> Are there any likely environmental factors that would affect the >> speed of the EnvironmentClassLoader? >> Is there any way to remove the bottleneck on the >> EnvironmentClassLoader? >> >> We've lately changed the GC from concurrent to parallel and increased >> the -Xmx on the app, for theoretically unrelated reasons. There have >> been none of these lock-ups since that change. I'd really like to >> put the -Xmx back down. Does anybody have a compelling argument why >> having less memory would dramatically slow down Resin's class >> loading? >> >> Cheers, >> Don >> >> >> ___ >> resin-interest mailing list >> resin-interest@caucho.com >> http://maillist.caucho.com/mailman/listinfo/resin-interest > > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Locking the EnvironmentClassLoader
On Apr 2, 2008, at 2:25 PM, Knut Forkalsrud wrote: > This was with version 3.1.4, I haven't tried 3.1.5 yet. 3.1.5 is interesting, since it should handle the null classes faster (basically, 3.1.5 pre-analyzes all the jars). The locking in 3.1.5 is the same as 3.1.4. I've updated the locking for the next snapshot, since it was a little too broad. It would be interesting to see if there's a performance difference for your app. -- Scott > > > -Knut > > > > Knut Forkalsrud wrote: >> >> Just a comment on this, we found similar problems not too long ago. >> Performance under load turned out to be limited by the synchronized >> lock in EnvironmentClassloader. So our fancy new 8 core systems >> suddenly were reduced to single thread performance. Some profiling >> highlighted two major offenders. >> >> - Hibernate's "classic" HQL query parser (which is a legacy thing, >> the >>default Hibernate configuration no longer uses it). >> - Beanshell. We had a lot of beanshell interpreters running in >> the system. >> >> The common denominator for these two was that they do a lot of >> lookups >> (Class.forName()) on names that are not actual class names. We ended >> up (among other counter measures) writing our own class loader to >> cache >> all sorts of class lookups, blatantly violating all sorts of class >> loading rules. >> We will probably never be able to reload our app cleanly, but >> performance >> improved at least 10x. >> >> -Knut > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Locking the EnvironmentClassLoader
This was with version 3.1.4, I haven't tried 3.1.5 yet. -Knut Knut Forkalsrud wrote: Just a comment on this, we found similar problems not too long ago. Performance under load turned out to be limited by the synchronized lock in EnvironmentClassloader. So our fancy new 8 core systems suddenly were reduced to single thread performance. Some profiling highlighted two major offenders. - Hibernate's "classic" HQL query parser (which is a legacy thing, the default Hibernate configuration no longer uses it). - Beanshell. We had a lot of beanshell interpreters running in the system. The common denominator for these two was that they do a lot of lookups (Class.forName()) on names that are not actual class names. We ended up (among other counter measures) writing our own class loader to cache all sorts of class lookups, blatantly violating all sorts of class loading rules. We will probably never be able to reload our app cleanly, but performance improved at least 10x. -Knut ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Locking the EnvironmentClassLoader
Just a comment on this, we found similar problems not too long ago. Performance under load turned out to be limited by the synchronized lock in EnvironmentClassloader. So our fancy new 8 core systems suddenly were reduced to single thread performance. Some profiling highlighted two major offenders. - Hibernate's "classic" HQL query parser (which is a legacy thing, the default Hibernate configuration no longer uses it). - Beanshell. We had a lot of beanshell interpreters running in the system. The common denominator for these two was that they do a lot of lookups (Class.forName()) on names that are not actual class names. We ended up (among other counter measures) writing our own class loader to cache all sorts of class lookups, blatantly violating all sorts of class loading rules. We will probably never be able to reload our app cleanly, but performance improved at least 10x. -Knut PS: Perhaps obvious, but be careful running the JVM in debug mode to enable on-the-fly reloading of classes. Under load the debug JVM has a tendency to lock up in bad ways (at least the latest incarnations of Sun 1.5 for Linux). Scott Ferguson wrote: On Apr 1, 2008, at 11:26 PM, Don Willis wrote: Hi all, We have an application running on Resin 3.0.23. Lately it has been slowing down to a crawl on a regular basis. Thread dumps when it's slow show lots of threads (~30 to 100) waiting to lock the EnvironmentClassLoader. Like this one: If it's possible, can you try to reproduce this in 3.1.5? There are a good number of changes to speed up the classloading, so it would be useful to know if it's already resolved, or if you're seeing a new issue. -- Scott "resin-tcp-connection-j2ee.confluence.atlassian.com:8080-843" daemon prio=10 tid=0x2aab391d1000 nid=0x22a7 waiting for monitor entry [0x50f13000..0x50f15b90] java.lang.Thread.State: BLOCKED (on object monitor) at com.caucho.loader.DynamicClassLoader.loadClass (DynamicClassLoader.java:1044) - waiting to lock <0x2aaac72eaf18> (a com.caucho.loader.EnvironmentClassLoader) at com.caucho.loader.DynamicClassLoader.loadClass (DynamicClassLoader.java:1021) The causes for loading classes are wide and varied, and they are succeeding. That is, in successive thread dumps the thread locking the EnvironmentClassLoader changes, but many threads are still waiting to lock it 20 seconds later. My belief is that the Class Loading is just taking way too long, and we're doing too much of it. While I do think the app is a bit demanding of the ClassLoader, I will, predictably, assure you that we haven't encountered this bottleneck on other App Servers (or even any other deployments to Resin as far as I've heard.) Something that may be quite peculiar is that the app does a lot of requests to load classes that aren't there. These seem to be particularly slow. Are there any likely environmental factors that would affect the speed of the EnvironmentClassLoader? Is there any way to remove the bottleneck on the EnvironmentClassLoader? We've lately changed the GC from concurrent to parallel and increased the -Xmx on the app, for theoretically unrelated reasons. There have been none of these lock-ups since that change. I'd really like to put the -Xmx back down. Does anybody have a compelling argument why having less memory would dramatically slow down Resin's class loading? Cheers, Don ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Locking the EnvironmentClassLoader
On Apr 1, 2008, at 11:26 PM, Don Willis wrote: > Hi all, > > We have an application running on Resin 3.0.23. Lately it has been > slowing down to a crawl on a regular basis. Thread dumps when it's > slow show lots of threads (~30 to 100) waiting to lock the > EnvironmentClassLoader. Like this one: If it's possible, can you try to reproduce this in 3.1.5? There are a good number of changes to speed up the classloading, so it would be useful to know if it's already resolved, or if you're seeing a new issue. -- Scott > > > "resin-tcp-connection-j2ee.confluence.atlassian.com:8080-843" daemon > prio=10 tid=0x2aab391d1000 nid=0x22a7 waiting for monitor entry > [0x50f13000..0x50f15b90] >java.lang.Thread.State: BLOCKED (on object monitor) > at com.caucho.loader.DynamicClassLoader.loadClass > (DynamicClassLoader.java:1044) > - waiting to lock <0x2aaac72eaf18> (a > com.caucho.loader.EnvironmentClassLoader) > at com.caucho.loader.DynamicClassLoader.loadClass > (DynamicClassLoader.java:1021) > > > The causes for loading classes are wide and varied, and they are > succeeding. That is, in successive thread dumps the thread locking > the EnvironmentClassLoader changes, but many threads are still > waiting to lock it 20 seconds later. My belief is that the Class > Loading is just taking way too long, and we're doing too much of it. > While I do think the app is a bit demanding of the ClassLoader, I > will, predictably, assure you that we haven't encountered this > bottleneck on other App Servers (or even any other deployments to > Resin as far as I've heard.) Something that may be quite peculiar is > that the app does a lot of requests to load classes that aren't > there. These seem to be particularly slow. > > Are there any likely environmental factors that would affect the > speed of the EnvironmentClassLoader? > Is there any way to remove the bottleneck on the > EnvironmentClassLoader? > > We've lately changed the GC from concurrent to parallel and increased > the -Xmx on the app, for theoretically unrelated reasons. There have > been none of these lock-ups since that change. I'd really like to > put the -Xmx back down. Does anybody have a compelling argument why > having less memory would dramatically slow down Resin's class loading? > > Cheers, > Don > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Locking the EnvironmentClassLoader
Don Willis escribió: > ...Does anybody have a compelling argument why > having less memory would dramatically slow down Resin's class loading? Hi, Te only general reason I can think would be that the lack of memory is causing the GC to fire too often, so you have a situation where the JVM reserves some memory for the newly generated classes, GC fires and frees some, then another requests come and there's still not enough, but some other classes are not in use already, so the GC fires again... so this "memory thrashing" causes the GC overhead/pauses to slow down the application. When you have more memory, when the GC fires more memory can be freed at once hence it does not have to run for a while. Even if you don't have memory leaks, that can happen to some applications if they consume/release memory quite often. It's not a common case and might not be your case, but that's one of the reasons why increasing some memory can make wonders to an, otherwise healthy, application. S! D. --- Daniel Lopez Janariz ([EMAIL PROTECTED]) Web Services Centre for Information and Technology Balearic Islands University (SPAIN) --- ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
