Just a comment on this, we found similar problems not too long ago.
Performance under load turned out to be limited by the synchronized
lock in EnvironmentClassloader. So our fancy new 8 core systems
suddenly were reduced to single thread performance. Some profiling
highlighted two major offenders.
- Hibernate's "classic" HQL query parser (which is a legacy thing, the
default Hibernate configuration no longer uses it).
- Beanshell. We had a lot of beanshell interpreters running in the system.
The common denominator for these two was that they do a lot of lookups
(Class.forName()) on names that are not actual class names. We ended
up (among other counter measures) writing our own class loader to cache
all sorts of class lookups, blatantly violating all sorts of class
loading rules.
We will probably never be able to reload our app cleanly, but performance
improved at least 10x.
-Knut
PS: Perhaps obvious, but be careful running the JVM in debug mode to enable
on-the-fly reloading of classes. Under load the debug JVM has a tendency
to lock up in bad ways (at least the latest incarnations of Sun 1.5 for
Linux).
Scott Ferguson wrote:
On Apr 1, 2008, at 11:26 PM, Don Willis wrote:
Hi all,
We have an application running on Resin 3.0.23. Lately it has been
slowing down to a crawl on a regular basis. Thread dumps when it's
slow show lots of threads (~30 to 100) waiting to lock the
EnvironmentClassLoader. Like this one:
If it's possible, can you try to reproduce this in 3.1.5? There are a
good number of changes to speed up the classloading, so it would be
useful to know if it's already resolved, or if you're seeing a new
issue.
-- Scott
"resin-tcp-connection-j2ee.confluence.atlassian.com:8080-843" daemon
prio=10 tid=0x00002aab391d1000 nid=0x22a7 waiting for monitor entry
[0x0000000050f13000..0x0000000050f15b90]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.caucho.loader.DynamicClassLoader.loadClass
(DynamicClassLoader.java:1044)
- waiting to lock <0x00002aaac72eaf18> (a
com.caucho.loader.EnvironmentClassLoader)
at com.caucho.loader.DynamicClassLoader.loadClass
(DynamicClassLoader.java:1021)
The causes for loading classes are wide and varied, and they are
succeeding. That is, in successive thread dumps the thread locking
the EnvironmentClassLoader changes, but many threads are still
waiting to lock it 20 seconds later. My belief is that the Class
Loading is just taking way too long, and we're doing too much of it.
While I do think the app is a bit demanding of the ClassLoader, I
will, predictably, assure you that we haven't encountered this
bottleneck on other App Servers (or even any other deployments to
Resin as far as I've heard.) Something that may be quite peculiar is
that the app does a lot of requests to load classes that aren't
there. These seem to be particularly slow.
Are there any likely environmental factors that would affect the
speed of the EnvironmentClassLoader?
Is there any way to remove the bottleneck on the
EnvironmentClassLoader?
We've lately changed the GC from concurrent to parallel and increased
the -Xmx on the app, for theoretically unrelated reasons. There have
been none of these lock-ups since that change. I'd really like to
put the -Xmx back down. Does anybody have a compelling argument why
having less memory would dramatically slow down Resin's class loading?
Cheers,
Don
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
