Bugs item #482875, was opened at 2001-11-17 12:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=376685&aid=482875&group_id=22866
Category: JBossServer Group: v2.4 (stable) Status: Open Resolution: Works For Me Priority: 7 Submitted By: Joel Boehland (jolby) Assigned to: Scott M Stark (starksm) Summary: OutOfMemory after many Deploy cycles Initial Comment: I have been seeing OutOfMemoryErrors in my server after doing several deploy/redeploy cycles of my application .ear file (which is around 3 megs in size) Basic info: *Hardware: AMD-K2-400MHz, 196MB RAM *OS: linux 2.2.14 *jdk: Blackdown-1.3.0-FCS, mixed mode *Server: JBoss-2.4.3_Jetty-3.1.3-1 *ServerTrace: I don't have one now, but I can provide one later if needed. *Steps to reproduce: Deploy/Redeploy .ear file many times. My .ear is around 3 megs, so it may be necessary to use a large one to trigger this effect. Other info *I have had this happen using native threads mode, and green threads mode. *"Jozsa Kristof" <[EMAIL PROTECTED]> has also reported this error on the jboss-user list. He was using IBM jdk 1.3.0 on linux, so this doesn't appear to be tied to any particular version of the jdk. --Joel ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-12-07 04:33 Message: Logged In: YES user_id=9459 The RemoteMethodInvocation leak also applies to EJB remote calls. Regards, Adrian ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-30 15:53 Message: Logged In: YES user_id=9459 Me again, I figured out what the "copy and add" means the cache doesn't have to be synchronized. Adds can get lost, but that doesn't matter. I've never seen this idiom before, nice! It doesn't scale very well for large deployments, but all the hit is at deployment time. Noticed the ENCFactory fix has been committed. Regards Adrian ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-28 02:32 Message: Logged In: YES user_id=9459 Apologies, My post yesterday was based on a wrong assumption, although the fix works. I got it into my head that the cache in RemoteMethodInvocation was keyed by className, it is actually keyed by class. The class on a redeploy is different (it comes from a different class loader) so there is no chance of reusing old hash values from a previous deployment. I think my brain needs a Garbage Collector;-) The only fix required is to remove the Method object from the "values" side of the WeakHashMap. The easiest way to do this is to key the HashMap created in getInterfaces(Class) by the method signature string returned by Method.toString(). So ignore the code in my previous post and use the following changes in RemoteMethodInvocation.java In getInterfaces(Class) -map.put(method, new Long(hash)); +map.put(method.toString(), new Long(hash)); In calculateHash(Method) -return ((Long)methodHashes.get(method)).longValue(); +return ((Long)methodHashes.get(method.toString ())).longValue(); The downside is that Method.toString() uses more memory than the "pointer" Method.this, so the cache will be larger than before when no redeploys are performed. On a related point, what is the point of the code marked // Copy and add I think this should just be hashMap.put(method.getDeclaringClass(), methodHashes); but after my woolly thinking yesterday I wouldn't trust myself to make toast at the moment. Regards, Adrian ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-27 02:48 Message: Logged In: YES user_id=9459 You can let go now;-), I found the rest of the problem! There is another a hard link cache in org.jboss.ejb.plugins.jrmp.interfaces.RemoteMethodInvocation This one is harder to spot. I think the code in method calculateHash is wrong for two reasons. 1) The cache (hashMap) is a WeakHashMap, but its "values" contain HashMaps that have hard links to the application's classes. The garbage collector can't remove these, the weak behaviour is in the "keys". 2) You can't rely on the garbage collector to remove old entries BEFORE the application is redeployed. The old calculated hash values may still be in the cache when JRMPContainerInvoke.init is run over the new bean. This means it uses the (wrong? not sure how important this is?) old hash values. NOTE: ejb.plugins.local.BaseLocalContainerInvoker almost certainly has the same problem. My solution is make the cache a normal HashMap and get JRMPContainerInvoker.destroy to remove its cached values, ensuring the new bean has to recalculate them. I think the changes should be: (I have no CVS access) In RemoteInvocation.java -static Map hashMap = new WeakHashMap(); +static Map hashMap = new HashMap(); public static long calculateHash(Method method) { Map methodHashes = (Map)hashMap.get( method.getDeclaringClass()); if (methodHashes == null) { methodHashes = getInterfaceHashes( method.getDeclaringClass()); hashMap.put(method.getDeclaringClass()), methodHashes); } return ((Long)methodHashes.get(method)).longValue(); } public static void uncacheHash(Method method) { hashMap.remove(method.getDeclaringClass()); } In JRMPContainerInvoker.java public void destroy() { Method[] methods = ((ContainerInvokerContainer)container) .getRemoteClass().getMethods(); for (int i = 0; i < methods.length(); i++) RemoteMethodInvocation.uncacheHash(methods[i]); methods = ((ContainerInvokerContainer)container) .getHomeClass().getMethods(); for (int i = 0; i < methods.length(); i++) RemoteMethodInvocation.uncacheHash(methods[i]); } PEFORMANCE NOTE: I HAVEN'T TESTED THIS It should be possible to check the class loader before uncaching. Only those loaded through the application's class loader need to be removed (I think?). i.e. public static void uncacheHash(Method method) { if (Thread.currentThread().getContextClassLoader() == method.getDeclaringClass().getClassLoader()) hashMap.remove(method.getDeclaringClass()); } This fix and the one for ENCFactory sorts the problem. Scott, I have been testing it on Windows XP using Sun 1.4.0- beta3 JDK on both JBoss 2.4.3 and 3.0.0alpha. On 2.4.3 I used -Xmx2049k, on 3.0.0 with Jetty running it was -Xmx4097k (spooky numbers, probably not) it always broke before 20 redeploys. Regards, Adrian ---------------------------------------------------------------------- Comment By: Scott M Stark (starksm) Date: 2001-11-26 18:37 Message: Logged In: YES user_id=175228 I can't reproduce this behavior on a RedHat 7.2(2.4.7-10) using the Sun 1.3.1_01 JDK or on a Win2ksp2 system running the Sun 1.3.1_01 JDK using the latest 2.4 branch code that will be released as 2.4.4. On the Win2k system I am running with JBoss under OptimizeIT using -Xms20m -Xmx24m and the heap stays at 20M and the heap usage ranges between 5M and 12M. I have redeployed the 14k bank.jar from the jbosstest suite 100 times and still see no hint of OutOfMemory problems. ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-25 07:04 Message: Logged In: YES user_id=9459 Hang on, there's more to it than this. I was testing it with the JMX Adapter and EJB management beans on V3. When I retested it with HelloWorld.jar I still got the OutOfMemory. ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-25 06:27 Message: Logged In: YES user_id=9459 Located the problem. org.jboss.naming.ENCFactory is holding a hard link to the classloader in a static Hashtable (encs). Changing encs to a WeakHashMap fixes the problem. This works because when nobody is using the classloader (i.e. all classes unloaded and other hardlinks such as the web server are removed) it will be removed from the WeakHashMap by the garbage collector. Two points. 1) WeakHashMap was introduced in 1.2, is it ok to use it in JBoss? 2) I don't have CVS access to fix this, could somebody look at it and apply the fix? Regards Adrian ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2001-11-19 07:55 Message: Logged In: YES user_id=9459 I can reproduce this with a 4k HelloWorld.jar using the -Xmx parameter to constrain the heap size of the JVM. It seems the classloaders created by J2EEDeployer are not being garbage collected. Here's a hack to monitor garbage collection from org.jboss.Main. You call org.jboss.Main.monitorGarbage(Object,String) to request monitoring and the list of uncollected stuff gets printed once a second. System.out.println("JBoss "+versionIdentifier+" Started in "+minutes+"m:"+seconds+"s"); <hack> try { while (true) { // Wait for garbage collection java.lang.ref.Reference collected = garbageQueue.remove(1000); while (collected !=null) { // Remove all available garbage garbage.remove(collected); collected = garbageQueue.remove(1); } // List uncollected garbage Iterator uncollected = garbage.values().iterator(); System.out.println("Uncollected items: " + garbage.size()); while (uncollected.hasNext()) System.out.println(uncollected.next()); System.gc(); } }catch (InterruptedException ignored){}; </hack> } <resume-hack> // Request garbage collection monitoring public static void monitorGarbage(Object object, String info) { garbage.put( new java.lang.ref.WeakReference(object,garbageQueue), info); System.out.println("MonitorGarbage: " + info); } // Garbage to monitor public static HashMap garbage = new HashMap(); // Garbage collected queue public static java.lang.ref.ReferenceQueue garbageQueue = new java.lang.ref.ReferenceQueue(); </resume-hack> Then I changed org.jboss.deployment.J2EEDeployer to monitor the garbage collection of the classloader // set the context classloader for this application createContextClassLoader(_d); // save the application classloader for later ClassLoader appCl = Thread.currentThread().getContextClassLoader(); <new-code> org.jboss.Main.monitorGarbage(appCl, "ClassLoader " + _d.getName()); </new-code> If you go through the cycle of deploying and undeploying a few times you'll notice the the classloaders are never garbage collected. There must be a class lying around somewhere that needs the classloader????? ---------------------------------------------------------------------- Comment By: Jim Cook (oravecz) Date: 2001-11-17 14:42 Message: Logged In: YES user_id=6215 Ditto, on Windows 2000 as well. Redeploy of a 700K WAR file causes an Out of Memory after 15-20 redeploys. Would there be any clues as to whether the WAR file contents are actually guilty of some kind of memory leak? Using WebWork and Velocity in the WAR... -jim ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=376685&aid=482875&group_id=22866 _______________________________________________ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development