Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
On 21/10/2011 03:00, Bob DeRemer wrote: While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. If that were the case, the Tomcat devs wouldn't have put the investment they have into the Windows installer in recent months (auto detection of 32 or 64 bit JRE, greater control over ports, multiple service installs of the same version, ...) With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. Maybe. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. You need to understand the root cause. I'd recommend getting a profiler (I use YourKit since they give free copies to Tomcat committers) and finding out exactly what is holding references to those JARs. Once you know that the next questions are 1. What creates those references and 2. How to get rid of them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
Hi Mark, With regard to using Tomcat in production, that was meant sarcastically - the team has done a nice job on later Tomcat 6 and Tomcat 7 32/64-bit installations - including Windows Service support, etc. - nice work. With regard to using YourKit, I have a licensed copy, but must be missing something - where can I look to see what is holding a reference to the JAR? I've used it for memory analysis and CPU profiling - primarily, but I'm not sure where to look for JAR references. Thanks, Bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 2:46 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 03:00, Bob DeRemer wrote: While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. If that were the case, the Tomcat devs wouldn't have put the investment they have into the Windows installer in recent months (auto detection of 32 or 64 bit JRE, greater control over ports, multiple service installs of the same version, ...) With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. Maybe. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. You need to understand the root cause. I'd recommend getting a profiler (I use YourKit since they give free copies to Tomcat committers) and finding out exactly what is holding references to those JARs. Once you know that the next questions are 1. What creates those references and 2. How to get rid of them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
On 21/10/2011 12:22, Bob DeRemer wrote: Hi Mark, With regard to using Tomcat in production, that was meant sarcastically - the team has done a nice job on later Tomcat 6 and Tomcat 7 32/64-bit installations - including Windows Service support, etc. - nice work. That comment was more aimed at the archives than you ;) With regard to using YourKit, I have a licensed copy, but must be missing something - where can I look to see what is holding a reference to the JAR? I've used it for memory analysis and CPU profiling - primarily, but I'm not sure where to look for JAR references. File locks can be tricky to pin down. I usually do something along the lines of: 1. Take a memory snapshot 2. Switch to the object view 3. Search for the name of the JAR 4. Trace the GC roots of a promising looking object Normally, the JAR name appears in a String for a JarFile or File object and that traces back to whatever is holding the reference. I'd be happy to take a look at the snapshot for you if you can put the snapshot somewhere where I can download it. Mark Thanks, Bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 2:46 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 03:00, Bob DeRemer wrote: While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. If that were the case, the Tomcat devs wouldn't have put the investment they have into the Windows installer in recent months (auto detection of 32 or 64 bit JRE, greater control over ports, multiple service installs of the same version, ...) With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. Maybe. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. You need to understand the root cause. I'd recommend getting a profiler (I use YourKit since they give free copies to Tomcat committers) and finding out exactly what is holding references to those JARs. Once you know that the next questions are 1. What creates those references and 2. How to get rid of them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
Thanks Mark, I appreciate the offer. Let me dig into a bit today - based on your steps. if I can't get anywhere, I'll gladly take you up on your offer. What is the best way to communicate out-of-band if I need to get you a snapshot? -bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 7:42 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 12:22, Bob DeRemer wrote: Hi Mark, With regard to using Tomcat in production, that was meant sarcastically - the team has done a nice job on later Tomcat 6 and Tomcat 7 32/64-bit installations - including Windows Service support, etc. - nice work. That comment was more aimed at the archives than you ;) With regard to using YourKit, I have a licensed copy, but must be missing something - where can I look to see what is holding a reference to the JAR? I've used it for memory analysis and CPU profiling - primarily, but I'm not sure where to look for JAR references. File locks can be tricky to pin down. I usually do something along the lines of: 1. Take a memory snapshot 2. Switch to the object view 3. Search for the name of the JAR 4. Trace the GC roots of a promising looking object Normally, the JAR name appears in a String for a JarFile or File object and that traces back to whatever is holding the reference. I'd be happy to take a look at the snapshot for you if you can put the snapshot somewhere where I can download it. Mark Thanks, Bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 2:46 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 03:00, Bob DeRemer wrote: While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. If that were the case, the Tomcat devs wouldn't have put the investment they have into the Windows installer in recent months (auto detection of 32 or 64 bit JRE, greater control over ports, multiple service installs of the same version, ...) With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. Maybe. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. You need to understand the root cause. I'd recommend getting a profiler (I use YourKit since they give free copies to Tomcat committers) and finding out exactly what is holding references to those JARs. Once you know that the next questions are 1. What creates those references and 2. How to get rid of them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
On 21/10/2011 13:12, Bob DeRemer wrote: Thanks Mark, I appreciate the offer. Let me dig into a bit today - based on your steps. if I can't get anywhere, I'll gladly take you up on your offer. What is the best way to communicate out-of-band if I need to get you a snapshot? Bob and I took at look at this off-list using the standard approach for finding web-app reload memory leaks [1] and found what appears to be an issue with Tomcat's memory leak protection for the http keep-alive thread. Tomcat only clears the WebappClassLoader from the http keep-alive thread if the thread is still running. It should clear it in all circumstances. Ill be updating Tomcat to address this shortly. Mark [1] http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf -bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 7:42 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 12:22, Bob DeRemer wrote: Hi Mark, With regard to using Tomcat in production, that was meant sarcastically - the team has done a nice job on later Tomcat 6 and Tomcat 7 32/64-bit installations - including Windows Service support, etc. - nice work. That comment was more aimed at the archives than you ;) With regard to using YourKit, I have a licensed copy, but must be missing something - where can I look to see what is holding a reference to the JAR? I've used it for memory analysis and CPU profiling - primarily, but I'm not sure where to look for JAR references. File locks can be tricky to pin down. I usually do something along the lines of: 1. Take a memory snapshot 2. Switch to the object view 3. Search for the name of the JAR 4. Trace the GC roots of a promising looking object Normally, the JAR name appears in a String for a JarFile or File object and that traces back to whatever is holding the reference. I'd be happy to take a look at the snapshot for you if you can put the snapshot somewhere where I can download it. Mark Thanks, Bob -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Friday, October 21, 2011 2:46 AM To: Tomcat Users List Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp On 21/10/2011 03:00, Bob DeRemer wrote: While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. If that were the case, the Tomcat devs wouldn't have put the investment they have into the Windows installer in recent months (auto detection of 32 or 64 bit JRE, greater control over ports, multiple service installs of the same version, ...) With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. Maybe. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. You need to understand the root cause. I'd recommend getting a profiler (I use YourKit since they give free copies to Tomcat committers) and finding out exactly what is holding references to those JARs. Once you know that the next questions are 1. What creates those references and 2. How to get rid of them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
On 21/10/2011 15:31, Mark Thomas wrote: On 21/10/2011 13:12, Bob DeRemer wrote: Thanks Mark, I appreciate the offer. Let me dig into a bit today - based on your steps. if I can't get anywhere, I'll gladly take you up on your offer. What is the best way to communicate out-of-band if I need to get you a snapshot? Bob and I took at look at this off-list using the standard approach for finding web-app reload memory leaks [1] and found what appears to be an issue with Tomcat's memory leak protection for the http keep-alive thread. Tomcat only clears the WebappClassLoader from the http keep-alive thread if the thread is still running. It should clear it in all circumstances. Ill be updating Tomcat to address this shortly. I have updated 7.0.x trunk. If you want to try it out, you'll need to build Tomcat from source. Instructions to do this are here: http://tomcat.apache.org/tomcat-7.0-doc/building.html The short version is: svn co http://svn.apache.org/repos/asf/tomcat/tc7.0.x/trunk/ tomcat-7 cd tomcat-7 ant Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
On 20/10/2011 18:18, Bob DeRemer wrote:
Hi Tomcat Community:
BACKGROUND:
I have a Servlet-based WEBAPP that is running in Tomcat7, JDK 1.6.0_27 on
Windows Server 2008 R2 64-bit - all 64-bit. We make use of class.newInstance
quite a bit as we have a fairly modular, interface-based system, and create
class instances on the fly. As part of our extensibility, I'm also using a
pretty well documented hack [The code is listed below] for dynamically
adding JAR(s) so they are accessible immediately without stopping our WEBAPP
or the actual Tomcat server process. This process has worked great for both
JAR(s) in the WEB-INF/lib and for ones we dynamically added - until you try
and stop the WEBAPP via the Tomcat manager app.
PROBLEM:
This solution for dynamically loading JARs worked great as long as we
start/stop the actual Tomcat server process, but I found out the JAR(s) that
I have been dynamically loading this way are not released when you stop the
WEBAPP. I am using SysInternals Process Explorer to monitor the open file
handles of the Tomcat7 process. I can see that when I stop the WEBAPP, the
dynamically loaded JAR is still referenced.
QUESTION:
I made an assumption that calling
Thread.currentThread().getContextClassLoader() would provide me the
classloader of my WEBAPP - which should be the same classloader that all the
JARs in the WEB-INF/lib are in. We control the WEBAPP, so I assumed this was
a safe assumption.
The usage of the dynamically loaded JAR(s) is no different than the default
JAR(s) in WEB-INF/lib. None of the WEB-INF/lib JARs are left open when we
stop the WEBAPP - only the JAR(s) that are loaded via the addUrl approach
below.
I have been through a number of online articles when I first created this
logic - including a PDF by Ted Neward [while at Developmentor] that described
these approaches, but am not sure how to troubleshoot this further now.
If anyone has an answer or any suggestions, I would greatly appreciate it.
Do you have 'antiJarLocking' enabled on the Context?
p
Thanks in advance,
Bob
Code to dynamically load JAR(s):
public static void loadJarOnTheFly(File jarFile) throws IOException
{
Class?[] parameters = new Class[]{URL.class};
// IMPORTANT: MUST use the webapp classloader - so derived extension
classes can resolve their base classes
ClassLoader contextClassLoader =
Thread.currentThread().getContextClassLoader();
// cast to a URL class loader so we can additional JAR(s) to the search
path
URLClassLoader webappClassLoader = (URLClassLoader)contextClassLoader;
Class? sysclass = URLClassLoader.class;
try
{
URL jarUrl = jarFile.toURI().toURL();
Method method = sysclass.getDeclaredMethod(addURL, parameters);
method.setAccessible(true);
method.invoke(webappClassLoader, new Object[]{ jarUrl });
}
catch (Throwable t)
{
throw new IOException(Error, could not add URL to system
classloader);
}
}
Bob DeRemer
Senior Director, Architecture and Development
http://www.thingworx.comhttp://www.thingworx.com/
Skype: bob.deremer
O: 717.505.7923
M: 717.881.3986
signature.asc
Description: OpenPGP digital signature
RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
I am not using a local app context and everything I've seen recommends NOT
using antijarlocking in production settings.
Thx,
bob
-Original Message-
From: Pid [mailto:p...@pidster.com]
Sent: Thursday, October 20, 2011 5:53 PM
To: Tomcat Users List
Subject: Re: JAR locked in Tomcat after using addUrl to dynamically add JAR to
running webapp
On 20/10/2011 18:18, Bob DeRemer wrote:
Hi Tomcat Community:
BACKGROUND:
I have a Servlet-based WEBAPP that is running in Tomcat7, JDK 1.6.0_27 on
Windows Server 2008 R2 64-bit - all 64-bit. We make use of class.newInstance
quite a bit as we have a fairly modular, interface-based system, and create
class instances on the fly. As part of our extensibility, I'm also using a
pretty well documented hack [The code is listed below] for dynamically
adding JAR(s) so they are accessible immediately without stopping our WEBAPP
or the actual Tomcat server process. This process has worked great for both
JAR(s) in the WEB-INF/lib and for ones we dynamically added - until you try
and stop the WEBAPP via the Tomcat manager app.
PROBLEM:
This solution for dynamically loading JARs worked great as long as we
start/stop the actual Tomcat server process, but I found out the JAR(s) that
I have been dynamically loading this way are not released when you stop the
WEBAPP. I am using SysInternals Process Explorer to monitor the open file
handles of the Tomcat7 process. I can see that when I stop the WEBAPP, the
dynamically loaded JAR is still referenced.
QUESTION:
I made an assumption that calling
Thread.currentThread().getContextClassLoader() would provide me the
classloader of my WEBAPP - which should be the same classloader that all the
JARs in the WEB-INF/lib are in. We control the WEBAPP, so I assumed this was
a safe assumption.
The usage of the dynamically loaded JAR(s) is no different than the default
JAR(s) in WEB-INF/lib. None of the WEB-INF/lib JARs are left open when we
stop the WEBAPP - only the JAR(s) that are loaded via the addUrl approach
below.
I have been through a number of online articles when I first created this
logic - including a PDF by Ted Neward [while at Developmentor] that described
these approaches, but am not sure how to troubleshoot this further now.
If anyone has an answer or any suggestions, I would greatly appreciate it.
Do you have 'antiJarLocking' enabled on the Context?
p
Thanks in advance,
Bob
Code to dynamically load JAR(s):
public static void loadJarOnTheFly(File jarFile) throws IOException
{
Class?[] parameters = new Class[]{URL.class};
// IMPORTANT: MUST use the webapp classloader - so derived extension
classes can resolve their base classes
ClassLoader contextClassLoader =
Thread.currentThread().getContextClassLoader();
// cast to a URL class loader so we can additional JAR(s) to the search
path
URLClassLoader webappClassLoader = (URLClassLoader)contextClassLoader;
Class? sysclass = URLClassLoader.class;
try
{
URL jarUrl = jarFile.toURI().toURL();
Method method = sysclass.getDeclaredMethod(addURL, parameters);
method.setAccessible(true);
method.invoke(webappClassLoader, new Object[]{ jarUrl });
}
catch (Throwable t)
{
throw new IOException(Error, could not add URL to system
classloader);
}
}
Bob DeRemer
Senior Director, Architecture and Development
http://www.thingworx.comhttp://www.thingworx.com/
Skype: bob.deremer
O: 717.505.7923
M: 717.881.3986
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
From: Bob DeRemer [mailto:bob.dere...@thingworx.com] Subject: RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp I am not using a local app context and everything I've seen recommends NOT using antijarlocking in production settings. Yes, except for the fact that you're doing rather odd things, including running on Windows... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp
While I can appreciate the desire to never use Windows, that isn't reality in many of the industries that use our product, so running Tomcat on windows can't be considered an oddity or edge case. If it is, we better look for another app server. With regard to finding a solution, I'll try the antiJarLocking setting, but I don't have any problem with any other JARS, so I don't hold out much hope for this fixing my problem. I may look at upgrading to JDK 7 - which has added a close method to the URLClassLoader. Perhaps this might fix it. If anyone else has any other ideas why ONLY JARs added via the addURL hack I've shown below don't get unloaded when the WEBAPP is shutdown - vs stopping Tomcat completely, I would greatly appreciate it. Thanks, Bob -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Thursday, October 20, 2011 6:07 PM To: Tomcat Users List Subject: RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp From: Bob DeRemer [mailto:bob.dere...@thingworx.com] Subject: RE: JAR locked in Tomcat after using addUrl to dynamically add JAR to running webapp I am not using a local app context and everything I've seen recommends NOT using antijarlocking in production settings. Yes, except for the fact that you're doing rather odd things, including running on Windows... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
