Re: tomcat randomly undeploys and redeploys the applications
Ive done some more research into this problem with my co developer, and we have a question. Is it possible that memory leaks would cause tomcat to crash? Silly question I know, but heres technically what the situation is. We did not add code to shut down all threads when application is undeployed through tomcat manager. When undeploying and redeploying we cause memory leaks due to our application. Now, I guess the real question is, we seem to always deploy it fine, but randomly at some point during the night it will crash, and we only know that from checking next morning. We have nothing logged that indicates why. Would memory leaks cause it to crash randomly even though nothing is trying to connect to it, yet it would seem to deploy fine? Could that be the issue we are seeing? Is there any specific class to debug in logging.properties that might indicate whats going on? I added that line you guys mentioned about setting log level to DEBUG from that class loader class, but it hasnt crashed since then On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 5:20 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:54 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I use Eclipse for development, but our real build process is ant-based. We have some watched resources configured outside the default (stuff like Struts config files, etc.). Ah, makes sense. When I do a build while Tomcat is running, usually I get one webapp reload, but sometimes I get a series of reloads. It usually gets so irritating (our webapp takes about 10 seconds to reload) that I just kill Tomcat and immediately restart it. It starts up once and all is well after that. Yep, and in the process more files are copied about, and that triggers another reload. Fun, fun. No, the deployment update takes like one or two seconds. It's usually something like copying less than 10 class files or whatever. It's nearly instantaneous. Whatever happens, it's not because I'm updating files during the reload. I could understand that situation. What I observe is that I update my application, I wait maybe 10 seconds, and then Tomcat reloads my application multiple times before I just kill it. I've not seen this, but it could explain some issues some the developers I support are seeing. It definitely happens, and I never remember to enable the DEBUG logging to find out what resource it thinks has been updated until after it happens, at which point I just don't care. Perhaps I should enable it right now :) - -chris I've managed to make this happen in my environment now (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I just needed an application that takes a bit longer to load. I only managed to trigger two reloads, so it's not much of an issue. Maybe look at adding the backgroundProcessorDelay attribute to the context? I don't know what would happen if the context got a string of reload requests within the delay interval. Would it queue them up one after the other, or would it just execute one? I think it's more important to see what file(s) Tomcat thinks is(are) being updated. I wonder if it's the same file, or if there's some weird timestamp issue happening. Perhaps there is even some kind of edge case where a resource's last-modified date isn't being updated properly. In most cases, Tomcat reloads my application a single time, as expected.
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Elias, On 4/4/14, 11:03 AM, Elias Kopsiaftis wrote: Ive done some more research into this problem with my co developer, and we have a question. Is it possible that memory leaks would cause tomcat to crash? Yes, but you'll likely get an OutOfMemoryError in your log, and then everything will go crazy. Some requests will work, others will not. Things will get slow. For us, the background thread often quits and sessions never expire, exacerbating the problem. But it won't happen silently (you may have to look for it) and it won't spontaneously undeploy and redeploy your web application. Silly question I know, but heres technically what the situation is. We did not add code to shut down all threads when application is undeployed through tomcat manager. Tomcat will do this itself: you don't need to add any code. Or did you mean that you have Threads created by your application that continue to tun after your web app has been shut down? When undeploying and redeploying we cause memory leaks due to our application. Yes, this can happen. The Threads cause the WebappClassLoader to stay in memory, which means that all Class objects for the previous deployment are still hanging around. It can be several tens of MiB per reload. You should read this excellent presentation which explains the problem(s) and the solution(s): http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf Now, I guess the real question is, we seem to always deploy it fine, but randomly at some point during the night it will crash, and we only know that from checking next morning. What do you mean when you say crash? Exception? Spontaneous re-deployment? OutOfMemory? JVM crash (process exits)? We have nothing logged that indicates why. If the process is gone without any indication of why (no shutdown message, no nothing), my only guess is Linux OOM-killer. You never actually came back to describe the environment you are in after saying you'd post that information later today. If you're not on Linux I'm not sure. Would memory leaks cause it to crash randomly even though nothing is trying to connect to it, yet it would seem to deploy fine? I've never seen an OOME actually cause a JVM to quit. It just slogs onward, often not able to accomplish any useful work, but the process stays running. Could that be the issue we are seeing? Is there any specific class to debug in logging.properties that might indicate whats going on? I added that line you guys mentioned about setting log level to DEBUG from that class loader class, but it hasnt crashed since then Describe what you mean by crash, including as much details about it, and definitely read the presentation referenced above. Also definitely fix your web application to shut down its Threads. Whatever component launches them (e.g. ServletContextListener, Servlet.init, etc.) should stop them in the appropriate callback method (ServletContextListener.contextDestroyed, Servlet.destroy, etc.). - -chris On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz ch...@christopherschultz.net wrote: Mark, On 4/2/14, 5:20 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:54 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I use Eclipse for development, but our real build process is ant-based. We have some watched resources configured outside the default (stuff like Struts config files, etc.). Ah, makes sense. When I do a build while Tomcat is running, usually I get one webapp reload, but sometimes I get a series
Re: tomcat randomly undeploys and redeploys the applications
Im sorry about that. The reason I did not post same day was that through the messages in this thread someone suggested adding a debug statement to debug WebappClassLoader, and I wanted to wait to see if it happened again so I could see the debug output, then I came up with the idea in my last message. yes we are on Linux and we are using java 7 and tomcat 7 and we are on linux 3.11.6 on a 64 bit build for x86. When I say crash, I mean that when we use the client application to connect to the web app it wont connect and we have to go into tomcat manager and start it back up. We use an executor with 20 threads in it to process things on the server, and that was never shut down, so that would explain the memory leak and the extra threads. We need them to run for the duration of the application life cycle, but they should be cleaned up on undeployment. Thanks for the reference to the presentation. I will go through it now. I think I may have my answer and hopefully this wont happen again after we fix our web app to shut down those threads. On Fri, Apr 4, 2014 at 11:17 AM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Elias, On 4/4/14, 11:03 AM, Elias Kopsiaftis wrote: Ive done some more research into this problem with my co developer, and we have a question. Is it possible that memory leaks would cause tomcat to crash? Yes, but you'll likely get an OutOfMemoryError in your log, and then everything will go crazy. Some requests will work, others will not. Things will get slow. For us, the background thread often quits and sessions never expire, exacerbating the problem. But it won't happen silently (you may have to look for it) and it won't spontaneously undeploy and redeploy your web application. Silly question I know, but heres technically what the situation is. We did not add code to shut down all threads when application is undeployed through tomcat manager. Tomcat will do this itself: you don't need to add any code. Or did you mean that you have Threads created by your application that continue to tun after your web app has been shut down? When undeploying and redeploying we cause memory leaks due to our application. Yes, this can happen. The Threads cause the WebappClassLoader to stay in memory, which means that all Class objects for the previous deployment are still hanging around. It can be several tens of MiB per reload. You should read this excellent presentation which explains the problem(s) and the solution(s): http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf Now, I guess the real question is, we seem to always deploy it fine, but randomly at some point during the night it will crash, and we only know that from checking next morning. What do you mean when you say crash? Exception? Spontaneous re-deployment? OutOfMemory? JVM crash (process exits)? We have nothing logged that indicates why. If the process is gone without any indication of why (no shutdown message, no nothing), my only guess is Linux OOM-killer. You never actually came back to describe the environment you are in after saying you'd post that information later today. If you're not on Linux I'm not sure. Would memory leaks cause it to crash randomly even though nothing is trying to connect to it, yet it would seem to deploy fine? I've never seen an OOME actually cause a JVM to quit. It just slogs onward, often not able to accomplish any useful work, but the process stays running. Could that be the issue we are seeing? Is there any specific class to debug in logging.properties that might indicate whats going on? I added that line you guys mentioned about setting log level to DEBUG from that class loader class, but it hasnt crashed since then Describe what you mean by crash, including as much details about it, and definitely read the presentation referenced above. Also definitely fix your web application to shut down its Threads. Whatever component launches them (e.g. ServletContextListener, Servlet.init, etc.) should stop them in the appropriate callback method (ServletContextListener.contextDestroyed, Servlet.destroy, etc.). - -chris On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz ch...@christopherschultz.net wrote: Mark, On 4/2/14, 5:20 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:54 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 5:20 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:54 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I use Eclipse for development, but our real build process is ant-based. We have some watched resources configured outside the default (stuff like Struts config files, etc.). Ah, makes sense. When I do a build while Tomcat is running, usually I get one webapp reload, but sometimes I get a series of reloads. It usually gets so irritating (our webapp takes about 10 seconds to reload) that I just kill Tomcat and immediately restart it. It starts up once and all is well after that. Yep, and in the process more files are copied about, and that triggers another reload. Fun, fun. No, the deployment update takes like one or two seconds. It's usually something like copying less than 10 class files or whatever. It's nearly instantaneous. Whatever happens, it's not because I'm updating files during the reload. I could understand that situation. What I observe is that I update my application, I wait maybe 10 seconds, and then Tomcat reloads my application multiple times before I just kill it. I've not seen this, but it could explain some issues some the developers I support are seeing. It definitely happens, and I never remember to enable the DEBUG logging to find out what resource it thinks has been updated until after it happens, at which point I just don't care. Perhaps I should enable it right now :) - -chris I've managed to make this happen in my environment now (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I just needed an application that takes a bit longer to load. I only managed to trigger two reloads, so it's not much of an issue. Maybe look at adding the backgroundProcessorDelay attribute to the context? I don't know what would happen if the context got a string of reload requests within the delay interval. Would it queue them up one after the other, or would it just execute one? I think it's more important to see what file(s) Tomcat thinks is(are) being updated. I wonder if it's the same file, or if there's some weird timestamp issue happening. Perhaps there is even some kind of edge case where a resource's last-modified date isn't being updated properly. In most cases, Tomcat reloads my application a single time, as expected. These reload-storms are fairly rare. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPYcLAAoJEBzwKT+lPKRYf/gQAKhUNIp0yIP24g6fctOppPGC 0WbacCWGaJwC9Xbuh/lNqtYRZck2943+rOZwOw4sBfwxox3dlVYGl+NHj0C4NRyW KJmH11v5gO59Di7S4NZeYGlHpbYWSZt/2HiMZAVQ6zElXN5qkSEa5WbjXJJsduSe FHD6RFLST7pvlbOuEj8L/+MldsflYTe7Mu5CykBK52GLZGMAYTFYWqcs6nrsscxc ZWEKAt1QU1barvojnoZjk4pZksihi/QwOmCJ1a+rHWUZPmtzp/9gdTon47WHDcXU NVEQLOHJgtolJI2XYMVXFZPVOEeD80PWdQ+aIUAozOR954odw9RcZRz71EiNAagB YBImgFi0zFNwVKivi4yqKHGex8LPj6qGuDI2Nd48Za4s6gN90fpHwpYFzrq3dnnv ep7Jf1qHNiGc70s//TH8iKZToCkN/N2Sythlv729NEFGdXBkn0Ph8RVhPmxiYiwu WrIXrk875m2241QBubcuaNFnPPmNA2cj0IkHB8QDM+35wUI40sx+vtMvLV3U674O JBATY5Oiq26jj585OAsNJJsNY1y33rkH0tqZNEPTYUfqSW4FUxHe0J6TWf7CMujF sPjQqk1kXawsyNJNeGqKxUqW6LHvo+dX40/FmgAl4llkOCU9DFXxdhZVBg14UQdC OeWlyGTxwovZZEdJJqtO =r81a -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: tomcat randomly undeploys and redeploys the applications
From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. If you want anything more than speculation, you'll need to provide real information: Tomcat version, JRE version, platform, server.xml (with comments removed), context.xml for the webapp, etc. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Chuck, Thanks for your response. I will get you all the information you asked for later tonight when I get out of work. In the meantime, the only threads our application creates are those created by a ScheduledThreadPoolExecutor, and in fact, when I do a thread dump to analyze what is going on, there are 30 threads associated with this executor that are in a BLOCKED state. We have had trouble with executors in the past, but these threads are supposed to be running for the lifecycle of the application. Is there a problem using an executor in a tomcat application? I will provide you with the following later today: Tomcat version, JRE version, platform, server.xml (with comments removed), context.xml for the webapp, etc. Would you like me to also attach the thread dump? On Wed, Apr 2, 2014 at 8:21 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. If you want anything more than speculation, you'll need to provide real information: Tomcat version, JRE version, platform, server.xml (with comments removed), context.xml for the webapp, etc. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPG2nAAoJEBzwKT+lPKRY9oIQAIFI0aUQ33he8KqNQgwBaigY otTiL3Cr6dvVfYi2pLP9YK/2BqOkjtmCXAZ99gOhV0h3QntMWrBbYmkcu6BuibPH gnm2vME/B8XdNhGatPQgs1dW3OWa/OTra6req0CcLX3IucGERJWrMotOSLrsylUH lVkVWoRXZ3m6NikExKCmB/DUzy9dsecHGZM3hETEloye6p7EVez+RHFg/7VWDU42 GyPZqiGFVRDELobbc4FVAy2R0OLPVSXzbsnuM3ZFS/TRqX+PKItfsnYczzrck3rK ozeYjmgG5IKD4eL/Nl0iUcZ/CjmHxTJCSlVwRhnQ7InFoNjR0TYur2dyw/+mqdlj U0u61XYJiUqHUnk5LUh5pFvxfKOyRSy34YlHAHTOBg6Nl/TZoAMtGBXQ5gsLssHC uf1U/oTl1hfhgIynQ3ijn10TXXXskEux5GHegUGVv8HqZNEcPVtuPw+3fEhGsXEu vgHmXUJIbLBDwIVD9WGC7iYHR4zNwHetlFwUE+sxAxP/zLwPd+Ps4TcoKabUh0nG sE7oj/qZxBHRxGIXg6BnGuzerHJPJT8dcaAo+U1zSmQdlAy1mCdvUrFpA7i63RdT XU8mwimmr93WBBOkhnWg4iJjTIl2LKEo1GmV0JiB/eqZaWIFNFy8oBnW56KnEhLV coHAGbSi2o7XZUdVpn0v =kMSx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Thats very interesting. When you say update, what do you mean technically? On Wed, Apr 2, 2014 at 4:05 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPG2nAAoJEBzwKT+lPKRY9oIQAIFI0aUQ33he8KqNQgwBaigY otTiL3Cr6dvVfYi2pLP9YK/2BqOkjtmCXAZ99gOhV0h3QntMWrBbYmkcu6BuibPH gnm2vME/B8XdNhGatPQgs1dW3OWa/OTra6req0CcLX3IucGERJWrMotOSLrsylUH lVkVWoRXZ3m6NikExKCmB/DUzy9dsecHGZM3hETEloye6p7EVez+RHFg/7VWDU42 GyPZqiGFVRDELobbc4FVAy2R0OLPVSXzbsnuM3ZFS/TRqX+PKItfsnYczzrck3rK ozeYjmgG5IKD4eL/Nl0iUcZ/CjmHxTJCSlVwRhnQ7InFoNjR0TYur2dyw/+mqdlj U0u61XYJiUqHUnk5LUh5pFvxfKOyRSy34YlHAHTOBg6Nl/TZoAMtGBXQ5gsLssHC uf1U/oTl1hfhgIynQ3ijn10TXXXskEux5GHegUGVv8HqZNEcPVtuPw+3fEhGsXEu vgHmXUJIbLBDwIVD9WGC7iYHR4zNwHetlFwUE+sxAxP/zLwPd+Ps4TcoKabUh0nG sE7oj/qZxBHRxGIXg6BnGuzerHJPJT8dcaAo+U1zSmQdlAy1mCdvUrFpA7i63RdT XU8mwimmr93WBBOkhnWg4iJjTIl2LKEo1GmV0JiB/eqZaWIFNFy8oBnW56KnEhLV coHAGbSi2o7XZUdVpn0v =kMSx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Elias, On 4/2/14, 4:05 PM, Christopher Schultz wrote: Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. It's WebappClassLoader. You'll want to add this to your logging.properties in order to emit those messages: org.apache.catalina.loader.WebappClassLoader.level=DEBUG - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPHDjAAoJEBzwKT+lPKRYr4wP/3AzjqhtU8MKyuEoHSwyrwTt 3z0GaYZEolH+kW46+QOOnFUOXvWaK8MHjy5VgRU1rZP9DK/dwGYGbKrA7sMd9iuG nvjo7v/hLrPxtvvt8LKiH+vQslhyMU86uNReXu90QUWD0shj3twfC78IEWJKp0rl NR2/+mIardJv4KeZ06HQVgD/+xWpWVSxeDOeJJ3i1/alxhh7P7ybs8M3v9QTmaHC XFQEKSdb4MEn1tgEl88ucbhaibKLJKizOYRBfG2k0J3CcVsRtfsefmh0KDXSSBT+ C6cOKULYCf4JimHdLAo8Ap0C4d6dBIu2MwIXRR+voV0rEtsn2uf43VaJmp9XsQie 9Gb6PQz7LUHUBAMb4FNrTRAWqfAb0LSq9Q1wXVOa9Ai6HMkYKnIZaO9pHfnAaQKK 34Zc5JMahtsbPtRj2jKSWiss92N5EDcKZHPLnhz1Z0q+zKRfCiBxDC3aS/bb4vIY HuYLc1MZq9+2PXjj6Mp9TP/eYOLkGgmT9OLsN+sdNSEmwpFQd/p5zQ9He7OW05O9 q0wMjNmx0SRfjV5BrtG2k4eSvVaJpUP0FNKWWhaBAkreIyLs3P5C2QMjZ36NnJkA zO6jDsCJjwc7QOroSFyB++qm3FISDxGTuestkv0UAi92I+ePgjxooOPqSLmaVv5i 7UiJvtiB6XzAMxpn8wny =cKnh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Thanks Chris! I will look into this and let you guys know what I find On Wed, Apr 2, 2014 at 4:19 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Elias, On 4/2/14, 4:05 PM, Christopher Schultz wrote: Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. It's WebappClassLoader. You'll want to add this to your logging.properties in order to emit those messages: org.apache.catalina.loader.WebappClassLoader.level=DEBUG - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPHDjAAoJEBzwKT+lPKRYr4wP/3AzjqhtU8MKyuEoHSwyrwTt 3z0GaYZEolH+kW46+QOOnFUOXvWaK8MHjy5VgRU1rZP9DK/dwGYGbKrA7sMd9iuG nvjo7v/hLrPxtvvt8LKiH+vQslhyMU86uNReXu90QUWD0shj3twfC78IEWJKp0rl NR2/+mIardJv4KeZ06HQVgD/+xWpWVSxeDOeJJ3i1/alxhh7P7ybs8M3v9QTmaHC XFQEKSdb4MEn1tgEl88ucbhaibKLJKizOYRBfG2k0J3CcVsRtfsefmh0KDXSSBT+ C6cOKULYCf4JimHdLAo8Ap0C4d6dBIu2MwIXRR+voV0rEtsn2uf43VaJmp9XsQie 9Gb6PQz7LUHUBAMb4FNrTRAWqfAb0LSq9Q1wXVOa9Ai6HMkYKnIZaO9pHfnAaQKK 34Zc5JMahtsbPtRj2jKSWiss92N5EDcKZHPLnhz1Z0q+zKRfCiBxDC3aS/bb4vIY HuYLc1MZq9+2PXjj6Mp9TP/eYOLkGgmT9OLsN+sdNSEmwpFQd/p5zQ9He7OW05O9 q0wMjNmx0SRfjV5BrtG2k4eSvVaJpUP0FNKWWhaBAkreIyLs3P5C2QMjZ36NnJkA zO6jDsCJjwc7QOroSFyB++qm3FISDxGTuestkv0UAi92I+ePgjxooOPqSLmaVv5i 7UiJvtiB6XzAMxpn8wny =cKnh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Elias, On 4/2/14, 4:12 PM, Elias Kopsiaftis wrote: Thats very interesting. When you say update, what do you mean technically? I mean a build that triggers new class files, libraries, or some other watched resource to trigger an automatic reload. If you want to disable automatic reloads completely, set reloadable=false on your Context (which is actually the default). If you want to still be able to reload your web application sometimes, consider installing the Manager webapp, which will let you do reloads, etc. via a text/HTTP or GUI interface. - -chris On Wed, Apr 2, 2014 at 4:05 PM, Christopher Schultz ch...@christopherschultz.net wrote: Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPHFUAAoJEBzwKT+lPKRYwKEQALm1X3IqSHJLcLRyeM41gDeh vcnu1zgUP1KqGJeV1BZHq3aREUBELdcpCncvVALd4xuJT+1JR6qDbHRINNSUAEbF JNx8j0r+/Y2eYFFwWt4gNa+7USSxvy7xzz0Sj726U74MCoiSQULKB/53VNn8tPPA GhNsZcIcclYaaaDyEV5OQ0joyLwrVXo7AWbvsh82AJmpoQ0WnkKmsX40u4rTbMfb bb3JLnDEs93WGbYzKd7UfXxs9gaVFLg+TFeBP+hNiAfRz2JFiBiQnpwsNBURA0lo SySFylSVXI2JjHexaT4N+OR3ZB4d2p4eEwOA1Cv9ZGEU/9wH6cTIykDsL2SYWubj 6+3VeGL2xepq6PYxaniDaRLx+dxcEOs97dVxHS/w+M5VzUm2LZm7O9vXuhtQTLGt WIOhp5ffpvYVvLhuDoxm7XmPMnMH4kBOjc5Cz4v3NgL9eVUJIRWJxWPkdqnZFiU/ S9pQmWl6BBgt5gOUUjLu8almvYWoi8ogGt8Sv4iX1EYdGrGpU1eyqxlHV3C2pR58 eBUqaVE4GLaa0H8ZknMm0leLRdzdhMaEUDbPBdHl2XfCo9+CkqX059zqp6UPFiFr ap5UgkFl8llobJoxLM1hbE635Wxg26LpxagQ9kUR1wWHrlyIeldrEV0dQtfuvqf5 b7KIADcRBXm5hTpRTXap =EZOG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I've not seen this, but it could explain some issues some the developers I support are seeing. I believe Tomcat will emit a log message informing the administrator which file has triggered the reload, but I think it's at the DEBUG log level. - -chris Thanks, /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I use Eclipse for development, but our real build process is ant-based. We have some watched resources configured outside the default (stuff like Struts config files, etc.). When I do a build while Tomcat is running, usually I get one webapp reload, but sometimes I get a series of reloads. It usually gets so irritating (our webapp takes about 10 seconds to reload) that I just kill Tomcat and immediately restart it. It starts up once and all is well after that. I've not seen this, but it could explain some issues some the developers I support are seeing. It definitely happens, and I never remember to enable the DEBUG logging to find out what resource it thinks has been updated until after it happens, at which point I just don't care. Perhaps I should enable it right now :) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTPHjmAAoJEBzwKT+lPKRYtzMP/2cBUk4nK2Ox5IL4BvxT0YF1 xb5WGrh+QvRBuOyPF1M2X2SVIijyh0k9geRD3zWaGl4YZ8dLVZY4mXEmh2lXt4g9 B1jhNrUpeNHUVtb5gVrzwj063H+duabm67OWzXXJMzTY2MlaroPhx9W80Fu7OpJY Nsh7WNsFxBa5KPAV2H3XjbP6Oskbpv1F329AaBHM6HAH2J4eHahtpEujaIolSQbA L8rOT1UYw3dblMQRuvryydB42btw1hTpgZX7UHGIsNxtELPy8LkwrmaCxAKaHBvR zbw2vKj5VWjpaBmWzPT4qIrdIJg2wUAfXYjyAKuZ5JRxIxEPiTWB6SM5224Jzknv DFwic0FxervIcLudfSb96Xcto0fqNzUJeHHPhRD8eGzSOIdkqiI8Faqml/f207ma ibXQp6OIo1P8YLGkA21ZWBwSrp3hvKpV0+Nq+LarVTueODv4Mig1E8S3B5n9CzZQ 8N6vVZolnZlpkXC88VuQk6Dct0u8kmx5sqR/HUY7ACSVZHskRABVi5C+AxWg89TN ElXmPydGyjgvOp1Ldf5S6jBSdTY9qI7LEl7kBrEUKMz4aBzwfmTw2/08yozAjaQW TxJuOderFvslr+C2iQ6g8Qqsk+davD/bZ05REw9vz4/QgGzGCNhAW9pKoH5GoU0x ZHwMuvTSzwM754whVzKf =fzrA -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Chris, On 4/2/2014 1:54 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/2/14, 4:30 PM, Mark Eggers wrote: Chris, On 4/2/2014 1:05 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 4/2/14, 8:21 AM, Caldarale, Charles R wrote: From: Elias Kopsiaftis [mailto:yemi...@gmail.com] Subject: tomcat randomly undeploys and redeploys the applications I deploy the application, then in the log file catalina.out i get many messages from WebappClassLoader clearReferencesThreads saying threads appear to have started but have failed to stop This is an indication that your webapp is not managing its threads properly. then finally, Ill get a message from HostConfig checkResources that says its undeploying the context, and then it redeploys. This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously. I've found that in development, a single update can cause Tomcat to go into a loop of redeployments, re-deploying my web application every few seconds or so. If I let it go, it does finally stop reloading and settle down. Can you describe your development environment a little bit, and any thoughts as to what might trigger this loop of redeployments? I use Eclipse for development, but our real build process is ant-based. We have some watched resources configured outside the default (stuff like Struts config files, etc.). Ah, makes sense. When I do a build while Tomcat is running, usually I get one webapp reload, but sometimes I get a series of reloads. It usually gets so irritating (our webapp takes about 10 seconds to reload) that I just kill Tomcat and immediately restart it. It starts up once and all is well after that. Yep, and in the process more files are copied about, and that triggers another reload. Fun, fun. I've not seen this, but it could explain some issues some the developers I support are seeing. It definitely happens, and I never remember to enable the DEBUG logging to find out what resource it thinks has been updated until after it happens, at which point I just don't care. Perhaps I should enable it right now :) - -chris I've managed to make this happen in my environment now (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I just needed an application that takes a bit longer to load. I only managed to trigger two reloads, so it's not much of an issue. Maybe look at adding the backgroundProcessorDelay attribute to the context? I don't know what would happen if the context got a string of reload requests within the delay interval. Would it queue them up one after the other, or would it just execute one? /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
