Re: Web service not starting up as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 3:57 PM, Debbie Shapiro wrote: I'm pretty sure that's accurate in this case, however. I had 7.0.23 installed as of last week and then this week did the upgrade to 7.0.26. I'm wondering about your question regarding timing though. It's possible that Tomcat is doing more cleanup than it has in the past and it's taking longer to shut down? Maybe I do just need some kind of pause in between stopping and restarting. Perhaps. I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM So there's no delay between requesting a service-stop and calling taskkill (I assume that's like *NIX 'kill' that basically murders the process)? I would consider giving Tomcat some time to shut down. You never know what kinds of things need time to shut down. REM delay batch job to wait for services to stop I don't see an actual delay, do you? sc \\bidev2 start Tomcat7 Yeah, I'll bet you need some delay in this script in order for it to work properly. I have actually mentioned the memory leaks to them before. They said not to be concerned, but maybe others will report this as well. They aren't concerned because they aren't running their own software in production :( They need to fix this because it is definitely a memory leak. If this is really the end of the log file, then it looks like Tomcat didn't even attempt to start itself again. When you perform a restart, do you move the log file out of the way or anything like that? If so, is there a new log file that might have some indication on an error? This was the end of the file, yes. I thought it looked odd too. I don't move any of the Tomcat logs. I only compress and rename the old InetSoft logs. Okay, then Tomcat isn't even trying to start again (or at least isn't logging anything). It's possible that if the old process is still shutting down, the new process can't get a lock on the log file for writing, so no logs are written. I have a scheduled task that kicks off the .bat file I copied above. Okay, add some delay. I dunno how to do that on win32. This second scheduled task started up fine. So maybe it is just a timing issue. 7.0.26 takes longer to shut down than 7.0.23 maybe? It probably takes about the same amount of time, you're just noticing it now for some reason. I'm going to test out a delay in the restart after I shut down, but just wanted to throw out there that after my stop service command, which happens at exactly 11:30, the restart failure in the event viewer has a timestamp of 11:30:06. That's not much of a delay between the stop and restart, so maybe it's still technically shutting down at this point. 6 seconds to shut down wouldn't be out of the ordinary. If a webapp is doing things properly, it will have some of its own shutdown code running to clean up resources, etc. that will take non-zero time. Most of the shutdown time will be the webapp and not Tomcat itself, which really just has to un-deploy the webapp and then stop the connectors. Yes, it is pretty common in the windows world to restart servers and services due to various memory issues. I'm doing this because the application is a reporting tool and if the users run any enormous reports that cause it to run out of java heap or something, this provides for an automated way of maintaining the application so that it doesn't require manual intervention to fix it and I know that even if no one calls me, it will be reset at some point. I also use it as an opportunity to save off a daily copy of the InetSoft logs if I need to go back and review errors later. Personally, I'm a Mac fan at heart. :-) I'm curious how long you could run without a restart. Usually for reporting types of webapps, it's either a single request that brings-down the server (e.g. it needs a huge amount of memory to run a single report) or you have lots of smaller requests that need a ton of memory in aggregate. It's not really possible to fix the former, other than getting more memory or changing the way reports are done to use less of it. Threads can't get their own memory usage so you can't really limit one particular request. It's easy to fix the latter: just find our whatever you maximum load is in terms of simultaneous reports, and simply never let that many run at once. You can do that with a report-counter in the webapp itself, or you can do it by limiting the maximum number of connections that the server will accept. For my money, I'd off-load the actual report-generation to an out-of-process (duh) process and pick-up the result when it's done. That way, your webapp doesn't become unstable even if one report runs out of memory. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17
Re: Web service not starting up as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 5:56 PM, Debbie Shapiro wrote: -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Now wait.. I am not so familiar with these Windows commands, but do I see above a) a command to stop the Tomcat service b) a command killing java.exe ? Yes. This code is sort of a holdover from another application that I could only kill the service by also killing the java.exe processes that it was tied to, otherwise it couldn't obtain the port because it was already in use. Probably not necessary for this application, but I figured it couldn't hurt. It *can* hurt: what about any other Java processes that are running? REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c and c) a series of command tending to indicate that this application puts its logfiles inside the classes directory ? (not fatal, but at least bizarre) Yes, it does. This is the way the vendor coded it. I have brought it to their attention but they don't seem to see a problem with it. Idiots. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9eH4AACgkQ9CaO5/Lv0PCXTQCfSWLx0kzneT+PTnL2ZQs/ygs2 WlIAnRHlaN7hcWnIcXif9xIGQRlY72FG =G+vp -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Web service not starting up as expected sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM So there's no delay between requesting a service-stop and calling taskkill (I assume that's like *NIX 'kill' that basically murders the process)? The taskkill is a red herring in this particular situation, since the executable name is not java.exe when Tomcat is running as a service. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Web service not starting up as expected
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, March 12, 2012 9:09 AM To: Tomcat Users List Subject: Re: Web service not starting up as expected -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 5:56 PM, Debbie Shapiro wrote: -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Now wait.. I am not so familiar with these Windows commands, but do I see above a) a command to stop the Tomcat service b) a command killing java.exe ? Yes. This code is sort of a holdover from another application that I could only kill the service by also killing the java.exe processes that it was tied to, otherwise it couldn't obtain the port because it was already in use. Probably not necessary for this application, but I figured it couldn't hurt. It *can* hurt: what about any other Java processes that are running? This is a dedicated server that is only running this process. There should be only the one java process running. REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c and c) a series of command tending to indicate that this application puts its logfiles inside the classes directory ? (not fatal, but at least bizarre) Yes, it does. This is the way the vendor coded it. I have brought it to their attention but they don't seem to see a problem with it. Idiots. Yes, well, unfortunately, I can't force them to change how this is coded. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9eH4AACgkQ9CaO5/Lv0PCXTQCfSWLx0kzneT+PTnL2ZQs/ygs2 WlIAnRHlaN7hcWnIcXif9xIGQRlY72FG =G+vp -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
I'm still seeing a java.exe process being launched in the Tomcat7.exe process tree. Please create a HelpStar ticket for any requests for assistance. This will help us better track your request. http://helpstar Debbie Shapiro Data Warehouse Manager Cardiac Science Office: 425.402.2233 Visit us at www.cardiacscience.com Suppliers of Cardiac Science, Criticare, Unetixs, Powerheart, Burdick, and Quinton products Part of the Opto Circuits Family -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Monday, March 12, 2012 10:28 AM To: Tomcat Users List Subject: RE: Web service not starting up as expected From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Web service not starting up as expected sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM So there's no delay between requesting a service-stop and calling taskkill (I assume that's like *NIX 'kill' that basically murders the process)? The taskkill is a red herring in this particular situation, since the executable name is not java.exe when Tomcat is running as a service. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, March 12, 2012 9:07 AM To: Tomcat Users List Subject: Re: Web service not starting up as expected -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 3:57 PM, Debbie Shapiro wrote: I'm pretty sure that's accurate in this case, however. I had 7.0.23 installed as of last week and then this week did the upgrade to 7.0.26. I'm wondering about your question regarding timing though. It's possible that Tomcat is doing more cleanup than it has in the past and it's taking longer to shut down? Maybe I do just need some kind of pause in between stopping and restarting. Perhaps. I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM So there's no delay between requesting a service-stop and calling taskkill (I assume that's like *NIX 'kill' that basically murders the process)? I would consider giving Tomcat some time to shut down. You never know what kinds of things need time to shut down. REM delay batch job to wait for services to stop I don't see an actual delay, do you? No, I had left that there as a placeholder from my previous code. When I set up this server as 64-bit, I was trying to hold off from installing the Microsoft resource toolkit that contains the sleep command. I have that installed now and added a sleep command here. sc \\bidev2 start Tomcat7 Yeah, I'll bet you need some delay in this script in order for it to work properly. I have actually mentioned the memory leaks to them before. They said not to be concerned, but maybe others will report this as well. They aren't concerned because they aren't running their own software in production :( They need to fix this because it is definitely a memory leak. I agree, this doesn't look right. I'll ping them about it again. If this is really the end of the log file, then it looks like Tomcat didn't even attempt to start itself again. When you perform a restart, do you move the log file out of the way or anything like that? If so, is there a new log file that might have some indication on an error? This was the end of the file, yes. I thought it looked odd too. I don't move any of the Tomcat logs. I only compress and rename the old InetSoft logs. Okay, then Tomcat isn't even trying to start again (or at least isn't logging anything). It's possible that if the old process is still shutting down, the new process can't get a lock on the log file for writing, so no logs are written. I have a scheduled task that kicks off the .bat file I copied above. Okay, add some delay. I dunno how to do that on win32. Install the Microsoft Resource Toolkit. It contains a sleep command. This second scheduled task started up fine. So maybe it is just a timing issue. 7.0.26 takes longer to shut down than 7.0.23 maybe? It probably takes about the same amount of time, you're just noticing it now for some reason. Something must be different because 7.0.23 didn't require me to add the delay to my script and I hadn't received this error until I installed 7.0.26. The sleep command seems to be working. I have to add at least 30 seconds or it doesn't startup. I'm going to test out a delay in the restart after I shut down, but just wanted to throw out there that after my stop service command, which happens at exactly 11:30, the restart failure in the event viewer has a timestamp of 11:30:06. That's not much of a delay between the stop and restart, so maybe it's still technically shutting down at this point. 6 seconds to shut down wouldn't be out of the ordinary. If a webapp is doing things properly, it will have some of its own shutdown code running to clean up resources, etc. that will take non-zero time. Most of the shutdown time will be the webapp and not Tomcat itself, which really just has to un-deploy the webapp and then stop the connectors. Yes, it is pretty common in the windows world to restart servers and services due to various memory issues. I'm doing this because the application is a reporting tool and if the users run any enormous reports that cause it to run out of java heap or something, this provides for an automated way of maintaining the application so that it doesn't require manual intervention to fix it and I know that even if no one calls me, it will be reset at some point. I also use it as an opportunity to save off a daily copy of the InetSoft logs if I need to go back and review errors later. Personally, I'm a Mac fan at heart. :-) I'm curious how long you could run without a restart. Usually for reporting types of webapps, it's either a single request that brings-down the server (e.g. it needs a huge amount of memory to run a single report) or you have lots
Re: Web service not starting up as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/12/12 1:48 PM, Debbie Shapiro wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] For my money, I'd off-load the actual report-generation to an out-of-process (duh) process and pick-up the result when it's done. That way, your webapp doesn't become unstable even if one report runs out of memory. Yes, in our situation, it's usually someone running a HUGE report that takes the server down. Your last paragraph, how would one off-load the report-generation? Is it possible to do this with a third party application? I don't have access to their code. I'm just implementing their solution. You'd have to talk to the vendor about that. Given that they don't seem very responsive to other issues (leaking memory like a sieve), they will probably tell you that their reporting solution is first-rate and there's no reason to take it out-of-band. Depending on the types of reports, you could look into using JasperReports (no-cost, OSS... we use it). You *do* have to write your own reports, but the framework is there and no webapp is required unless you want one. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9es0QACgkQ9CaO5/Lv0PD/dwCgt7lnSAwIsh0Bl0ozz0tRCocc O0wAnjLNVWIq22RnWONGI1GX9oXK4Xlt =v2wl -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
Hi Chris - -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, March 08, 2012 11:54 AM To: Tomcat Users List Subject: Re: Web service not starting up as expected -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/8/12 2:46 PM, Debbie Shapiro wrote: I recently upgraded my Tomcat to version 7.0.26. I'm using a third party application within Tomcat (InetSoft). All I do is place their web application folder within the Tomcat webapps directory. Most recently, I had 7.0.23 installed and it would start up as expected. I have a process each night that automatically restarts the services. In my web.xml, I have code that is supposed to automatically start it up (the automated process restarts it at 11:30PM) using the load-on-startup tag. servlet servlet-namereplets/servlet-name servlet-classinetsoft.sree.web.ServletRepository/servlet-class load-on-startup1/load-on-startup /servlet So, your load-on-startup servlet somehow restarts your services? When you say services, do you mean it restarts Tomcat? How does it do that? * No, it forces (or is supposed to) InetSoft's application to load when the service starts up. * Since the upgrade to 7.0.26, when we try to access the web app the next morning, it is not accessible. However, the same process I use to (automatically) restart it each evening, when I kick it off manually, I am then able to access the web app. There is nothing different in how I am manually kicking it off versus the automated process. I'm using the same .bat file (on Win2003 server) and the automated process is a simple Scheduled Task. My immediate guess would be that when you run it manually, you are logged-in as yourself, while the scheduled task runs as some other user. Any file permission issues that might be in play? * Nothing has changed on this server between having Tomcat 7.0.23 installed and upgrading to 7.0.26. I typically don't log out of these servers (though it might be locked), so the same user is logged in regardless of whether it is starting up automatically or a manual start up. Tomcat is using a Local System account for login. * The only difference I am seeing is that in the Catalina log, the automated process ends with: Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [ajp-bio-8009] That's Tomcat stopping. Is there more above that line? Perhaps some kind of exception and/or stack trace? * There are errors shown here. This is everything logged from the point of the automated restart. I mentioned the warnings about the memory leaks to InetSoft and they indicated that these are only warnings and that I shouldn't be concerned about them, but I see there are other errors here now as well. Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [ajp-bio-8009] Mar 7, 2012 11:30:01 PM org.apache.catalina.core.StandardService stopInternal INFO: Stopping service Catalina Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate destroy INFO: WSSERVLET15: JAX-WS servlet destroyed Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed INFO: WSSERVLET13: JAX-WS context listener destroyed Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/sree] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-8] but has failed to stop it. This is very likely to create
Re: Web service not starting up as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 1:09 PM, Debbie Shapiro wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] So, your load-on-startup servlet somehow restarts your services? When you say services, do you mean it restarts Tomcat? How does it do that? No, it forces (or is supposed to) InetSoft's application to load when the service starts up. Right: that's what's supposed to happen. I just wanted to make sure you understood the difference between starting the Windows service and starting your InetSoft servlet (which could be thought of as a service). My immediate guess would be that when you run it manually, you are logged-in as yourself, while the scheduled task runs as some other user. Any file permission issues that might be in play? Nothing has changed on this server between having Tomcat 7.0.23 installed and upgrading to 7.0.26. Famous last words :) I typically don't log out of these servers (though it might be locked), so the same user is logged in regardless of whether it is starting up automatically or a manual start up. Tomcat is using a Local System account for login. How do you start Tomcat when you start it manually? How about when you schedule a restart? The only difference I am seeing is that in the Catalina log, the automated process ends with: Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [ajp-bio-8009] That's Tomcat stopping. Is there more above that line? Perhaps some kind of exception and/or stack trace? There are errors shown here. This is everything logged from the point of the automated restart. I mentioned the warnings about the memory leaks to InetSoft and they indicated that these are only warnings and that I shouldn't be concerned about them, but I see there are other errors here now as well. Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [ajp-bio-8009] Mar 7, 2012 11:30:01 PM org.apache.catalina.core.StandardService stopInternal INFO: Stopping service Catalina Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate destroy INFO: WSSERVLET15: JAX-WS servlet destroyed Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed INFO: WSSERVLET13: JAX-WS context listener destroyed Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/sree] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-8] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/sree] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@400a3bdc]) and a value of type [inetsoft.util.CurrentException.Infomation] (value [inetsoft.util.CurrentException$Infomation@5fe0095c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/sree] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@400a3bdc]) and a value of type [inetsoft.util.CurrentException.Infomation] (value [inetsoft.util.CurrentException$Infomation@6d86a830]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/sree] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value
RE: Web service not starting up as expected
From: Debbie Shapiro [mailto:dshap...@cardiacscience.com] Subject: RE: Web service not starting up as expected 3. Schedule a restart 2 minutes into the future. Exactly what does your restart procedure consist of? From the symptoms, it could be that the restart mechanism is simply not waiting for Tomcat to terminate before kicking off a new one. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-5] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/sree] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1809b1ac]) and a value of type [inetsoft.util.CurrentException.Infomation] (value [inetsoft.util.CurrentException$Infomation@246f9f88]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Obviously, your webapp indulges in some rather anti-social behavior; that will slow down Tomcat termination, but not prevent it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, March 09, 2012 10:56 AM To: Tomcat Users List Subject: Re: Web service not starting up as expected -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 1:09 PM, Debbie Shapiro wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] So, your load-on-startup servlet somehow restarts your services? When you say services, do you mean it restarts Tomcat? How does it do that? No, it forces (or is supposed to) InetSoft's application to load when the service starts up. Right: that's what's supposed to happen. I just wanted to make sure you understood the difference between starting the Windows service and starting your InetSoft servlet (which could be thought of as a service). My immediate guess would be that when you run it manually, you are logged-in as yourself, while the scheduled task runs as some other user. Any file permission issues that might be in play? Nothing has changed on this server between having Tomcat 7.0.23 installed and upgrading to 7.0.26. Famous last words :) I'm pretty sure that's accurate in this case, however. I had 7.0.23 installed as of last week and then this week did the upgrade to 7.0.26. I'm wondering about your question regarding timing though. It's possible that Tomcat is doing more cleanup than it has in the past and it's taking longer to shut down? Maybe I do just need some kind of pause in between stopping and restarting. I typically don't log out of these servers (though it might be locked), so the same user is logged in regardless of whether it is starting up automatically or a manual start up. Tomcat is using a Local System account for login. How do you start Tomcat when you start it manually? How about when you schedule a restart? I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c sc \\bidev2 start Tomcat7 cd c:\Scheduled Jobs The only difference I am seeing is that in the Catalina log, the automated process ends with: Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [ajp-bio-8009] That's Tomcat stopping. Is there more above that line? Perhaps some kind of exception and/or stack trace? There are errors shown here. This is everything logged from the point of the automated restart. I mentioned the warnings about the memory leaks to InetSoft and they indicated that these are only warnings and that I shouldn't be concerned about them, but I see there are other errors here now as well. Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [ajp-bio-8009] Mar 7, 2012 11:30:01 PM org.apache.catalina.core.StandardService stopInternal INFO: Stopping service Catalina Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate destroy INFO: WSSERVLET15: JAX-WS servlet destroyed Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed INFO: WSSERVLET13: JAX-WS context listener destroyed Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/sree] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Mar 7, 2012 11:30:06 PM org.apache.catalina.loader.WebappClassLoader
RE: Web service not starting up as expected
Hi Chuck - Please create a HelpStar ticket for any requests for assistance. This will help us better track your request. http://helpstar Debbie Shapiro Data Warehouse Manager Cardiac Science Office: 425.402.2233 Visit us at www.cardiacscience.com Suppliers of Cardiac Science, Criticare, Unetixs, Powerheart, Burdick, and Quinton products Part of the Opto Circuits Family -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Friday, March 09, 2012 10:55 AM To: Tomcat Users List Subject: RE: Web service not starting up as expected From: Debbie Shapiro [mailto:dshap...@cardiacscience.com] Subject: RE: Web service not starting up as expected 3. Schedule a restart 2 minutes into the future. Exactly what does your restart procedure consist of? From the symptoms, it could be that the restart mechanism is simply not waiting for Tomcat to terminate before kicking off a new one. I use a bat file that contains the following. I'm going to see about adding in a wait. sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c sc \\bidev2 start Tomcat7 cd c:\Scheduled Jobs Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-5] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/sree] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak. Mar 8, 2012 4:37:04 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/sree] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1809b1ac]) and a value of type [inetsoft.util.CurrentException.Infomation] (value [inetsoft.util.CurrentException$Infomation@246f9f88]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Obviously, your webapp indulges in some rather anti-social behavior; that will slow down Tomcat termination, but not prevent it. Well, technically it's not MY webapp. It's a third party vendor. They ship with a different web server bundled with their app, but you can easily swap that out for tomcat, which has better performance. So, I don't really have any control for how they developed this. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Web service not starting up as expected
Debbie Shapiro wrote: ... How do you start Tomcat when you start it manually? How about when you schedule a restart? I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM Now wait.. I am not so familiar with these Windows commands, but do I see above a) a command to stop the Tomcat service b) a command killing java.exe ? REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c and c) a series of command tending to indicate that this application puts its logfiles inside the classes directory ? (not fatal, but at least bizarre) sc \\bidev2 start Tomcat7 cd c:\Scheduled Jobs ... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
I was able to add a sleep command (I had to do it for at least 30 seconds, any less and starting up was still overlapping with the previous stop command). After adding this in, it seems to be starting up without a problem. Glad it was something simple. Unfortunately, it forces me to install an additional Microsoft windows resource kit app on the server, but no biggie. Please create a HelpStar ticket for any requests for assistance. This will help us better track your request. http://helpstar Debbie Shapiro Data Warehouse Manager Cardiac Science Office: 425.402.2233 Visit us at www.cardiacscience.com Suppliers of Cardiac Science, Criticare, Unetixs, Powerheart, Burdick, and Quinton products Part of the Opto Circuits Family -Original Message- From: Debbie Shapiro Sent: Friday, March 09, 2012 12:58 PM To: Tomcat Users List Subject: RE: Web service not starting up as expected -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, March 09, 2012 10:56 AM To: Tomcat Users List Subject: Re: Web service not starting up as expected -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/9/12 1:09 PM, Debbie Shapiro wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] So, your load-on-startup servlet somehow restarts your services? When you say services, do you mean it restarts Tomcat? How does it do that? No, it forces (or is supposed to) InetSoft's application to load when the service starts up. Right: that's what's supposed to happen. I just wanted to make sure you understood the difference between starting the Windows service and starting your InetSoft servlet (which could be thought of as a service). My immediate guess would be that when you run it manually, you are logged-in as yourself, while the scheduled task runs as some other user. Any file permission issues that might be in play? Nothing has changed on this server between having Tomcat 7.0.23 installed and upgrading to 7.0.26. Famous last words :) I'm pretty sure that's accurate in this case, however. I had 7.0.23 installed as of last week and then this week did the upgrade to 7.0.26. I'm wondering about your question regarding timing though. It's possible that Tomcat is doing more cleanup than it has in the past and it's taking longer to shut down? Maybe I do just need some kind of pause in between stopping and restarting. I typically don't log out of these servers (though it might be locked), so the same user is logged in regardless of whether it is starting up automatically or a manual start up. Tomcat is using a Local System account for login. How do you start Tomcat when you start it manually? How about when you schedule a restart? I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c sc \\bidev2 start Tomcat7 cd c:\Scheduled Jobs The only difference I am seeing is that in the Catalina log, the automated process ends with: Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [ajp-bio-8009] That's Tomcat stopping. Is there more above that line? Perhaps some kind of exception and/or stack trace? There are errors shown here. This is everything logged from the point of the automated restart. I mentioned the warnings about the memory leaks to InetSoft and they indicated that these are only warnings and that I shouldn't be concerned about them, but I see there are other errors here now as well. Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:01 PM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler [ajp-bio-8009] Mar 7, 2012 11:30:01 PM org.apache.catalina.core.StandardService stopInternal INFO: Stopping service Catalina Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate destroy INFO: WSSERVLET15: JAX-WS servlet destroyed Mar 7, 2012 11:30:06 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed INFO: WSSERVLET13: JAX-WS context listener destroyed Mar 7, 2012 11:30:06 PM
Re: Web service not starting up as expected
Christopher Schultz wrote: ... I wonder if this is a timing issue. I've never used Tomcat on Windows, but I used to use an old Java-based TiVo media server that came with a Windows Service. I seem to remember that clicking the restart widget in the service snap-in would seem to take *forever* and then it would finally say sorry, this service isn't responding. For whatever reason, the service interface wasn't getting the feedback it needed to declare the service stopped. or, the service is just taking longer to respond, than what the service manager expects (there is some kind of built-in delay there, within which the service must respond something to the start service message). As I remember, this can be changed. Must be in the Registry somewhere. When the service takes too long to respond, the service manager pops up an error like you indicate above, but the service itself is not interrupted, and continues to start and eventually run. Of course, if you think it hasn't started, and do a start service again, then things may get somewhat out-of-sync and confusing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Web service not starting up as expected
-Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Friday, March 09, 2012 1:35 PM To: Tomcat Users List Subject: Re: Web service not starting up as expected Debbie Shapiro wrote: ... How do you start Tomcat when you start it manually? How about when you schedule a restart? I either directly run the .bat command or launch the scheduled task that normally runs the same .bat command. The contents of the .bat file is as follows: sc \\bidev2 stop Tomcat7 taskkill /F /FI IMAGENAME eq java.exe /FI USERNAME eq NT AUTHORITY\SYSTEM Now wait.. I am not so familiar with these Windows commands, but do I see above a) a command to stop the Tomcat service b) a command killing java.exe ? Yes. This code is sort of a holdover from another application that I could only kill the service by also killing the java.exe processes that it was tied to, otherwise it couldn't obtain the port because it was already in use. Probably not necessary for this application, but I figured it couldn't hurt. REM delay batch job to wait for services to stop for /F tokens=1-4 delims=/- %%A in ('date/T') do set DATE=%%B%%C%%D for /F tokens=1-4 delims=:., %%a in ('time/T') do set TIME=%%a%%b%%c cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes rename sree.log %DATE%_%TIME%_sree.log rename schedule.log %DATE%_%TIME%_schedule.log cd c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\sree\WEB-INF\classes compact %DATE%_%TIME%_sree.log /c compact %DATE%_%TIME%_schedule.log /c and c) a series of command tending to indicate that this application puts its logfiles inside the classes directory ? (not fatal, but at least bizarre) Yes, it does. This is the way the vendor coded it. I have brought it to their attention but they don't seem to see a problem with it. sc \\bidev2 start Tomcat7 cd c:\Scheduled Jobs ... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Web service not starting up as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Debbie, On 3/8/12 2:46 PM, Debbie Shapiro wrote: I recently upgraded my Tomcat to version 7.0.26. I'm using a third party application within Tomcat (InetSoft). All I do is place their web application folder within the Tomcat webapps directory. Most recently, I had 7.0.23 installed and it would start up as expected. I have a process each night that automatically restarts the services. In my web.xml, I have code that is supposed to automatically start it up (the automated process restarts it at 11:30PM) using the load-on-startup tag. servlet servlet-namereplets/servlet-name servlet-classinetsoft.sree.web.ServletRepository/servlet-class load-on-startup1/load-on-startup /servlet So, your load-on-startup servlet somehow restarts your services? When you say services, do you mean it restarts Tomcat? How does it do that? Since the upgrade to 7.0.26, when we try to access the web app the next morning, it is not accessible. However, the same process I use to (automatically) restart it each evening, when I kick it off manually, I am then able to access the web app. There is nothing different in how I am manually kicking it off versus the automated process. I'm using the same .bat file (on Win2003 server) and the automated process is a simple Scheduled Task. My immediate guess would be that when you run it manually, you are logged-in as yourself, while the scheduled task runs as some other user. Any file permission issues that might be in play? The only difference I am seeing is that in the Catalina log, the automated process ends with: Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [http-bio-8080] Mar 7, 2012 11:30:06 PM org.apache.coyote.AbstractProtocol stop INFO: Stopping ProtocolHandler [ajp-bio-8009] That's Tomcat stopping. Is there more above that line? Perhaps some kind of exception and/or stack trace? When I start it up manually, I see this: Mar 8, 2012 9:48:42 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Mar 8, 2012 9:48:42 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Mar 8, 2012 9:48:42 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 20377 ms That's Tomcat starting. So, for some reason, the load-on-startup doesn't seem to be working properly. Any ideas why? Is this a bug in 7.0.26? I think load-on-startup isn't your problem, because Tomcat itself does not appear to even be attempting to start (or fails to start, and shuts down). Try this: 1. Shut down Tomcat and remove all log files form wherever they are. 2. Start Tomcat manually. 3. Schedule a restart 2 minutes into the future. 4. Wait 2 minutes. 5. Confirm that Tomcat is *not* running (that's your complaint, right?) 6. Post back with full (sanitized if necessary) log files - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ZDmIACgkQ9CaO5/Lv0PBXBwCcCZp/tKQic9+42LeGq8VXeXxr hVYAn03Eh18eOavUQfdHoxzgRnHdQYj2 =ceNx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org