Tomcat is shutting down the web service when I deploy a new .war file. There is no documentation on the threads and what they are for in oracle or tomcats documentation.
The threads are not shutting down and they stay alive when the web service is shut down. I know this because I count the number of threads each time the web service context is destroyed and number keeps growing for each deployment of the .war file. Thank you, Chris Brown Systems Consultant National Council on Compensation Insurance 901 Peninsula Corporate Circle Boca Raton, FL 33487-1362 (P) 561-893-2324 chris_br...@ncci.com -----Original Message----- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, March 07, 2016 9:47 AM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service Shutdown Can't Stop Threads -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris, On 3/7/16 8:26 AM, Chris Brown wrote: > The .jar comes with the Oracle 12c client. Oracle JDBC Drivers > release 12.1.0.2.0 Thanks. > I upgraded to tomcat 8. The latest stable version. The issue still > occurred, but the messages in the logs were "Warnings" > instead of being "SEVERE" errors. You said that. How about taking a thread dump after your web application stops, to see a few things: 1. Whether this is a spurious notification, since Tomcat was just going down faster than the threads were able to (gracefully) shut down 2. What those threads are doing, especially the ones with the somewhat anonymous names You haven't said how you were shutting-down the thread pool. You also haven't said whether you read any or Oracle's documentation about what those threads are doing or how they should be terminated. - -chris > -----Original Message----- From: Christopher Schultz > [mailto:ch...@christopherschultz.net] Sent: Friday, March 04, 2016 > 4:33 PM To: Tomcat Users List Subject: Re: Tomcat 7 Web Service > Shutdown Can't Stop Threads > > Chris, > > On 3/4/16 2:13 PM, Chris Brown wrote: >> Tomcat version: 7.0.42 Server OS: Sun Solaris UCP Version: Oracle 12c >> Oracle Driver: ojdbc7 > > That doesn't look like an Oracle JDBC driver version I've seen before. > Can you clarify? > >> JDK version: 1.8.0_31 > > You might want to upgrade Tomcat and Java. Those are somewhat old. > >> Issue Description: When tomcat shuts down our web service during a >> hot deployment threads are left running that tomcat cannot shutdown. >> I believe the threads are from the universal connection pool. We use >> oracle's universal connection pool (ucp.jar). After multiple >> deployments the threads build up and cause high CPU usage from the 2 >> tomcat threads. > >> Log Error Examples: Feb 08, 2016 8:00:25 PM >> org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named [UCP-worker-thread-3] but has >> failed to stop it. This is very likely to create a memory leak. Feb >> 08, 2016 8:00:25 PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named [Thread-438] but has failed to >> stop it. This is very likely to create a memory leak. Feb 27, 2016 >> 6:15:14 PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application [/RatingService] >> appears to have started a thread named >> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockRelea s > >> er] >> but has failed to stop it. This is very likely to create a >> memory leak. Feb 27, 2016 6:15:14 PM >> org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application >> [/RatingService] appears to have started a thread named >> [ForkJoinPool.commonPool-worker-32] but has failed to stop it. >> This is very likely to create a memory leak. Feb 27, 2016 6:15:14 >> PM org.apache.catalina.loader.WebappClassLoader >> clearReferencesThreads SEVERE: The web application >> [/RatingService] appears to have started a thread named >> [Timer-121] but has failed to stop it. This is very likely to >> create a memory leak. > >> Correction made to code to try and fix issue: I added a >> ServletContextListener to our web service. On the >> contextDestroyed method we destroy the connection pools. Our web >> service uses 2 connection pools. > > How exactly are you destroying the connection pools? > >> I also upgraded to Tomcat 8 and the issue still occurred, but >> the messages are now warnings instead of SEVERE errors. > > That's just an effect of Tomcat downgrading the log level under > which it logs those issues. > >> I've searched the web and have tried many suggestions that have >> not helped except destroying the connection pools, but the issue >> still occurs. Please help!!! > > After you get those messages in the log, if you do a thread dump on > the JVM, do you still see those threads running? > > Generally speaking, Tomcat should be able to shut down even with > the above errors. Hopefully, none of those threads are non-daemon > threads, meaning that they shouldn't stop the JVM from stopping > once the JVM's main thread terminates. If your web application > leaves those threads running after deployment, then multiple > re-deployments without a Tomcat/JVM restart will likely result in a > large memory leak. (Those threads will pin the web application's > ClassLoader in memory along with all the classes it ever loaded, > and these will pile-up until you have exhausted your permgen or > other heap spaces). > >> [UCP-worker-thread-3] > > This is almost certainly something form Oracle's UCP > >> [Thread-438] > > This could be anything. A thread dump could tell you what it might > be. > > > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleas er > > ] > > Obviously, this is a thread from Oracle's JDBC driver. > >> [ForkJoinPool.commonPool-worker-32] > > Maybe something from UCP? Not sure. > >> [Timer-121] > > This is usually a thread started by TimerTask, a stock Java class. > But any component may have created this thread. It might be worth > searching your own code to see if a TimerTask is being launched and > never shut-down when your web application is shutting down. Make > arrangements to make sure that you shut these down when the > application is going out of service. > > For the Oracle-specific threads, read the documentation or search > online for how to stop those threads as the application is coming > down. There could be a bug in the library(ies) that might be fixed > in a more recent version. > > -chris > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > ________________________________ > > The information contained in this e-mail message is intended only > for the personal and confidential use of the recipient(s) named > above. This message may be an attorney-client communication and/or > work product and as such is privileged and confidential. If the > reader of this message is not the intended recipient or an agent > responsible for delivering it to the intended recipient, you are > hereby notified that you have received this document in error and > that any review, dissemination, distribution, or copying of this > message is strictly prohibited. If you have received this > communication in error, please notify us immediately by e-mail, and > delete the original message. > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlbdlFQACgkQ9CaO5/Lv0PDJTwCgnvouJT6lwijxCJnymRAP+SAe wFsAoKqbcrXDoiAV0RccWdFNIMHiE99N =YMav -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
