Further, if we do not run OM5, Tomcat shuts down properly. Environment is CentOS 8 (using docker) and MariaDB 10.3.x
From: Coscend@OM [mailto:[email protected]] Sent: Friday, November 22, 2019 12:12 PM To: [email protected] Subject: RE: OM5: Tomcat not shutting down due to OM5 method Dear Maxim, I am sending this to users, but we think one possibility (not confirmation) is this error is related to OM5 code [1]. There is a method in OM5 that is triggering memory leak while shutting down Tomcat. The most common reason is something hanging or delaying in Web application initializations [2]. This prevents Tomcat to properly shutdown [1, 2]. Or, is due to something else? (We are using OM5 binary from OM Web site. There are no customization or configuration changes.) Here is the relevant log. The key line in the log is: “The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak.” ------------------------ 21-Nov-2019 23:24:30.024 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:719) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)> 21-Nov-2019 23:24:30.025 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555) [1] https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005 [2] https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1 Thank you. Sincerely, Hemant K. Sabat <http://www.coscend.com/> www.Coscend.com ------------------------------------------------------------------ Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly… ------------------------------------------------------------------ CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html From: Maxim Solodovnik [mailto:[email protected]] Sent: Friday, November 22, 2019 11:36 AM To: dev <[email protected] <mailto:[email protected]> >; [email protected] <mailto:[email protected]> Subject: Re: OM5: Tomcat not shutting down due to OM5 method Please send these messages to users@ It is not about OM/OM_community development shutdown works for me as expected please provide additional details (if any) On Fri, 22 Nov 2019 at 13:04, Coscend@OM <[email protected] <mailto:[email protected]> > wrote: Dear OM Community, While shutting down OM5-Tomcat via shutdown.sh, Tomcat is not shutting down. These links state some method in Webapp could be preventing the shutdown process of Tomcat: https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shu <https://stackoverflow.com/questions/18340873/why-tomcat-will-not-bind-to-shutdown-port-8005> tdown-port-8005 https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listen <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1> ing?noredirect=1 <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-liste <https://stackoverflow.com/questions/2217382/tomcat6-shutdown-port-not-listening?noredirect=1&lq=1> ning?noredirect=1&lq=1> &lq=1 Could anyone provide any guidance what we need to modify in OM5 code or any other place to properly shutdown Tomcat? Here is the log snippet. There are several more such errors while shutting down. ------------------------------ 21-Nov-2019 23:24:29.843 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance. 21-Nov-2019 23:24:29.844 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-5080"] 21-Nov-2019 23:24:29.850 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"] 21-Nov-2019 23:24:29.856 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina] [34mINFO [0;39m 11-21 23:24:29.884 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket extensions initializer [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket core library initializer [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: DevUtils DebugBar Initializer [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: org.wicketstuff.dashboard.DashboardContextInitializer@72690a1 <mailto:org.wicketstuff.dashboard.DashboardContextInitializer@72690a1> [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket jQuery UI initializer (plugins) [34mINFO [0;39m 11-21 23:24:29.885 [36mo.a.w.Application:539 [main][0;39m - [OpenmeetingsApplication] destroy: Wicket Kendo UI initializer (theme-default) 21-Nov-2019 23:24:29.886 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] [172.17.0.1]:5701 is SHUTTING_DOWN 21-Nov-2019 23:24:29.891 INFO [hz.server-1.cached.thread-2] com.hazelcast.internal.partition.impl.MigrationManager.null [172.17.0.1]:5701 [dev] [3.12] Shutdown request of Member [172.17.0.1]:5701 - 799babea-5e78-4cf9-a099-f4b1fb09b9c9 this is handled 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12] Shutting down connection manager... 21-Nov-2019 23:24:29.965 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12] Shutting down node engine... 21-Nov-2019 23:24:29.979 INFO [main] com.hazelcast.instance.NodeExtension.null [172.17.0.1]:5701 [dev] [3.12] Destroying node NodeExtension. 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.instance.Node.null [172.17.0.1]:5701 [dev] [3.12] Hazelcast Shutdown is completed in 90 ms. 21-Nov-2019 23:24:29.980 INFO [main] com.hazelcast.core.LifecycleService.null [172.17.0.1]:5701 [dev] [3.12] [172.17.0.1]:5701 is SHUTDOWN 21-Nov-2019 23:24:30.023 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [Openmeetings] registered the JDBC driver [com.mysql.cj.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. 21-Nov-2019 23:24:30.024 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [nioEventLoopGroup-2-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@11/java.lang.Thread.sleep(Native <mailto:java.base@11/java.lang.Thread.sleep(Native> Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThr eadEventExecutor.java:719) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:505) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventEx ecutor.java:905) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable .java:30) java.base@11/java.lang.Thread.run(Thread.java:834) <mailto:java.base@11/java.lang.Thread.run(Thread.java:834)> 21-Nov-2019 23:24:30.025 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [Openmeetings] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@11/java.lang.Object.wait(Native <mailto:java.base@11/java.lang.Object.wait(Native> Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:555 ) Thank you. Sincerely, Hemant K. Sabat <http://www.coscend.com/> www.Coscend.com <http://www.Coscend.com> ------------------------------------------------------------------ Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly. ------------------------------------------------------------------ CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: <http://www.coscend.com/Anchor/Common/Terms_and_Conditions.html> http://www.Coscend.com/Anchor/Common/Terms_and_Conditions.html -- WBR Maxim aka solomax
