Hi,
I am attaching the thread dump. FYI, this is happening on several machines,
both with Oracle Java and OpenJDK.
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
and:
Openjdk version "1.8.0_222"OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
Thanks,Jim
On Tuesday, January 28, 2020, 7:34:16 AM UTC, Francesco Chicchiriccò
<[email protected]> wrote:
Hi, in my experience, such behavior by Tomcat can be investigated by
obtaining a thread dump after running shutdown, e.g. when the process hangs:
see
https://cwiki.apache.org/confluence/display/TOMCAT/HowTo#HowTo-HowdoIobtainathreaddumpofmyrunningwebapp?
for more details.
This is definitely an error, all-but-ordinary, situation that shouldn't
happen, I confirm there is no parameter to control such behavior.
Regards.
On 27/01/20 20:25, [email protected] wrote:
Hi,
We are seeing a problem where, after Syncope has been deployed to Tomcat, the
Tomcat shutdown.sh no longer seems to be shutting Tomcat down properly, and we
have to resort to using "kill".
I was looking into this, and it looks like when the shutdown.sh is run, it
DOES cause Tomcat to try to shutdown, and it looks like Tomcat does shutdown
the connections, so after running the shutdown.sh, the 8080 and 8005 ports are
no longer listening, but the Tomcat process is still running!!
When I look at the end of the catalina.out log file I see these lines:
27-Jan-2020 19:12:35.718 WARNING [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web
application [syncope-enduser] appears to have started a thread named [Thread-9]
but has failed to stop it. This is very likely to create a memory leak. Stack
trace of thread:
java.lang.Thread.sleep(Native Method)
org.apache.commons.io.monitor.FileAlterationMonitor.run(FileAlterationMonitor.java:189)
java.lang.Thread.run(Thread.java:748)
27-Jan-2020 19:12:35.725 INFO [main] org.apache.coyote.AbstractProtocol.stop
Stopping ProtocolHandler ["http-nio-8080"]
27-Jan-2020 19:12:35.728 INFO [main] org.apache.coyote.AbstractProtocol.stop
Stopping ProtocolHandler ["ajp-nio-8009"]
27-Jan-2020 19:12:35.731 INFO [main]
org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
["http-nio-8080"]
27-Jan-2020 19:12:35.731 INFO [main]
org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler
["ajp-nio-8009"]
From the above, I think that the Syncope apps are somehow preventing Tomcat
from shutting down completely and end up leaving the ports shutdown, but the
process is still running.
When that happens, and if whoever is working on it doesn't realize it (note:
if they do "netstat -an" the ports seem to stay in wait state for a long while,
then eventually disappear), and they startup Tomcat again, they will end up
having all kinds lof lock problems (which is what caused me to look into this).
Anyone seen this before? Also, is there something we need to do to the
Syncope configuration to allow it to allow Tomcat to shutdown properly?
FYI, we are using: - Java: openjdk version "1.8.0_222
- Tomcat: apache-tomcat-9.0.20 --
Francesco Chicchiriccò
Tirasa - Open Source Excellence
http://www.tirasa.net/
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/ Attaching to process ID 23211, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.231-b11
Deadlock Detection:
No deadlocks found.
Thread 23212: (state = BLOCKED)
Thread 23245: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
- com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(long,
com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence,
com.lmax.disruptor.SequenceBarrier) @bci=46, line=38 (Compiled frame)
- com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long) @bci=18, line=56
(Compiled frame)
- com.lmax.disruptor.BatchEventProcessor.processEvents() @bci=17, line=159
(Interpreted frame)
- com.lmax.disruptor.BatchEventProcessor.run() @bci=37, line=125 (Interpreted
frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23239: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.apache.commons.io.monitor.FileAlterationMonitor.run() @bci=57, line=189
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23238: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.apache.commons.io.monitor.FileAlterationMonitor.run() @bci=57, line=189
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23236: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
- com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(long,
com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence,
com.lmax.disruptor.SequenceBarrier) @bci=46, line=38 (Compiled frame)
- com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long) @bci=18, line=56
(Compiled frame)
- com.lmax.disruptor.BatchEventProcessor.processEvents() @bci=17, line=159
(Interpreted frame)
- com.lmax.disruptor.BatchEventProcessor.run() @bci=37, line=125 (Interpreted
frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23233: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
- com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(long,
com.lmax.disruptor.Sequence, com.lmax.disruptor.Sequence,
com.lmax.disruptor.SequenceBarrier) @bci=46, line=38 (Compiled frame)
- com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long) @bci=18, line=56
(Compiled frame)
- com.lmax.disruptor.BatchEventProcessor.processEvents() @bci=17, line=159
(Interpreted frame)
- com.lmax.disruptor.BatchEventProcessor.run() @bci=37, line=125 (Interpreted
frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23226: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- sun.misc.GC$Daemon.run() @bci=51, line=117 (Interpreted frame)
Thread 23225: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run() @bci=10, line=85
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1149 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23224: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Interpreted frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074
(Interpreted frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Thread 23223: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
- java.util.concurrent.LinkedBlockingDeque.pollFirst(long,
java.util.concurrent.TimeUnit) @bci=52, line=522 (Compiled frame)
- java.util.concurrent.LinkedBlockingDeque.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=684 (Compiled frame)
- org.apache.juli.AsyncFileHandler$LoggerThread.run() @bci=10, line=159
(Interpreted frame)
Thread 23218: (state = BLOCKED)
Thread 23217: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=216 (Compiled
frame)
Thread 23216: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
- java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191
(Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Compiled
frame)
