Hello Everyone, I was investigating one thread pool exhaustion issue. Thread dump analysis showed that all HTTP threads were waiting for a ReentrantLock object. Object address 0x000000066d727f28 were same for all of the waiting threads:
"http-nio-18100-exec-86" #32808 daemon prio=5 os_prio=0 tid=0x0000000051835800 nid=0x29bc waiting on condition [0x000000007a5be000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000066d727f28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2385) at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1274) There was no hint in the thread dump about which thread was owning the lock. Luckily, one heap dump was taken before generating thread dump. When I queried the heap dump for that ReentrantLock object, I saw that another thread(http-nio-18100-exec-4) was holding the lock(exclusiveOwnerThread). There was NO trace of http-nio-18100-exec-4 thread in any of the thread dumps! So it was a "lock without an owner" case. After glancing through the Tomcat’s JNDIRealm.get() code and beyond[1], I can see lock is being acquired on singleConnectionLock. That lock is getting released either in the close() or release() method. So, if something bad happens to the thread which is trying to establish a connection, then lock will be held without a proper owner and a thread blocking situation will be created. Am I interpreting the code correctly? Should we not handle any failure inside get()? Also, I still have not got the reason why the thread got terminated. Any suggestions on how I can enable any specific logging? My setup is: Tomcat version: 9.0.39 Connector: NIO JDK: AdoptOpenJDK: 1.8.192 OS: Windows 2016 [1] https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/catalina/realm/JNDIRealm.java#L2553 Thanks & Regards, Suvendu --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
