Chris,
Sorry - didn't realize that images would be stripped off.
The number of TIMED_WAITING threads starts at 150 or so and every hour
increases by 10.
Below is an example of where the threads are stuck (by looking at the JVM
stack):
https-jsse-nio2-443-exec-90
TIMED_WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48986fd7
jdk.internal.misc.Unsafe.park(Native Method);
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252);
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674);
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1113);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1175);
app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659);
app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63);
java.base@17.0.12/java.lang.Thread.run(Thread.java:840);
The connector configuration is:
<Connector protocol="org.apache.coyote.http11.Http11Nio2Protocol"
port="443" maxThreads="1536" minSpareThreads="64"
sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
scheme="https" secure="true" SSLEnabled="true"
enableLookups="false" acceptCount="12000"
connectionTimeout="60000" socket.rxBufSize="131072" socket.txBufSize="131072"
maxHttpHeaderSize="131072" maxPostSize="-1" maxKeepAliveRequests="1"
maxConnections="1536" URIEncoding="UTF-8" useSendfile="false"
tcpNoDelay="true" compression="force"
compressibleMimeType="font/woff,font/woff2,text/html,text/xml,text/plain,application/x-java-applet,application/octet-stream,application/xml,text/javascript,text/css,image/png,image/bmp,image/jpeg,image/gif,application/pdf,application/x-javascript,application/javascript,application/json,application/x-shockwave-flash,application/xhtml+xml,application/xml+xhtml"
noCompressionUserAgents="gozilla, traviata">
<SSLHostConfig honorCipherOrder="true"
ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
certificateVerification="none" protocols="TLSv1.2"
sslProtocol="TLS"><Certificate certificateKeystoreFile="..."
certificateKeyAlias="..." certificateKeystorePassword="..." /></SSLHostConfig>
</Connector>
Regards,
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, September 30, 2024 10:38 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
[You don't often get email from ch...@christopherschultz.net. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.
John,
On 9/30/24 10:30, John Williams wrote:
> The issue is that the number of threads in the TIMED_WAIT state keeps
> increasing over time. Starts at 150 and keeps growing by 10 every hour.
> Once it reaches close to the maxThreads setting we see connection drops.
>
> This application was working fine with Tomcat 10.0 and JDK 12. Unless
> the application was seeing load, threads would not be created in
> Tomcat's JVM.
>
> You can also see the thread stacks in this image.
This list strips out image. Can you post text?
Please post your <Connector> configuration taking care to remove any secrets.
Thanks,
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, September 30, 2024 10:20 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads
>
> [You don't often get email from ch...@christopherschultz.net
> <mailto:ch...@christopherschultz.net>. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka
> .ms%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C5c93ec3de0884e786
> 7d708dce15d7e73%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C638633038
> 912339442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6q8HQHLjceSHYLqEnLN9
> ue%2Fpa3zz5cougsajpZIsjDw%3D&reserved=0
> LearnAboutSenderIdentification> ]
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender
> and know the content is safe.
>
> John,
>
> On 9/28/24 05:38, John Williams wrote:
>
> >
>
> > Hi Everyone,
>
> >
>
> > I am running Apache Tomcat 10.1.30 for a web application and notice
> ~950 timed_waiting threads.
>
> >
>
> > The stack trace for these threads is below:
>
> >
>
> > java.base@17.0.12/
> jdk.internal.misc.Unsafe.park(Native<mailto:java.bas
> <mailto:java.base@17.0.12/
> jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas>
>
> > e@17.0.12/jdk.internal.misc.Unsafe.park(Native <mailto:e@17.0.12/
> jdk.internal.misc.Unsafe.park(Native>> Method);
>
> > java.base@17.0.12/
> java.util.concurrent.locks.LockSupport.parkNanos(Loc
> <mailto:java.base@17.0.12/
> java.util.concurrent.locks.LockSupport.parkNanos(Loc>
>
> >
> kSupport.java:252)<mailto:java.base@17.0.12/java.util.concurrent.locks
>
> > .LockSupport.parkNanos(LockSupport.java:252)>;
>
> > java.base@17.0.12/
> java.util.concurrent.locks.AbstractQueuedSynchronize
> <mailto:java.base@17.0.12/
> java.util.concurrent.locks.AbstractQueuedSynchronize>
>
> >
> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)<mai
>
> >
> lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchro
>
> >
> nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)
>
> > >;
>
> > java.base@17.0.12/
> java.util.concurrent.LinkedBlockingQueue.poll(Linked
> <mailto:java.base@17.0.12/
> java.util.concurrent.LinkedBlockingQueue.poll(Linked>
>
> > BlockingQueue.java:460)<mailto:java.base@17.0.12/java.util.concurrent.
>
> > LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>;
>
> >
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99);
>
> >
> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33);
>
> >
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP
>
> > oolExecutor.java:1113);
>
> >
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa
>
> > dPoolExecutor.java:1175);
>
> >
> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre
>
> > adPoolExecutor.java:659);
>
> >
> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
>
> > skThread.java:63);
>
> > java.base@17.0.12/
> java.lang.Thread.run(Thread.java:840)<mailto:java.ba
> <mailto:java.base@17.0.12/
> java.lang.Thread.run(Thread.java:840)%3cmailto:java.ba>
>
> > se@17.0.12/java.lang.Thread.run(Thread.java:840)
> <mailto:se@17.0.12/ java.lang.Thread.run(Thread.java:840)>>;
>
> >
>
> > I am using an executor with maxThreads set to 1500 and
> minSpareThreads set to 64.
>
> >
>
> > I have not seen this issue with Tomcat 10.0.
>
> >
>
> > What could be the issue and how to resolve it? Any help is much
> appreciated.
>
> This looks like you have a large number of threads waiting for work.
>
> That seems normal to me if you have a max thread pool of 1500 threads.
>
> What's the problem?
>
> -chris
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> <mailto:users-unsubscr...@tomcat.apache.org>
>
> For additional commands, e-mail: users-h...@tomcat.apache.org
> <mailto:users-h...@tomcat.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
