Re: tomcat thread incurring CPU load

2019-11-19 Thread Mark Thomas
On 19/11/2019 11:16, M. Manna wrote: > Sorry Dumb question earlier - It is in Coyote jar. > > We are currently in a rollback state in our prod, but what you reported > above sounds promising. If revert the Coyote jar resolves issues, which is > the earliest/latest version we can revert to? I

Re: tomcat thread incurring CPU load

2019-11-19 Thread M. Manna
Sorry Dumb question earlier - It is in Coyote jar. We are currently in a rollback state in our prod, but what you reported above sounds promising. If revert the Coyote jar resolves issues, which is the earliest/latest version we can revert to? Thanks, MAnna On Tue, 19 Nov 2019 at 10:32, M.

Re: tomcat thread incurring CPU load

2019-11-19 Thread M. Manna
Mark, On Mon, 18 Nov 2019 at 19:28, Mark Thomas wrote: > On 18/11/2019 14:14, Mark Thomas wrote: > > On 18/11/2019 12:06, M. Manna wrote: > >> Mark and others, > >> > >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> > >>> OK, it looks like I can reproduce this. > >>> > >>> Steps to

Re: tomcat thread incurring CPU load

2019-11-18 Thread Mark Thomas
On 18/11/2019 14:14, Mark Thomas wrote: > On 18/11/2019 12:06, M. Manna wrote: >> Mark and others, >> >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: >> >>> OK, it looks like I can reproduce this. >>> >>> Steps to reproduce: >>> >>> - Windows 2016 Server fully patched >>> - Java 1.8.0u144 >>>

Re: tomcat thread incurring CPU load

2019-11-18 Thread Mark Thomas
On 18/11/2019 12:06, M. Manna wrote: > Mark and others, > > On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> OK, it looks like I can reproduce this. >> >> Steps to reproduce: >> >> - Windows 2016 Server fully patched >> - Java 1.8.0u144 >> - Install Tomcat 8.5.45 from windows installer >>

Re: tomcat thread incurring CPU load

2019-11-18 Thread M. Manna
Mark and others, On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > OK, it looks like I can reproduce this. > > Steps to reproduce: > > - Windows 2016 Server fully patched > - Java 1.8.0u144 > - Install Tomcat 8.5.45 from windows installer > - Add tcnative-1.dll (64-bit) from Tomcat Native

Re: tomcat thread incurring CPU load

2019-11-18 Thread Mark Thomas
OK, it looks like I can reproduce this. Steps to reproduce: - Windows 2016 Server fully patched - Java 1.8.0u144 - Install Tomcat 8.5.45 from windows installer - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 - Modify server.xml to use Http11AprProtocol on port 8080 - Make a single

Re: tomcat thread incurring CPU load

2019-11-13 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark and M, On 11/13/19 19:31, Mark Thomas wrote: > On November 13, 2019 11:42:34 PM UTC, "M. Manna" > wrote: >> I see this update on Windows which may have been responsible >> (suspicion only, haven’t rolled it back yet) >> >> >>

Re: tomcat thread incurring CPU load

2019-11-13 Thread Mark Thomas
On November 13, 2019 11:42:34 PM UTC, "M. Manna" wrote: >I see this update on Windows which may have been responsible (suspicion >only, haven’t rolled it back yet) > > >https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates > >Was 8.5.45 built on Windows 10 in presence

Re: tomcat thread incurring CPU load

2019-11-13 Thread M. Manna
I see this update on Windows which may have been responsible (suspicion only, haven’t rolled it back yet) https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates Was 8.5.45 built on Windows 10 in presence of this update ? Thanks, On Wed, 13 Nov 2019 at 17:55, M.

Re: tomcat thread incurring CPU load

2019-11-13 Thread M. Manna
Hi Chris, On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 11/13/19 11:20, M. Manna wrote: > > HI Mark, > > > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > > wrote: > > > >> On 12/11/2019 19:11,

Re: tomcat thread incurring CPU load

2019-11-13 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/13/19 11:20, M. Manna wrote: > HI Mark, > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > wrote: > >> On 12/11/2019 19:11, M. Manna wrote: >>> HI Mark, >>> >>> following my previous reply, we have now confirmed that it's >>> indeed >> 8.5.45

Re: tomcat thread incurring CPU load

2019-11-13 Thread M. Manna
HI Mark, On Wed, 13 Nov 2019 at 15:38, Mark Thomas wrote: > On 12/11/2019 19:11, M. Manna wrote: > > HI Mark, > > > > following my previous reply, we have now confirmed that it's indeed > 8.5.45 > > with APR 1.2.23 that's causing such high JVM CPU usage. > > We used took out 2 out of 50 servers

Re: tomcat thread incurring CPU load

2019-11-13 Thread Mark Thomas
On 12/11/2019 19:11, M. Manna wrote: > HI Mark, > > following my previous reply, we have now confirmed that it's indeed 8.5.45 > with APR 1.2.23 that's causing such high JVM CPU usage. > We used took out 2 out of 50 servers from the load balancer config, > reverted tomcat, and redeployed. With

Re: tomcat thread incurring CPU load

2019-11-12 Thread M. Manna
HI Mark, following my previous reply, we have now confirmed that it's indeed 8.5.45 with APR 1.2.23 that's causing such high JVM CPU usage. We used took out 2 out of 50 servers from the load balancer config, reverted tomcat, and redeployed. With near to identical user traffic, the two servers are

Re: tomcat thread incurring CPU load

2019-11-12 Thread M. Manna
Hey Mark (appreciate your response in US holiday time) On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > On November 12, 2019 12:54:53 AM UTC, "M. Manna" > wrote: > >Just to give an update again: > > > >1) We reverted the APR to 1.2.21 - but observed no difference. > >2) We took 3 thread

Re: tomcat thread incurring CPU load

2019-11-11 Thread Mark Thomas
On November 12, 2019 12:54:53 AM UTC, "M. Manna" wrote: >Just to give an update again: > >1) We reverted the APR to 1.2.21 - but observed no difference. >2) We took 3 thread dumps over 1 min interval (without any user >sessions) - >All threads are tomcat's internal pool threads. > >When we

Re: tomcat thread incurring CPU load

2019-11-11 Thread M. Manna
Just to give an update again: 1) We reverted the APR to 1.2.21 - but observed no difference. 2) We took 3 thread dumps over 1 min interval (without any user sessions) - All threads are tomcat's internal pool threads. When we checked the thread details (using fasthread.io) - we didn't see any of

Re: tomcat thread incurring CPU load

2019-11-11 Thread M. Manna
Hello All, Any thoughts regarding this? Slightly clueless at this point, so any direction will be appreciated. We are seeing the poll taking all the CPU time. We are using OperatingSystemMXBean.getProcessCpuLoad() and OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then x100 to get

tomcat thread incurring CPU load

2019-11-11 Thread M. Manna
Hello, after migrating to 8.5.45, we are seeing a lot of cpu load by following JVM thread dump: "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : cpu=172902703125000 : cpuLoad= 74.181015 BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 LockOwnerName:null WaitedCount:5397