Re: tomcat thread incurring CPU load
On 19/11/2019 11:16, M. Manna wrote: > Sorry Dumb question earlier - It is in Coyote jar. > > We are currently in a rollback state in our prod, but what you reported > above sounds promising. If revert the Coyote jar resolves issues, which is > the earliest/latest version we can revert to? I wouldn't recommend reverting just the Coyote JAR. It might work, but such configurations are a) untested and b) unsupported. 8.5.42 is the most recent version without the regression until 8.0.50 is released. Alternatively, reducing maxConnections (default is 8192) to 2048 or possibly 3072 should also work around it. Mark > > Thanks, > MAnna > > On Tue, 19 Nov 2019 at 10:32, M. Manna wrote: > >> Mark, >> >> On Mon, 18 Nov 2019 at 19:28, Mark Thomas wrote: >> >>> On 18/11/2019 14:14, Mark Thomas wrote: On 18/11/2019 12:06, M. Manna wrote: > Mark and others, > > On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> OK, it looks like I can reproduce this. >> >> Steps to reproduce: >> >> - Windows 2016 Server fully patched >> - Java 1.8.0u144 >> - Install Tomcat 8.5.45 from windows installer >> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >> - Modify server.xml to use Http11AprProtocol on port 8080 >> - Make a single request >> >> I then see 1 core running at 100% until the connection times out after >> 20s. Make another request and a core goes back up to 100% for 20s (the >> default keep-alive time out). >> > > I have also successfully reproduced this with making a single request > (sorry for not replying in the weekend). Not sure how your graph looked > like, but the Jvisualvm showed me a Sinusoidal modulation curve as >>> soon as > the request hit the server. and it didn't go down at all. I see similar behaviour on Windows 7 but the the CPU usage drops after >>> ~5s. A binary search indicates that the issue was introduced with this >>> commit: >>> https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) However, that code was removed when APR was switched to a single poll set. >>> >>> Ah ha. It was removed in 9.0.x but not in 8.5.x (only 9.0.x switched to >>> a single Poller) so it does look like this change is responsible. >>> >>> Thanks for Debugging this. Are you saying that we will have a patch for >> this? If so, which jar could we replace in 8.5.45 until >> we move to new tomcat? >> >> Thanks, >> >> >>> Mark >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
Sorry Dumb question earlier - It is in Coyote jar. We are currently in a rollback state in our prod, but what you reported above sounds promising. If revert the Coyote jar resolves issues, which is the earliest/latest version we can revert to? Thanks, MAnna On Tue, 19 Nov 2019 at 10:32, M. Manna wrote: > Mark, > > On Mon, 18 Nov 2019 at 19:28, Mark Thomas wrote: > >> On 18/11/2019 14:14, Mark Thomas wrote: >> > On 18/11/2019 12:06, M. Manna wrote: >> >> Mark and others, >> >> >> >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: >> >> >> >>> OK, it looks like I can reproduce this. >> >>> >> >>> Steps to reproduce: >> >>> >> >>> - Windows 2016 Server fully patched >> >>> - Java 1.8.0u144 >> >>> - Install Tomcat 8.5.45 from windows installer >> >>> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >> >>> - Modify server.xml to use Http11AprProtocol on port 8080 >> >>> - Make a single request >> >>> >> >>> I then see 1 core running at 100% until the connection times out after >> >>> 20s. Make another request and a core goes back up to 100% for 20s (the >> >>> default keep-alive time out). >> >>> >> >> >> >> I have also successfully reproduced this with making a single request >> >> (sorry for not replying in the weekend). Not sure how your graph looked >> >> like, but the Jvisualvm showed me a Sinusoidal modulation curve as >> soon as >> >> the request hit the server. and it didn't go down at all. >> > >> > I see similar behaviour on Windows 7 but the the CPU usage drops after >> ~5s. >> > >> > A binary search indicates that the issue was introduced with this >> commit: >> > >> > >> https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 >> > >> > (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) >> > >> > However, that code was removed when APR was switched to a single poll >> > set. >> >> Ah ha. It was removed in 9.0.x but not in 8.5.x (only 9.0.x switched to >> a single Poller) so it does look like this change is responsible. >> >> Thanks for Debugging this. Are you saying that we will have a patch for > this? If so, which jar could we replace in 8.5.45 until > we move to new tomcat? > > Thanks, > > >> Mark >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: tomcat thread incurring CPU load
Mark, On Mon, 18 Nov 2019 at 19:28, Mark Thomas wrote: > On 18/11/2019 14:14, Mark Thomas wrote: > > On 18/11/2019 12:06, M. Manna wrote: > >> Mark and others, > >> > >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> > >>> OK, it looks like I can reproduce this. > >>> > >>> Steps to reproduce: > >>> > >>> - Windows 2016 Server fully patched > >>> - Java 1.8.0u144 > >>> - Install Tomcat 8.5.45 from windows installer > >>> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 > >>> - Modify server.xml to use Http11AprProtocol on port 8080 > >>> - Make a single request > >>> > >>> I then see 1 core running at 100% until the connection times out after > >>> 20s. Make another request and a core goes back up to 100% for 20s (the > >>> default keep-alive time out). > >>> > >> > >> I have also successfully reproduced this with making a single request > >> (sorry for not replying in the weekend). Not sure how your graph looked > >> like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon > as > >> the request hit the server. and it didn't go down at all. > > > > I see similar behaviour on Windows 7 but the the CPU usage drops after > ~5s. > > > > A binary search indicates that the issue was introduced with this commit: > > > > > https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 > > > > (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) > > > > However, that code was removed when APR was switched to a single poll > > set. > > Ah ha. It was removed in 9.0.x but not in 8.5.x (only 9.0.x switched to > a single Poller) so it does look like this change is responsible. > > Thanks for Debugging this. Are you saying that we will have a patch for this? If so, which jar could we replace in 8.5.45 until we move to new tomcat? Thanks, > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: tomcat thread incurring CPU load
On 18/11/2019 14:14, Mark Thomas wrote: > On 18/11/2019 12:06, M. Manna wrote: >> Mark and others, >> >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: >> >>> OK, it looks like I can reproduce this. >>> >>> Steps to reproduce: >>> >>> - Windows 2016 Server fully patched >>> - Java 1.8.0u144 >>> - Install Tomcat 8.5.45 from windows installer >>> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >>> - Modify server.xml to use Http11AprProtocol on port 8080 >>> - Make a single request >>> >>> I then see 1 core running at 100% until the connection times out after >>> 20s. Make another request and a core goes back up to 100% for 20s (the >>> default keep-alive time out). >>> >> >> I have also successfully reproduced this with making a single request >> (sorry for not replying in the weekend). Not sure how your graph looked >> like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as >> the request hit the server. and it didn't go down at all. > > I see similar behaviour on Windows 7 but the the CPU usage drops after ~5s. > > A binary search indicates that the issue was introduced with this commit: > > https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 > > (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) > > However, that code was removed when APR was switched to a single poll > set. Ah ha. It was removed in 9.0.x but not in 8.5.x (only 9.0.x switched to a single Poller) so it does look like this change is responsible. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
On 18/11/2019 12:06, M. Manna wrote: > Mark and others, > > On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> OK, it looks like I can reproduce this. >> >> Steps to reproduce: >> >> - Windows 2016 Server fully patched >> - Java 1.8.0u144 >> - Install Tomcat 8.5.45 from windows installer >> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >> - Modify server.xml to use Http11AprProtocol on port 8080 >> - Make a single request >> >> I then see 1 core running at 100% until the connection times out after >> 20s. Make another request and a core goes back up to 100% for 20s (the >> default keep-alive time out). >> > > I have also successfully reproduced this with making a single request > (sorry for not replying in the weekend). Not sure how your graph looked > like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as > the request hit the server. and it didn't go down at all. I see similar behaviour on Windows 7 but the the CPU usage drops after ~5s. A binary search indicates that the issue was introduced with this commit: https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) However, that code was removed when APR was switched to a single poll set. Also, that the issue does not happens on all platforms (I can't repeat this on Linux) suggests the issue may not be in Java code. More research required... Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
Mark and others, On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > OK, it looks like I can reproduce this. > > Steps to reproduce: > > - Windows 2016 Server fully patched > - Java 1.8.0u144 > - Install Tomcat 8.5.45 from windows installer > - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 > - Modify server.xml to use Http11AprProtocol on port 8080 > - Make a single request > > I then see 1 core running at 100% until the connection times out after > 20s. Make another request and a core goes back up to 100% for 20s (the > default keep-alive time out). > I have also successfully reproduced this with making a single request (sorry for not replying in the weekend). Not sure how your graph looked like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as the request hit the server. and it didn't go down at all. Thanks, > > Next steps are to try and track down the root cause. > > Mark > > > > > Mark and M, > > > > On 11/13/19 19:31, Mark Thomas wrote: > >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" > >> wrote: > >>> I see this update on Windows which may have been responsible > >>> (suspicion only, haven’t rolled it back yet) > >>> > >>> > >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > > ocode-updates > >>> > >>> > >>> > > Was 8.5.45 built on Windows 10 in presence of this update ? > > > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully > >> patched at the time of the build Windows 7 64-bit VM. > > Also it doesn't matter because binaries don't include CPU microcode. > > > > It's more likely that the target system has microcode updates such as > > these that may negatively impact performance. > > > > -chris > > > >>> > >>> Thanks, > >>> > >>> On Wed, 13 Nov 2019 at 17:55, M. Manna > >>> wrote: > >>> > Hi Chris, > > On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > >> On 11/13/19 11:20, M. Manna wrote: > >>> HI Mark, > >>> > >>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas > >>> wrote: > >>> > On 12/11/2019 19:11, M. Manna wrote: > > HI Mark, > > > > following my previous reply, we have now confirmed > > that it's indeed > 8.5.45 > > with APR 1.2.23 that's causing such high JVM CPU > > usage. We used took out 2 out of 50 servers from the > > load balancer config, reverted tomcat, and > > redeployed. With near to identical user traffic, the > > two servers are responding normally without/without > > traffic with 8.5.41. The JVM dump looks a lot better > > with 8.5.41. > > > > We do think that the recent changes in APR and some > > other tomcat jar may have caused compatibility issue > > on Windows server 2016 (64-bit) platform. But > > unfortunately, we cannot pinpoint exactly what change > > may have caused this (i.e. actual OS vs Security > > Updates). With this in mind, we are also being wary > > to move to 8.5.47 as we don't know if the same issue > > will > occur > > again. Since 8.5.41 has been packaged with previously > > accepted > application > > installer, we are more comfortable rolling back. > > Just to confirm, you see this high CPU usage with a > clean install (no additional web applications deployed, > no configuration changes) on Windows 2016 DataCenter > (64-bit)? > > If this is the case, it should be fairly easy to > reproduce. > > Mark > > We do not deploy multiple applications. In fact, Under > tomcat > >>> webapps/ROOT we only have one application (ours). Each > >>> tomcat instance is hosted on a VM (total 50) and all of > >>> them are identically configured (server.xml, web.xml, > >>> logging, CPU/RAM). We have not made any other > >>> configuration change between 8.5.41 and 8.5.45. And yes, > >>> I agree with you that it's fairly easy to reproduce. > > > >> I think the question is whether or not your application is > >> required > to > >> be deployed. Can you reproduce this issue with just the stock > >> applications bundled with Tomcat? > > > > > > My apologies, but our application needs to be deployed. We > > have not > (or > > didn't try in the past) to simply deploy tomcat with stock > application (in > > other words, simply starting the tomcat OOB) on our prod > > servers. This is the first time it has hit us with such > > disparity. I’ll try to investigate and get a stock > > application data. But we may not be able > to do > > that quite easily as it’s in our production. > > > > What I can see is that 3 Windows updates may have been > > responsible > for > > this, but we aren’t sure
Re: tomcat thread incurring CPU load
OK, it looks like I can reproduce this. Steps to reproduce: - Windows 2016 Server fully patched - Java 1.8.0u144 - Install Tomcat 8.5.45 from windows installer - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 - Modify server.xml to use Http11AprProtocol on port 8080 - Make a single request I then see 1 core running at 100% until the connection times out after 20s. Make another request and a core goes back up to 100% for 20s (the default keep-alive time out). Next steps are to try and track down the root cause. Mark > Mark and M, > > On 11/13/19 19:31, Mark Thomas wrote: >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" >> wrote: >>> I see this update on Windows which may have been responsible >>> (suspicion only, haven’t rolled it back yet) >>> >>> >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > ocode-updates >>> >>> >>> > Was 8.5.45 built on Windows 10 in presence of this update ? > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully >> patched at the time of the build Windows 7 64-bit VM. > Also it doesn't matter because binaries don't include CPU microcode. > > It's more likely that the target system has microcode updates such as > these that may negatively impact performance. > > -chris > >>> >>> Thanks, >>> >>> On Wed, 13 Nov 2019 at 17:55, M. Manna >>> wrote: >>> Hi Chris, On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < ch...@christopherschultz.net> wrote: >> On 11/13/19 11:20, M. Manna wrote: >>> HI Mark, >>> >>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>> wrote: >>> On 12/11/2019 19:11, M. Manna wrote: > HI Mark, > > following my previous reply, we have now confirmed > that it's indeed 8.5.45 > with APR 1.2.23 that's causing such high JVM CPU > usage. We used took out 2 out of 50 servers from the > load balancer config, reverted tomcat, and > redeployed. With near to identical user traffic, the > two servers are responding normally without/without > traffic with 8.5.41. The JVM dump looks a lot better > with 8.5.41. > > We do think that the recent changes in APR and some > other tomcat jar may have caused compatibility issue > on Windows server 2016 (64-bit) platform. But > unfortunately, we cannot pinpoint exactly what change > may have caused this (i.e. actual OS vs Security > Updates). With this in mind, we are also being wary > to move to 8.5.47 as we don't know if the same issue > will occur > again. Since 8.5.41 has been packaged with previously > accepted application > installer, we are more comfortable rolling back. Just to confirm, you see this high CPU usage with a clean install (no additional web applications deployed, no configuration changes) on Windows 2016 DataCenter (64-bit)? If this is the case, it should be fairly easy to reproduce. Mark We do not deploy multiple applications. In fact, Under tomcat >>> webapps/ROOT we only have one application (ours). Each >>> tomcat instance is hosted on a VM (total 50) and all of >>> them are identically configured (server.xml, web.xml, >>> logging, CPU/RAM). We have not made any other >>> configuration change between 8.5.41 and 8.5.45. And yes, >>> I agree with you that it's fairly easy to reproduce. > >> I think the question is whether or not your application is >> required to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? > > > My apologies, but our application needs to be deployed. We > have not (or > didn't try in the past) to simply deploy tomcat with stock application (in > other words, simply starting the tomcat OOB) on our prod > servers. This is the first time it has hit us with such > disparity. I’ll try to investigate and get a stock > application data. But we may not be able to do > that quite easily as it’s in our production. > > What I can see is that 3 Windows updates may have been > responsible for > this, but we aren’t sure about that. I’ll let you know if we > can get anything with the stock application instance. > > Thanks, > > - -chris > > > >>> - > >>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: > users-h...@tomcat.apache.org > > > > >> - > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > >
Re: tomcat thread incurring CPU load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark and M, On 11/13/19 19:31, Mark Thomas wrote: > On November 13, 2019 11:42:34 PM UTC, "M. Manna" > wrote: >> I see this update on Windows which may have been responsible >> (suspicion only, haven’t rolled it back yet) >> >> >> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr ocode-updates >> >> >> Was 8.5.45 built on Windows 10 in presence of this update ? > > No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully > patched at the time of the build Windows 7 64-bit VM. Also it doesn't matter because binaries don't include CPU microcode. It's more likely that the target system has microcode updates such as these that may negatively impact performance. - -chris >> >> Thanks, >> >> On Wed, 13 Nov 2019 at 17:55, M. Manna >> wrote: >> >>> Hi Chris, >>> >>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >>> ch...@christopherschultz.net> wrote: >>> > On 11/13/19 11:20, M. Manna wrote: >> HI Mark, >> >> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >> wrote: >> >>> On 12/11/2019 19:11, M. Manna wrote: HI Mark, following my previous reply, we have now confirmed that it's indeed >>> 8.5.45 with APR 1.2.23 that's causing such high JVM CPU usage. We used took out 2 out of 50 servers from the load balancer config, reverted tomcat, and redeployed. With near to identical user traffic, the two servers are responding normally without/without traffic with 8.5.41. The JVM dump looks a lot better with 8.5.41. We do think that the recent changes in APR and some other tomcat jar may have caused compatibility issue on Windows server 2016 (64-bit) platform. But unfortunately, we cannot pinpoint exactly what change may have caused this (i.e. actual OS vs Security Updates). With this in mind, we are also being wary to move to 8.5.47 as we don't know if the same issue will >>> occur again. Since 8.5.41 has been packaged with previously accepted >>> application installer, we are more comfortable rolling back. >>> >>> Just to confirm, you see this high CPU usage with a >>> clean install (no additional web applications deployed, >>> no configuration changes) on Windows 2016 DataCenter >>> (64-bit)? >>> >>> If this is the case, it should be fairly easy to >>> reproduce. >>> >>> Mark >>> >>> We do not deploy multiple applications. In fact, Under >>> tomcat >> webapps/ROOT we only have one application (ours). Each >> tomcat instance is hosted on a VM (total 50) and all of >> them are identically configured (server.xml, web.xml, >> logging, CPU/RAM). We have not made any other >> configuration change between 8.5.41 and 8.5.45. And yes, >> I agree with you that it's fairly easy to reproduce. > > I think the question is whether or not your application is > required >>> to > be deployed. Can you reproduce this issue with just the stock > applications bundled with Tomcat? > My apologies, but our application needs to be deployed. We have not >>> (or didn't try in the past) to simply deploy tomcat with stock >>> application (in other words, simply starting the tomcat OOB) on our prod servers. This is the first time it has hit us with such disparity. I’ll try to investigate and get a stock application data. But we may not be able >>> to do that quite easily as it’s in our production. What I can see is that 3 Windows updates may have been responsible >>> for this, but we aren’t sure about that. I’ll let you know if we can get anything with the stock application instance. Thanks, - -chris >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3M004ACgkQHPApP6U8 pFjz9Q//diCZmm58oaIbBDwJhwJRJakuuBR+e995QT6/LGyif76ltMj5Wk3js8iB 0jm7T9TM+mUQyjn9WZ3QYuCUuwl5q9z7pSOCxRru0LRXqMWtkZWpU3u6VBisnGQg j+Admu5DPgg7nEXvjR/a09RZ0lmC6ubHi0LxnAMCeEFp0YAe7Hg2GVRJ5Qzc1dUk ndBeAmc4WiNbgWlFo/3GhlrWK12TOHOk+Rhi7s6BojHArVE2ptMu72EYxFv+xLfP 4wn72pGEZuxwOtzTATL+QIVNCIar4GNDtxHtcWL28oXAlwVws8OSe4gfEvtJhUIu Zwa/RnS1gwP/68hdlII5jVtZclyQhqnxu3Vjxd4GcEERNtGCi4K1MG1mu6TAXh22
Re: tomcat thread incurring CPU load
On November 13, 2019 11:42:34 PM UTC, "M. Manna" wrote: >I see this update on Windows which may have been responsible (suspicion >only, haven’t rolled it back yet) > > >https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates > >Was 8.5.45 built on Windows 10 in presence of this update ? No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully patched at the time of the build Windows 7 64-bit VM. Mark > >Thanks, > >On Wed, 13 Nov 2019 at 17:55, M. Manna wrote: > >> Hi Chris, >> >> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >> ch...@christopherschultz.net> wrote: >> >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> On 11/13/19 11:20, M. Manna wrote: >>> > HI Mark, >>> > >>> > On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>> > wrote: >>> > >>> >> On 12/11/2019 19:11, M. Manna wrote: >>> >>> HI Mark, >>> >>> >>> >>> following my previous reply, we have now confirmed that it's >>> >>> indeed >>> >> 8.5.45 >>> >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >>> >>> took out 2 out of 50 servers from the load balancer config, >>> >>> reverted tomcat, and redeployed. With near to identical user >>> >>> traffic, the two servers are responding normally >>> >>> without/without traffic with 8.5.41. The JVM dump looks a lot >>> >>> better with 8.5.41. >>> >>> >>> >>> We do think that the recent changes in APR and some other >>> >>> tomcat jar may have caused compatibility issue on Windows >>> >>> server 2016 (64-bit) platform. But unfortunately, we cannot >>> >>> pinpoint exactly what change may have caused this (i.e. actual >>> >>> OS vs Security Updates). With this in mind, we are also being >>> >>> wary to move to 8.5.47 as we don't know if the same issue will >>> >> occur >>> >>> again. Since 8.5.41 has been packaged with previously accepted >>> >> application >>> >>> installer, we are more comfortable rolling back. >>> >> >>> >> Just to confirm, you see this high CPU usage with a clean install >>> >> (no additional web applications deployed, no configuration >>> >> changes) on Windows 2016 DataCenter (64-bit)? >>> >> >>> >> If this is the case, it should be fairly easy to reproduce. >>> >> >>> >> Mark >>> >> >>> >> We do not deploy multiple applications. In fact, Under tomcat >>> > webapps/ROOT we only have one application (ours). Each tomcat >>> > instance is hosted on a VM (total 50) and all of them are >>> > identically configured (server.xml, web.xml, logging, CPU/RAM). We >>> > have not made any other configuration change between 8.5.41 and >>> > 8.5.45. And yes, I agree with you that it's fairly easy to >>> > reproduce. >>> >>> I think the question is whether or not your application is required >to >>> be deployed. Can you reproduce this issue with just the stock >>> applications bundled with Tomcat? >>> >> >> My apologies, but our application needs to be deployed. We have not >(or >> didn't try in the past) to simply deploy tomcat with stock >application (in >> other words, simply starting the tomcat OOB) on our prod servers. >> This is the first time it has hit us with such disparity. I’ll try to >> investigate and get a stock application data. But we may not be able >to do >> that quite easily as it’s in our production. >> >> What I can see is that 3 Windows updates may have been responsible >for >> this, but we aren’t sure about that. I’ll let you know if we can get >> anything with the stock application instance. >> >> Thanks, >> >> - -chris >> >>> -BEGIN PGP SIGNATURE- >>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ >>> >>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 >>> pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 >>> 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB >>> Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO >>> WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h >>> 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv >>> kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY >>> mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP >>> skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp >>> zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 >>> e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp >>> XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= >>> =VqkL >>> -END PGP SIGNATURE- >>> >>> >- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
I see this update on Windows which may have been responsible (suspicion only, haven’t rolled it back yet) https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates Was 8.5.45 built on Windows 10 in presence of this update ? Thanks, On Wed, 13 Nov 2019 at 17:55, M. Manna wrote: > Hi Chris, > > On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA256 >> >> On 11/13/19 11:20, M. Manna wrote: >> > HI Mark, >> > >> > On Wed, 13 Nov 2019 at 15:38, Mark Thomas >> > wrote: >> > >> >> On 12/11/2019 19:11, M. Manna wrote: >> >>> HI Mark, >> >>> >> >>> following my previous reply, we have now confirmed that it's >> >>> indeed >> >> 8.5.45 >> >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >> >>> took out 2 out of 50 servers from the load balancer config, >> >>> reverted tomcat, and redeployed. With near to identical user >> >>> traffic, the two servers are responding normally >> >>> without/without traffic with 8.5.41. The JVM dump looks a lot >> >>> better with 8.5.41. >> >>> >> >>> We do think that the recent changes in APR and some other >> >>> tomcat jar may have caused compatibility issue on Windows >> >>> server 2016 (64-bit) platform. But unfortunately, we cannot >> >>> pinpoint exactly what change may have caused this (i.e. actual >> >>> OS vs Security Updates). With this in mind, we are also being >> >>> wary to move to 8.5.47 as we don't know if the same issue will >> >> occur >> >>> again. Since 8.5.41 has been packaged with previously accepted >> >> application >> >>> installer, we are more comfortable rolling back. >> >> >> >> Just to confirm, you see this high CPU usage with a clean install >> >> (no additional web applications deployed, no configuration >> >> changes) on Windows 2016 DataCenter (64-bit)? >> >> >> >> If this is the case, it should be fairly easy to reproduce. >> >> >> >> Mark >> >> >> >> We do not deploy multiple applications. In fact, Under tomcat >> > webapps/ROOT we only have one application (ours). Each tomcat >> > instance is hosted on a VM (total 50) and all of them are >> > identically configured (server.xml, web.xml, logging, CPU/RAM). We >> > have not made any other configuration change between 8.5.41 and >> > 8.5.45. And yes, I agree with you that it's fairly easy to >> > reproduce. >> >> I think the question is whether or not your application is required to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? >> > > My apologies, but our application needs to be deployed. We have not (or > didn't try in the past) to simply deploy tomcat with stock application (in > other words, simply starting the tomcat OOB) on our prod servers. > This is the first time it has hit us with such disparity. I’ll try to > investigate and get a stock application data. But we may not be able to do > that quite easily as it’s in our production. > > What I can see is that 3 Windows updates may have been responsible for > this, but we aren’t sure about that. I’ll let you know if we can get > anything with the stock application instance. > > Thanks, > > - -chris > >> -BEGIN PGP SIGNATURE- >> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ >> >> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 >> pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 >> 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB >> Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO >> WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h >> 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv >> kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY >> mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP >> skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp >> zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 >> e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp >> XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= >> =VqkL >> -END PGP SIGNATURE- >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: tomcat thread incurring CPU load
Hi Chris, On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 11/13/19 11:20, M. Manna wrote: > > HI Mark, > > > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > > wrote: > > > >> On 12/11/2019 19:11, M. Manna wrote: > >>> HI Mark, > >>> > >>> following my previous reply, we have now confirmed that it's > >>> indeed > >> 8.5.45 > >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used > >>> took out 2 out of 50 servers from the load balancer config, > >>> reverted tomcat, and redeployed. With near to identical user > >>> traffic, the two servers are responding normally > >>> without/without traffic with 8.5.41. The JVM dump looks a lot > >>> better with 8.5.41. > >>> > >>> We do think that the recent changes in APR and some other > >>> tomcat jar may have caused compatibility issue on Windows > >>> server 2016 (64-bit) platform. But unfortunately, we cannot > >>> pinpoint exactly what change may have caused this (i.e. actual > >>> OS vs Security Updates). With this in mind, we are also being > >>> wary to move to 8.5.47 as we don't know if the same issue will > >> occur > >>> again. Since 8.5.41 has been packaged with previously accepted > >> application > >>> installer, we are more comfortable rolling back. > >> > >> Just to confirm, you see this high CPU usage with a clean install > >> (no additional web applications deployed, no configuration > >> changes) on Windows 2016 DataCenter (64-bit)? > >> > >> If this is the case, it should be fairly easy to reproduce. > >> > >> Mark > >> > >> We do not deploy multiple applications. In fact, Under tomcat > > webapps/ROOT we only have one application (ours). Each tomcat > > instance is hosted on a VM (total 50) and all of them are > > identically configured (server.xml, web.xml, logging, CPU/RAM). We > > have not made any other configuration change between 8.5.41 and > > 8.5.45. And yes, I agree with you that it's fairly easy to > > reproduce. > > I think the question is whether or not your application is required to > be deployed. Can you reproduce this issue with just the stock > applications bundled with Tomcat? > My apologies, but our application needs to be deployed. We have not (or didn't try in the past) to simply deploy tomcat with stock application (in other words, simply starting the tomcat OOB) on our prod servers. This is the first time it has hit us with such disparity. I’ll try to investigate and get a stock application data. But we may not be able to do that quite easily as it’s in our production. What I can see is that 3 Windows updates may have been responsible for this, but we aren’t sure about that. I’ll let you know if we can get anything with the stock application instance. Thanks, - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 > pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 > 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB > Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO > WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h > 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv > kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY > mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP > skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp > zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 > e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp > XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= > =VqkL > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: tomcat thread incurring CPU load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/13/19 11:20, M. Manna wrote: > HI Mark, > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > wrote: > >> On 12/11/2019 19:11, M. Manna wrote: >>> HI Mark, >>> >>> following my previous reply, we have now confirmed that it's >>> indeed >> 8.5.45 >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >>> took out 2 out of 50 servers from the load balancer config, >>> reverted tomcat, and redeployed. With near to identical user >>> traffic, the two servers are responding normally >>> without/without traffic with 8.5.41. The JVM dump looks a lot >>> better with 8.5.41. >>> >>> We do think that the recent changes in APR and some other >>> tomcat jar may have caused compatibility issue on Windows >>> server 2016 (64-bit) platform. But unfortunately, we cannot >>> pinpoint exactly what change may have caused this (i.e. actual >>> OS vs Security Updates). With this in mind, we are also being >>> wary to move to 8.5.47 as we don't know if the same issue will >> occur >>> again. Since 8.5.41 has been packaged with previously accepted >> application >>> installer, we are more comfortable rolling back. >> >> Just to confirm, you see this high CPU usage with a clean install >> (no additional web applications deployed, no configuration >> changes) on Windows 2016 DataCenter (64-bit)? >> >> If this is the case, it should be fairly easy to reproduce. >> >> Mark >> >> We do not deploy multiple applications. In fact, Under tomcat > webapps/ROOT we only have one application (ours). Each tomcat > instance is hosted on a VM (total 50) and all of them are > identically configured (server.xml, web.xml, logging, CPU/RAM). We > have not made any other configuration change between 8.5.41 and > 8.5.45. And yes, I agree with you that it's fairly easy to > reproduce. I think the question is whether or not your application is required to be deployed. Can you reproduce this issue with just the stock applications bundled with Tomcat? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= =VqkL -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
HI Mark, On Wed, 13 Nov 2019 at 15:38, Mark Thomas wrote: > On 12/11/2019 19:11, M. Manna wrote: > > HI Mark, > > > > following my previous reply, we have now confirmed that it's indeed > 8.5.45 > > with APR 1.2.23 that's causing such high JVM CPU usage. > > We used took out 2 out of 50 servers from the load balancer config, > > reverted tomcat, and redeployed. With near to identical user traffic, the > > two servers are responding normally without/without traffic with 8.5.41. > > The JVM dump looks a lot better with 8.5.41. > > > > We do think that the recent changes in APR and some other tomcat jar may > > have caused compatibility issue on Windows server 2016 (64-bit) platform. > > But unfortunately, we cannot pinpoint exactly what change may have caused > > this (i.e. actual OS vs Security Updates). With this in mind, we are also > > being wary to move to 8.5.47 as we don't know if the same issue will > occur > > again. Since 8.5.41 has been packaged with previously accepted > application > > installer, we are more comfortable rolling back. > > Just to confirm, you see this high CPU usage with a clean install (no > additional web applications deployed, no configuration changes) on > Windows 2016 DataCenter (64-bit)? > > If this is the case, it should be fairly easy to reproduce. > > Mark > > We do not deploy multiple applications. In fact, Under tomcat webapps/ROOT we only have one application (ours). Each tomcat instance is hosted on a VM (total 50) and all of them are identically configured (server.xml, web.xml, logging, CPU/RAM). We have not made any other configuration change between 8.5.41 and 8.5.45. And yes, I agree with you that it's fairly easy to reproduce. Thanks, > > > > > I would appreciate if this can be looked into. > > > > On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > > > >> Hey Mark (appreciate your response in US holiday time) > >> > >> On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > >> > >>> On November 12, 2019 12:54:53 AM UTC, "M. Manna" > >>> wrote: > Just to give an update again: > > 1) We reverted the APR to 1.2.21 - but observed no difference. > 2) We took 3 thread dumps over 1 min interval (without any user > sessions) - > All threads are tomcat's internal pool threads. > > When we checked the thread details (using fasthread.io) - we didn't > see > any > of our application stack. Since there is no user traffic, this is > coming > >>> >from tomcat internally. At this stage, we cannot really figure out > what's > the root cause. > > Any help is appreciated. > >>> > >>> Migrated from what (full version info please)? > >>> > >> from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > >> > >>> > >>> Operating system exact version? > >>> > >> Microsoft Windows Server 2016 DataCentre (64-bit) > >> > >>> > >>> JRE vendor and exact version? > >>> > >> C:\jdk1.8.0\bin>java.exe -version > >> java version "1.8.0_162" > >> Java(TM) SE Runtime Environment (build 1.8.0_162-b12) > >> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > >> > >> > >>> Do you see the same behavior with the latest 8.5.x and latest Tomcat > >>> Native? > >>> > >> We are using APR 1.2.23 which I can also see in latest tomcat. Due to > >> production due diligence we cannot roll to a different version that > easily. > >> Normally, we lag behind by 2 monthly releases of tomcat. We also > reverted > >> the APR to 1.2.21 (but no difference). > >> > >>> > >>> What triggers this behaviour? > >>> > >> That is quite strange. Due to US holidays, we had a low traffic on our > >> servers, and nothing has crept in to suggest that it's > application-driven. > >> We took one tomcat instance out of 50 instances and removed all user > >> sessions (i.e. no application activities or threads). Upon restart of > >> tomcat, the CPU spike lingered past the initial servlet startup period. > We > >> monitored that over 1-2 hours but there was no difference. > >> > >>> > >>> How often do you see this behaviour? > >>> > >> We took 2 sets of data > >> 1) 3 Jstack dump based on 10 seconds interval. > >> 2) 3 jstack dump based on 1 min interval. > >> > >> Both the above reveals that all background threads (http, pool etc.) > were > >> from tomcat. We didn't have any application threads lingered in those 3 > >> samples. So yes we see this almost all the time if we take samples. > >> However, when we compared with pre-production instances (with Windows > >> server R2 x64 bit), we don't see such abnormal spike. In fact, the > >> application instance doesn't incur such a big CPU spike. Whilst > composing > >> this email, I am now thinking if the APR is indeed incompatible with > >> WIndows Server R2 (or the presence of any Windows Updates) which blocks > the > >> native poll() call longer than usual. > >> > >> An example is that on Windows Server 2012 - APR poll() call takes about > >> 30% CPU time - but with Windows Server
Re: tomcat thread incurring CPU load
On 12/11/2019 19:11, M. Manna wrote: > HI Mark, > > following my previous reply, we have now confirmed that it's indeed 8.5.45 > with APR 1.2.23 that's causing such high JVM CPU usage. > We used took out 2 out of 50 servers from the load balancer config, > reverted tomcat, and redeployed. With near to identical user traffic, the > two servers are responding normally without/without traffic with 8.5.41. > The JVM dump looks a lot better with 8.5.41. > > We do think that the recent changes in APR and some other tomcat jar may > have caused compatibility issue on Windows server 2016 (64-bit) platform. > But unfortunately, we cannot pinpoint exactly what change may have caused > this (i.e. actual OS vs Security Updates). With this in mind, we are also > being wary to move to 8.5.47 as we don't know if the same issue will occur > again. Since 8.5.41 has been packaged with previously accepted application > installer, we are more comfortable rolling back. Just to confirm, you see this high CPU usage with a clean install (no additional web applications deployed, no configuration changes) on Windows 2016 DataCenter (64-bit)? If this is the case, it should be fairly easy to reproduce. Mark > > I would appreciate if this can be looked into. > > On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > >> Hey Mark (appreciate your response in US holiday time) >> >> On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: >> >>> On November 12, 2019 12:54:53 AM UTC, "M. Manna" >>> wrote: Just to give an update again: 1) We reverted the APR to 1.2.21 - but observed no difference. 2) We took 3 thread dumps over 1 min interval (without any user sessions) - All threads are tomcat's internal pool threads. When we checked the thread details (using fasthread.io) - we didn't see any of our application stack. Since there is no user traffic, this is coming >>> >from tomcat internally. At this stage, we cannot really figure out what's the root cause. Any help is appreciated. >>> >>> Migrated from what (full version info please)? >>> >> from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) >> >>> >>> Operating system exact version? >>> >> Microsoft Windows Server 2016 DataCentre (64-bit) >> >>> >>> JRE vendor and exact version? >>> >> C:\jdk1.8.0\bin>java.exe -version >> java version "1.8.0_162" >> Java(TM) SE Runtime Environment (build 1.8.0_162-b12) >> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) >> >> >>> Do you see the same behavior with the latest 8.5.x and latest Tomcat >>> Native? >>> >> We are using APR 1.2.23 which I can also see in latest tomcat. Due to >> production due diligence we cannot roll to a different version that easily. >> Normally, we lag behind by 2 monthly releases of tomcat. We also reverted >> the APR to 1.2.21 (but no difference). >> >>> >>> What triggers this behaviour? >>> >> That is quite strange. Due to US holidays, we had a low traffic on our >> servers, and nothing has crept in to suggest that it's application-driven. >> We took one tomcat instance out of 50 instances and removed all user >> sessions (i.e. no application activities or threads). Upon restart of >> tomcat, the CPU spike lingered past the initial servlet startup period. We >> monitored that over 1-2 hours but there was no difference. >> >>> >>> How often do you see this behaviour? >>> >> We took 2 sets of data >> 1) 3 Jstack dump based on 10 seconds interval. >> 2) 3 jstack dump based on 1 min interval. >> >> Both the above reveals that all background threads (http, pool etc.) were >> from tomcat. We didn't have any application threads lingered in those 3 >> samples. So yes we see this almost all the time if we take samples. >> However, when we compared with pre-production instances (with Windows >> server R2 x64 bit), we don't see such abnormal spike. In fact, the >> application instance doesn't incur such a big CPU spike. Whilst composing >> this email, I am now thinking if the APR is indeed incompatible with >> WIndows Server R2 (or the presence of any Windows Updates) which blocks the >> native poll() call longer than usual. >> >> An example is that on Windows Server 2012 - APR poll() call takes about >> 30% CPU time - but with Windows Server 2016 it's almost always 95%. >> >> >>> >>> And anything else you think might be relevant. >>> >> >> We are using end-2-end encryption using APR (with Certificate and >> SSLConfig resource setup in server.xml). But it's survived past 3 tomcat >> upgrades without any issue. >> Except OS we don't have any obvious culprit identified at the moment. >> >> Thanks, >> >>> >>> Mark >>> Thanks, On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > Hello All, > > Any thoughts regarding this? Slightly clueless at this point, so any > direction will be appreciated. > > We are seeing the poll taking all the CPU time. We are using >
Re: tomcat thread incurring CPU load
HI Mark, following my previous reply, we have now confirmed that it's indeed 8.5.45 with APR 1.2.23 that's causing such high JVM CPU usage. We used took out 2 out of 50 servers from the load balancer config, reverted tomcat, and redeployed. With near to identical user traffic, the two servers are responding normally without/without traffic with 8.5.41. The JVM dump looks a lot better with 8.5.41. We do think that the recent changes in APR and some other tomcat jar may have caused compatibility issue on Windows server 2016 (64-bit) platform. But unfortunately, we cannot pinpoint exactly what change may have caused this (i.e. actual OS vs Security Updates). With this in mind, we are also being wary to move to 8.5.47 as we don't know if the same issue will occur again. Since 8.5.41 has been packaged with previously accepted application installer, we are more comfortable rolling back. I would appreciate if this can be looked into. On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > Hey Mark (appreciate your response in US holiday time) > > On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > >> On November 12, 2019 12:54:53 AM UTC, "M. Manna" >> wrote: >> >Just to give an update again: >> > >> >1) We reverted the APR to 1.2.21 - but observed no difference. >> >2) We took 3 thread dumps over 1 min interval (without any user >> >sessions) - >> >All threads are tomcat's internal pool threads. >> > >> >When we checked the thread details (using fasthread.io) - we didn't see >> >any >> >of our application stack. Since there is no user traffic, this is >> >coming >> >from tomcat internally. At this stage, we cannot really figure out >> >what's >> >the root cause. >> > >> >Any help is appreciated. >> >> Migrated from what (full version info please)? >> > from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > >> >> Operating system exact version? >> > Microsoft Windows Server 2016 DataCentre (64-bit) > >> >> JRE vendor and exact version? >> > C:\jdk1.8.0\bin>java.exe -version > java version "1.8.0_162" > Java(TM) SE Runtime Environment (build 1.8.0_162-b12) > Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > > >> Do you see the same behavior with the latest 8.5.x and latest Tomcat >> Native? >> > We are using APR 1.2.23 which I can also see in latest tomcat. Due to > production due diligence we cannot roll to a different version that easily. > Normally, we lag behind by 2 monthly releases of tomcat. We also reverted > the APR to 1.2.21 (but no difference). > >> >> What triggers this behaviour? >> > That is quite strange. Due to US holidays, we had a low traffic on our > servers, and nothing has crept in to suggest that it's application-driven. > We took one tomcat instance out of 50 instances and removed all user > sessions (i.e. no application activities or threads). Upon restart of > tomcat, the CPU spike lingered past the initial servlet startup period. We > monitored that over 1-2 hours but there was no difference. > >> >> How often do you see this behaviour? >> > We took 2 sets of data > 1) 3 Jstack dump based on 10 seconds interval. > 2) 3 jstack dump based on 1 min interval. > > Both the above reveals that all background threads (http, pool etc.) were > from tomcat. We didn't have any application threads lingered in those 3 > samples. So yes we see this almost all the time if we take samples. > However, when we compared with pre-production instances (with Windows > server R2 x64 bit), we don't see such abnormal spike. In fact, the > application instance doesn't incur such a big CPU spike. Whilst composing > this email, I am now thinking if the APR is indeed incompatible with > WIndows Server R2 (or the presence of any Windows Updates) which blocks the > native poll() call longer than usual. > > An example is that on Windows Server 2012 - APR poll() call takes about > 30% CPU time - but with Windows Server 2016 it's almost always 95%. > > >> >> And anything else you think might be relevant. >> > > We are using end-2-end encryption using APR (with Certificate and > SSLConfig resource setup in server.xml). But it's survived past 3 tomcat > upgrades without any issue. > Except OS we don't have any obvious culprit identified at the moment. > > Thanks, > >> >> Mark >> >> > >> >Thanks, >> > >> >On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: >> > >> >> Hello All, >> >> >> >> Any thoughts regarding this? Slightly clueless at this point, so any >> >> direction will be appreciated. >> >> >> >> We are seeing the poll taking all the CPU time. We are using >> >> OperatingSystemMXBean.getProcessCpuLoad() and >> >> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then >> >x100 to >> >> get the pct). >> >> >> >> Thanks, >> >> >> >> >> >> On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: >> >> >> >>> Hello, >> >>> >> >>> after migrating to 8.5.45, we are seeing a lot of cpu load by >> >following >> >>> JVM thread dump: >> >>> >> >>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 :
Re: tomcat thread incurring CPU load
Hey Mark (appreciate your response in US holiday time) On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > On November 12, 2019 12:54:53 AM UTC, "M. Manna" > wrote: > >Just to give an update again: > > > >1) We reverted the APR to 1.2.21 - but observed no difference. > >2) We took 3 thread dumps over 1 min interval (without any user > >sessions) - > >All threads are tomcat's internal pool threads. > > > >When we checked the thread details (using fasthread.io) - we didn't see > >any > >of our application stack. Since there is no user traffic, this is > >coming > >from tomcat internally. At this stage, we cannot really figure out > >what's > >the root cause. > > > >Any help is appreciated. > > Migrated from what (full version info please)? > from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > > Operating system exact version? > Microsoft Windows Server 2016 DataCentre (64-bit) > > JRE vendor and exact version? > C:\jdk1.8.0\bin>java.exe -version java version "1.8.0_162" Java(TM) SE Runtime Environment (build 1.8.0_162-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > Do you see the same behavior with the latest 8.5.x and latest Tomcat > Native? > We are using APR 1.2.23 which I can also see in latest tomcat. Due to production due diligence we cannot roll to a different version that easily. Normally, we lag behind by 2 monthly releases of tomcat. We also reverted the APR to 1.2.21 (but no difference). > > What triggers this behaviour? > That is quite strange. Due to US holidays, we had a low traffic on our servers, and nothing has crept in to suggest that it's application-driven. We took one tomcat instance out of 50 instances and removed all user sessions (i.e. no application activities or threads). Upon restart of tomcat, the CPU spike lingered past the initial servlet startup period. We monitored that over 1-2 hours but there was no difference. > > How often do you see this behaviour? > We took 2 sets of data 1) 3 Jstack dump based on 10 seconds interval. 2) 3 jstack dump based on 1 min interval. Both the above reveals that all background threads (http, pool etc.) were from tomcat. We didn't have any application threads lingered in those 3 samples. So yes we see this almost all the time if we take samples. However, when we compared with pre-production instances (with Windows server R2 x64 bit), we don't see such abnormal spike. In fact, the application instance doesn't incur such a big CPU spike. Whilst composing this email, I am now thinking if the APR is indeed incompatible with WIndows Server R2 (or the presence of any Windows Updates) which blocks the native poll() call longer than usual. An example is that on Windows Server 2012 - APR poll() call takes about 30% CPU time - but with Windows Server 2016 it's almost always 95%. > > And anything else you think might be relevant. > We are using end-2-end encryption using APR (with Certificate and SSLConfig resource setup in server.xml). But it's survived past 3 tomcat upgrades without any issue. Except OS we don't have any obvious culprit identified at the moment. Thanks, > > Mark > > > > >Thanks, > > > >On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > > > >> Hello All, > >> > >> Any thoughts regarding this? Slightly clueless at this point, so any > >> direction will be appreciated. > >> > >> We are seeing the poll taking all the CPU time. We are using > >> OperatingSystemMXBean.getProcessCpuLoad() and > >> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then > >x100 to > >> get the pct). > >> > >> Thanks, > >> > >> > >> On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: > >> > >>> Hello, > >>> > >>> after migrating to 8.5.45, we are seeing a lot of cpu load by > >following > >>> JVM thread dump: > >>> > >>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : > >>> cpu=172902703125000 : cpuLoad= 74.181015 > >>> > >>> BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 > >>> LockOwnerName:null > >>> > >>> WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at > >>> org.apache.tomcat.jni.Poll.poll(Poll.java:-2) > >>> > >>> at > >>> > >org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) > >>> > >>> at java.lang.Thread.run(Thread.java:748) > >>> > >>> > >>> These are coming after 2-3 successful jvm dump. Is this something > >>> familiar to anybody? > >>> > >>> Thanks, > >>> > >> > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: tomcat thread incurring CPU load
On November 12, 2019 12:54:53 AM UTC, "M. Manna" wrote: >Just to give an update again: > >1) We reverted the APR to 1.2.21 - but observed no difference. >2) We took 3 thread dumps over 1 min interval (without any user >sessions) - >All threads are tomcat's internal pool threads. > >When we checked the thread details (using fasthread.io) - we didn't see >any >of our application stack. Since there is no user traffic, this is >coming >from tomcat internally. At this stage, we cannot really figure out >what's >the root cause. > >Any help is appreciated. Migrated from what (full version info please)? Operating system exact version? JRE vendor and exact version? Do you see the same behavior with the latest 8.5.x and latest Tomcat Native? What triggers this behaviour? How often do you see this behaviour? And anything else you think might be relevant. Mark > >Thanks, > >On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > >> Hello All, >> >> Any thoughts regarding this? Slightly clueless at this point, so any >> direction will be appreciated. >> >> We are seeing the poll taking all the CPU time. We are using >> OperatingSystemMXBean.getProcessCpuLoad() and >> OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then >x100 to >> get the pct). >> >> Thanks, >> >> >> On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: >> >>> Hello, >>> >>> after migrating to 8.5.45, we are seeing a lot of cpu load by >following >>> JVM thread dump: >>> >>> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : >>> cpu=172902703125000 : cpuLoad= 74.181015 >>> >>> BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 >>> LockOwnerName:null >>> >>> WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at >>> org.apache.tomcat.jni.Poll.poll(Poll.java:-2) >>> >>> at >>> >org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) >>> >>> at java.lang.Thread.run(Thread.java:748) >>> >>> >>> These are coming after 2-3 successful jvm dump. Is this something >>> familiar to anybody? >>> >>> Thanks, >>> >> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
Just to give an update again: 1) We reverted the APR to 1.2.21 - but observed no difference. 2) We took 3 thread dumps over 1 min interval (without any user sessions) - All threads are tomcat's internal pool threads. When we checked the thread details (using fasthread.io) - we didn't see any of our application stack. Since there is no user traffic, this is coming from tomcat internally. At this stage, we cannot really figure out what's the root cause. Any help is appreciated. Thanks, On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > Hello All, > > Any thoughts regarding this? Slightly clueless at this point, so any > direction will be appreciated. > > We are seeing the poll taking all the CPU time. We are using > OperatingSystemMXBean.getProcessCpuLoad() and > OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then x100 to > get the pct). > > Thanks, > > > On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: > >> Hello, >> >> after migrating to 8.5.45, we are seeing a lot of cpu load by following >> JVM thread dump: >> >> "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : >> cpu=172902703125000 : cpuLoad= 74.181015 >> >> BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 >> LockOwnerName:null >> >> WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at >> org.apache.tomcat.jni.Poll.poll(Poll.java:-2) >> >> at >> org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) >> >> at java.lang.Thread.run(Thread.java:748) >> >> >> These are coming after 2-3 successful jvm dump. Is this something >> familiar to anybody? >> >> Thanks, >> >
Re: tomcat thread incurring CPU load
Hello All, Any thoughts regarding this? Slightly clueless at this point, so any direction will be appreciated. We are seeing the poll taking all the CPU time. We are using OperatingSystemMXBean.getProcessCpuLoad() and OperatingSystemMXBean.getSystemCpuLoad() to get our metrics (then x100 to get the pct). Thanks, On Mon, 11 Nov 2019 at 17:46, M. Manna wrote: > Hello, > > after migrating to 8.5.45, we are seeing a lot of cpu load by following > JVM thread dump: > > "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : > cpu=172902703125000 : cpuLoad= 74.181015 > > BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 > LockOwnerName:null > > WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at > org.apache.tomcat.jni.Poll.poll(Poll.java:-2) > > at > org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) > > at java.lang.Thread.run(Thread.java:748) > > > These are coming after 2-3 successful jvm dump. Is this something familiar > to anybody? > > Thanks, >
tomcat thread incurring CPU load
Hello, after migrating to 8.5.45, we are seeing a lot of cpu load by following JVM thread dump: "https-openssl-apr-0.0.0.0-8443-Poller" : 102 : RUNNABLE : cpu=172902703125000 : cpuLoad= 74.181015 BlockedCount:8464 BlockedTime:0 LockName:null LockOwnerID:-1 LockOwnerName:null WaitedCount:5397 WaitedTime:0 InNative:false IsSuspended:false at org.apache.tomcat.jni.Poll.poll(Poll.java:-2) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1547) at java.lang.Thread.run(Thread.java:748) These are coming after 2-3 successful jvm dump. Is this something familiar to anybody? Thanks,
