Re: tomcat thread incurring CPU load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark and M, On 11/13/19 19:31, Mark Thomas wrote: > On November 13, 2019 11:42:34 PM UTC, "M. Manna" > wrote: >> I see this update on Windows which may have been responsible >> (suspicion only, haven’t rolled it back yet) >> >> >> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr ocode-updates >> >> >> Was 8.5.45 built on Windows 10 in presence of this update ? > > No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully > patched at the time of the build Windows 7 64-bit VM. Also it doesn't matter because binaries don't include CPU microcode. It's more likely that the target system has microcode updates such as these that may negatively impact performance. - -chris >> >> Thanks, >> >> On Wed, 13 Nov 2019 at 17:55, M. Manna >> wrote: >> >>> Hi Chris, >>> >>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >>> ch...@christopherschultz.net> wrote: >>> > On 11/13/19 11:20, M. Manna wrote: >> HI Mark, >> >> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >> wrote: >> >>> On 12/11/2019 19:11, M. Manna wrote: HI Mark, following my previous reply, we have now confirmed that it's indeed >>> 8.5.45 with APR 1.2.23 that's causing such high JVM CPU usage. We used took out 2 out of 50 servers from the load balancer config, reverted tomcat, and redeployed. With near to identical user traffic, the two servers are responding normally without/without traffic with 8.5.41. The JVM dump looks a lot better with 8.5.41. We do think that the recent changes in APR and some other tomcat jar may have caused compatibility issue on Windows server 2016 (64-bit) platform. But unfortunately, we cannot pinpoint exactly what change may have caused this (i.e. actual OS vs Security Updates). With this in mind, we are also being wary to move to 8.5.47 as we don't know if the same issue will >>> occur again. Since 8.5.41 has been packaged with previously accepted >>> application installer, we are more comfortable rolling back. >>> >>> Just to confirm, you see this high CPU usage with a >>> clean install (no additional web applications deployed, >>> no configuration changes) on Windows 2016 DataCenter >>> (64-bit)? >>> >>> If this is the case, it should be fairly easy to >>> reproduce. >>> >>> Mark >>> >>> We do not deploy multiple applications. In fact, Under >>> tomcat >> webapps/ROOT we only have one application (ours). Each >> tomcat instance is hosted on a VM (total 50) and all of >> them are identically configured (server.xml, web.xml, >> logging, CPU/RAM). We have not made any other >> configuration change between 8.5.41 and 8.5.45. And yes, >> I agree with you that it's fairly easy to reproduce. > > I think the question is whether or not your application is > required >>> to > be deployed. Can you reproduce this issue with just the stock > applications bundled with Tomcat? > My apologies, but our application needs to be deployed. We have not >>> (or didn't try in the past) to simply deploy tomcat with stock >>> application (in other words, simply starting the tomcat OOB) on our prod servers. This is the first time it has hit us with such disparity. I’ll try to investigate and get a stock application data. But we may not be able >>> to do that quite easily as it’s in our production. What I can see is that 3 Windows updates may have been responsible >>> for this, but we aren’t sure about that. I’ll let you know if we can get anything with the stock application instance. Thanks, - -chris >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3M004ACgkQHPApP6U8 pFjz9Q//diCZmm58oaIbBDwJhwJRJakuuBR+e995QT6/LGyif76ltMj5Wk3js8iB 0jm7T9TM+mUQyjn9WZ3QYuCUuwl5q9z7pSOCxRru0LRXqMWtkZWpU3u6VBisnGQg j+Admu5DPgg7nEXvjR/a09RZ0lmC6ubHi0LxnAMCeEFp0YAe7Hg2GVRJ5Qzc1dUk ndBeAmc4WiNbgWlFo/3GhlrWK12TOHOk+Rhi7s6BojHArVE2ptMu72EYxFv+xLfP 4wn72pGEZuxwOtzTATL+QIVNCIar4GNDtxHtcWL28oXAlwVws8OSe4gfEvtJhUIu Zwa/RnS1gwP/68hdlII5jVtZclyQhqnxu3Vjxd4GcEERNtGCi4K1MG1mu6TAXh22
Re: tomcat thread incurring CPU load
On November 13, 2019 11:42:34 PM UTC, "M. Manna" wrote: >I see this update on Windows which may have been responsible (suspicion >only, haven’t rolled it back yet) > > >https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates > >Was 8.5.45 built on Windows 10 in presence of this update ? No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully patched at the time of the build Windows 7 64-bit VM. Mark > >Thanks, > >On Wed, 13 Nov 2019 at 17:55, M. Manna wrote: > >> Hi Chris, >> >> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >> ch...@christopherschultz.net> wrote: >> >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> On 11/13/19 11:20, M. Manna wrote: >>> > HI Mark, >>> > >>> > On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>> > wrote: >>> > >>> >> On 12/11/2019 19:11, M. Manna wrote: >>> >>> HI Mark, >>> >>> >>> >>> following my previous reply, we have now confirmed that it's >>> >>> indeed >>> >> 8.5.45 >>> >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >>> >>> took out 2 out of 50 servers from the load balancer config, >>> >>> reverted tomcat, and redeployed. With near to identical user >>> >>> traffic, the two servers are responding normally >>> >>> without/without traffic with 8.5.41. The JVM dump looks a lot >>> >>> better with 8.5.41. >>> >>> >>> >>> We do think that the recent changes in APR and some other >>> >>> tomcat jar may have caused compatibility issue on Windows >>> >>> server 2016 (64-bit) platform. But unfortunately, we cannot >>> >>> pinpoint exactly what change may have caused this (i.e. actual >>> >>> OS vs Security Updates). With this in mind, we are also being >>> >>> wary to move to 8.5.47 as we don't know if the same issue will >>> >> occur >>> >>> again. Since 8.5.41 has been packaged with previously accepted >>> >> application >>> >>> installer, we are more comfortable rolling back. >>> >> >>> >> Just to confirm, you see this high CPU usage with a clean install >>> >> (no additional web applications deployed, no configuration >>> >> changes) on Windows 2016 DataCenter (64-bit)? >>> >> >>> >> If this is the case, it should be fairly easy to reproduce. >>> >> >>> >> Mark >>> >> >>> >> We do not deploy multiple applications. In fact, Under tomcat >>> > webapps/ROOT we only have one application (ours). Each tomcat >>> > instance is hosted on a VM (total 50) and all of them are >>> > identically configured (server.xml, web.xml, logging, CPU/RAM). We >>> > have not made any other configuration change between 8.5.41 and >>> > 8.5.45. And yes, I agree with you that it's fairly easy to >>> > reproduce. >>> >>> I think the question is whether or not your application is required >to >>> be deployed. Can you reproduce this issue with just the stock >>> applications bundled with Tomcat? >>> >> >> My apologies, but our application needs to be deployed. We have not >(or >> didn't try in the past) to simply deploy tomcat with stock >application (in >> other words, simply starting the tomcat OOB) on our prod servers. >> This is the first time it has hit us with such disparity. I’ll try to >> investigate and get a stock application data. But we may not be able >to do >> that quite easily as it’s in our production. >> >> What I can see is that 3 Windows updates may have been responsible >for >> this, but we aren’t sure about that. I’ll let you know if we can get >> anything with the stock application instance. >> >> Thanks, >> >> - -chris >> >>> -BEGIN PGP SIGNATURE- >>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ >>> >>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 >>> pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 >>> 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB >>> Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO >>> WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h >>> 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv >>> kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY >>> mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP >>> skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp >>> zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 >>> e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp >>> XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= >>> =VqkL >>> -END PGP SIGNATURE- >>> >>> >- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
I see this update on Windows which may have been responsible (suspicion only, haven’t rolled it back yet) https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-microcode-updates Was 8.5.45 built on Windows 10 in presence of this update ? Thanks, On Wed, 13 Nov 2019 at 17:55, M. Manna wrote: > Hi Chris, > > On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA256 >> >> On 11/13/19 11:20, M. Manna wrote: >> > HI Mark, >> > >> > On Wed, 13 Nov 2019 at 15:38, Mark Thomas >> > wrote: >> > >> >> On 12/11/2019 19:11, M. Manna wrote: >> >>> HI Mark, >> >>> >> >>> following my previous reply, we have now confirmed that it's >> >>> indeed >> >> 8.5.45 >> >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >> >>> took out 2 out of 50 servers from the load balancer config, >> >>> reverted tomcat, and redeployed. With near to identical user >> >>> traffic, the two servers are responding normally >> >>> without/without traffic with 8.5.41. The JVM dump looks a lot >> >>> better with 8.5.41. >> >>> >> >>> We do think that the recent changes in APR and some other >> >>> tomcat jar may have caused compatibility issue on Windows >> >>> server 2016 (64-bit) platform. But unfortunately, we cannot >> >>> pinpoint exactly what change may have caused this (i.e. actual >> >>> OS vs Security Updates). With this in mind, we are also being >> >>> wary to move to 8.5.47 as we don't know if the same issue will >> >> occur >> >>> again. Since 8.5.41 has been packaged with previously accepted >> >> application >> >>> installer, we are more comfortable rolling back. >> >> >> >> Just to confirm, you see this high CPU usage with a clean install >> >> (no additional web applications deployed, no configuration >> >> changes) on Windows 2016 DataCenter (64-bit)? >> >> >> >> If this is the case, it should be fairly easy to reproduce. >> >> >> >> Mark >> >> >> >> We do not deploy multiple applications. In fact, Under tomcat >> > webapps/ROOT we only have one application (ours). Each tomcat >> > instance is hosted on a VM (total 50) and all of them are >> > identically configured (server.xml, web.xml, logging, CPU/RAM). We >> > have not made any other configuration change between 8.5.41 and >> > 8.5.45. And yes, I agree with you that it's fairly easy to >> > reproduce. >> >> I think the question is whether or not your application is required to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? >> > > My apologies, but our application needs to be deployed. We have not (or > didn't try in the past) to simply deploy tomcat with stock application (in > other words, simply starting the tomcat OOB) on our prod servers. > This is the first time it has hit us with such disparity. I’ll try to > investigate and get a stock application data. But we may not be able to do > that quite easily as it’s in our production. > > What I can see is that 3 Windows updates may have been responsible for > this, but we aren’t sure about that. I’ll let you know if we can get > anything with the stock application instance. > > Thanks, > > - -chris > >> -BEGIN PGP SIGNATURE- >> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ >> >> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 >> pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 >> 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB >> Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO >> WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h >> 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv >> kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY >> mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP >> skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp >> zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 >> e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp >> XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= >> =VqkL >> -END PGP SIGNATURE- >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: tomcat thread incurring CPU load
Hi Chris, On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > On 11/13/19 11:20, M. Manna wrote: > > HI Mark, > > > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > > wrote: > > > >> On 12/11/2019 19:11, M. Manna wrote: > >>> HI Mark, > >>> > >>> following my previous reply, we have now confirmed that it's > >>> indeed > >> 8.5.45 > >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used > >>> took out 2 out of 50 servers from the load balancer config, > >>> reverted tomcat, and redeployed. With near to identical user > >>> traffic, the two servers are responding normally > >>> without/without traffic with 8.5.41. The JVM dump looks a lot > >>> better with 8.5.41. > >>> > >>> We do think that the recent changes in APR and some other > >>> tomcat jar may have caused compatibility issue on Windows > >>> server 2016 (64-bit) platform. But unfortunately, we cannot > >>> pinpoint exactly what change may have caused this (i.e. actual > >>> OS vs Security Updates). With this in mind, we are also being > >>> wary to move to 8.5.47 as we don't know if the same issue will > >> occur > >>> again. Since 8.5.41 has been packaged with previously accepted > >> application > >>> installer, we are more comfortable rolling back. > >> > >> Just to confirm, you see this high CPU usage with a clean install > >> (no additional web applications deployed, no configuration > >> changes) on Windows 2016 DataCenter (64-bit)? > >> > >> If this is the case, it should be fairly easy to reproduce. > >> > >> Mark > >> > >> We do not deploy multiple applications. In fact, Under tomcat > > webapps/ROOT we only have one application (ours). Each tomcat > > instance is hosted on a VM (total 50) and all of them are > > identically configured (server.xml, web.xml, logging, CPU/RAM). We > > have not made any other configuration change between 8.5.41 and > > 8.5.45. And yes, I agree with you that it's fairly easy to > > reproduce. > > I think the question is whether or not your application is required to > be deployed. Can you reproduce this issue with just the stock > applications bundled with Tomcat? > My apologies, but our application needs to be deployed. We have not (or didn't try in the past) to simply deploy tomcat with stock application (in other words, simply starting the tomcat OOB) on our prod servers. This is the first time it has hit us with such disparity. I’ll try to investigate and get a stock application data. But we may not be able to do that quite easily as it’s in our production. What I can see is that 3 Windows updates may have been responsible for this, but we aren’t sure about that. I’ll let you know if we can get anything with the stock application instance. Thanks, - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 > pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 > 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB > Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO > WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h > 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv > kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY > mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP > skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp > zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 > e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp > XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= > =VqkL > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Getting more information on connection refused error
Yes...i thought so...thanks for the JMX hint Chris. Best Regards, Saurav On Wed, Nov 13, 2019 at 10:00 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Saurav, > > On 11/13/19 10:19, Saurav Sarkar wrote: > > Hi All, > > > > We invoke one service which runs on a OSGI Virgo based embeddable > > tomcat servlet container. > > org.eclipse.gemini.web.tomcat_2.2.6.RELEASE.jar is the version we > > use. > > > > After sending some load to the service, it starts giving > > "connection refused error". > > > > errno: 111 (Connection refused), error: Connection refused (local > > port to address (), remote port > > 8443 to address ). > > > > I assume this is due to fact that the server socket has run out of > > connections and is not accepting any any more connection because > > this happens under heavy load testing. > > > > I would like to know how can i get more information on why > > connection was refused ? > > > > Will tomcat log this and what is logged under this scenario ? or > > does this reaches at all to tomcat and rather terminated at the OS > > layer itself ? > > The connections are being refused by the OS's TCP/IP stack. Tomcat > does not even know that these connections are being refused, so it > cannot put anything in the logs. > > If you monitor your Tomcat instance[1], you should be able to see how > many connections are in-flight and guess that you might be sometimes > hitting your limit. > > - -chris > > [1] http://tomcat.apache.org/presentations.html#latest-monitoring-with-j > mx > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3ML3MACgkQHPApP6U8 > pFh1EBAAhtUFRfSlsWdfHlcqrePQTV3wnVoNmJ3xQErZ4Cu26vM1Z6h+XhvvKxTw > WGJkLxLBm+BO/D+rrXY+DRczVZSCx1PnXM4YgsTtiegvwFj4F1C3n1HQaIqLYttZ > HSUh6iN06oFObDnNgQnkImx9l8QYB6snePoDM5fyQQaVdfJtm2yt8Ot+7TYJUZXW > tavrc5+jkGcbr/tlwUehN2OI+u+lZcK+gme8GKT60dBbllcLQ/D5IFTmsk/6bTya > iMvQ3NHPbji4J5hKxBYqMYB+wGoi7UljRusyOd4AQY8cbZ4UsuWrmWYb6yIAg6SW > YuHdzKQh2uy8Lrvljn43Nh4RguKnxMVycsWjhsXVP1HzbkEo/RdPXlaM4tLAdQak > y5A1ygb8Vlv1sYEzIYpsvgMdXO2eXSJ6UwvE+jC+Z+eKKrNMgBlYc0rS0f6KuCrB > rRjqw0sSZVmoOERbNJfYHU8JqxUC1TTkx9bb9AX1S2BuR/jq/9cC2lZllzjqe6Uc > P6kSavsW7r8CxB+pt6+GCG281Sa2tuednxeOXXom0liLmmRr8saWYd4H7zzMEGMq > ifiiMTsHyoSfBeyV7Ncp23RswywFeWe86/IpU6CSdlx7tb3YEU9i0Wtwl9Vof4Fq > nuQjVMznej/FCE+B156rLgwBUV5KWu5FM20R1cRdvmMJC+1AbUM= > =ebmC > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Getting more information on connection refused error
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Saurav, On 11/13/19 10:19, Saurav Sarkar wrote: > Hi All, > > We invoke one service which runs on a OSGI Virgo based embeddable > tomcat servlet container. > org.eclipse.gemini.web.tomcat_2.2.6.RELEASE.jar is the version we > use. > > After sending some load to the service, it starts giving > "connection refused error". > > errno: 111 (Connection refused), error: Connection refused (local > port to address (), remote port > 8443 to address ). > > I assume this is due to fact that the server socket has run out of > connections and is not accepting any any more connection because > this happens under heavy load testing. > > I would like to know how can i get more information on why > connection was refused ? > > Will tomcat log this and what is logged under this scenario ? or > does this reaches at all to tomcat and rather terminated at the OS > layer itself ? The connections are being refused by the OS's TCP/IP stack. Tomcat does not even know that these connections are being refused, so it cannot put anything in the logs. If you monitor your Tomcat instance[1], you should be able to see how many connections are in-flight and guess that you might be sometimes hitting your limit. - -chris [1] http://tomcat.apache.org/presentations.html#latest-monitoring-with-j mx -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3ML3MACgkQHPApP6U8 pFh1EBAAhtUFRfSlsWdfHlcqrePQTV3wnVoNmJ3xQErZ4Cu26vM1Z6h+XhvvKxTw WGJkLxLBm+BO/D+rrXY+DRczVZSCx1PnXM4YgsTtiegvwFj4F1C3n1HQaIqLYttZ HSUh6iN06oFObDnNgQnkImx9l8QYB6snePoDM5fyQQaVdfJtm2yt8Ot+7TYJUZXW tavrc5+jkGcbr/tlwUehN2OI+u+lZcK+gme8GKT60dBbllcLQ/D5IFTmsk/6bTya iMvQ3NHPbji4J5hKxBYqMYB+wGoi7UljRusyOd4AQY8cbZ4UsuWrmWYb6yIAg6SW YuHdzKQh2uy8Lrvljn43Nh4RguKnxMVycsWjhsXVP1HzbkEo/RdPXlaM4tLAdQak y5A1ygb8Vlv1sYEzIYpsvgMdXO2eXSJ6UwvE+jC+Z+eKKrNMgBlYc0rS0f6KuCrB rRjqw0sSZVmoOERbNJfYHU8JqxUC1TTkx9bb9AX1S2BuR/jq/9cC2lZllzjqe6Uc P6kSavsW7r8CxB+pt6+GCG281Sa2tuednxeOXXom0liLmmRr8saWYd4H7zzMEGMq ifiiMTsHyoSfBeyV7Ncp23RswywFeWe86/IpU6CSdlx7tb3YEU9i0Wtwl9Vof4Fq nuQjVMznej/FCE+B156rLgwBUV5KWu5FM20R1cRdvmMJC+1AbUM= =ebmC -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 11/13/19 11:20, M. Manna wrote: > HI Mark, > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > wrote: > >> On 12/11/2019 19:11, M. Manna wrote: >>> HI Mark, >>> >>> following my previous reply, we have now confirmed that it's >>> indeed >> 8.5.45 >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >>> took out 2 out of 50 servers from the load balancer config, >>> reverted tomcat, and redeployed. With near to identical user >>> traffic, the two servers are responding normally >>> without/without traffic with 8.5.41. The JVM dump looks a lot >>> better with 8.5.41. >>> >>> We do think that the recent changes in APR and some other >>> tomcat jar may have caused compatibility issue on Windows >>> server 2016 (64-bit) platform. But unfortunately, we cannot >>> pinpoint exactly what change may have caused this (i.e. actual >>> OS vs Security Updates). With this in mind, we are also being >>> wary to move to 8.5.47 as we don't know if the same issue will >> occur >>> again. Since 8.5.41 has been packaged with previously accepted >> application >>> installer, we are more comfortable rolling back. >> >> Just to confirm, you see this high CPU usage with a clean install >> (no additional web applications deployed, no configuration >> changes) on Windows 2016 DataCenter (64-bit)? >> >> If this is the case, it should be fairly easy to reproduce. >> >> Mark >> >> We do not deploy multiple applications. In fact, Under tomcat > webapps/ROOT we only have one application (ours). Each tomcat > instance is hosted on a VM (total 50) and all of them are > identically configured (server.xml, web.xml, logging, CPU/RAM). We > have not made any other configuration change between 8.5.41 and > 8.5.45. And yes, I agree with you that it's fairly easy to > reproduce. I think the question is whether or not your application is required to be deployed. Can you reproduce this issue with just the stock applications bundled with Tomcat? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= =VqkL -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Using CsrfPreventionFilter with GET-based submissions
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Peter,
On 11/12/19 18:52, Peter Kreuser wrote:
>
>
> Chris,
>
>> Am 13.11.2019 um 02:35 schrieb Christopher Schultz
>> :
>>
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
>>
>> Peter,
>>
>>> On 11/10/19 19:05, Peter Kreuser wrote: Chris,
>>>
Am 09.11.2019 um 03:58 schrieb Christopher Schultz
:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256
All,
I'm playing with the CsrfPreventionFilter and things are
working well in the following situations:
link text
and
...
As long as the URL has been passed through
request.encodeURL().
However, this one is causing me a problem:
...
This builds a form like this:
>>> action="https://host/path?org.apache.catalina.filters.CSRF_NONCE=[.
..
>>
]">
>> ...
Neither Firefox nor Chrome will send the query string present
in a action attribute if the method="GET". The method
must be "POST" in order for this to be sent. This is due to
the HTML standard[1].
Short of changing all methods to "POST", is there any
way around this?
I have read the code for CsrfPreventionFilter and it does
not appear that the nonce if stored anywhere except in the
CsrfResponseWrapper for the request (and the session's nonce
cache, but that isn't request-specific).
Would it be inappropriate to add the CSRF_NONCE to the
request attributes so that application code could use it
directly if necessary? Something like this:
... >>> name="org.apache.catalina.filters.CSRF_NONCE" value="<%=
request.getAttribute("CSRF_NONCE") %>" />
>>>
>>> If i remember correctly, this is the way struts handles CSRF
>>> Tokens.
>>
>> I'm not sure what Struts has to do with this. I'm using Tomcat's
>> CSRF filter which apparently cannot work with GET-based forms.
>> I'm not saying that a GET-based form is a good idea, but we have
>> a bunch of them so I'm looking into how they can be effectively
>> used with this implementation of a CSRF filter.
>
> I just wanted to point out that struts implements the CSRF
> protection only with a hidden field. That’s a bit of a hassle, plus
> you have to handle this per form. Wether it is a POST or a GET.
>
>> I'm really surprised this hasn't come up, yet. Maybe nobody
>> actually implements CSRF protection, or maybe nobody uses
>> Tomcat's filter to do it, or maybe nobody uses GET-based HTML
>> s. But I can't believe that I'm the only person in the
>> world who is trying to use all three at once.
>>
>>> However there the nonce comes directly from the session . Not
>>> request.
>>
>> The nonces are stored in the session, otherwise this wouldn't
>> work. But each request generates a new nonce, and that one would
>> be the "request's nonce".
>
> That’s another difference to struts and a bit of a nuisance, the
> nonce is only created once (and there is no cache). Once you remove
> it or generate a new nonce, all requests from “old” forms Will
> fail. But that is another story.
LOL. So it's another session id?
> But nevertheless the GET problem would be interesting to figure
> out. I will try this once I’m back from vacation.
It's easy to solve with the Tomcat CSRF prevention filter, but the
filter definitely needs a patch to help the application out.
It should be easy to put the "most recent" nonce into the request
scope and then all is well: the application can use it if necessary.
I'm thinking something simple like this:
diff --git a/java/org/apache/catalina/filters/Constants.java
b/java/org/apache/catalina/filters/Constants.java
index 4fe41cd0e8..eba743fb6f 100644
- --- a/java/org/apache/catalina/filters/Constants.java
+++ b/java/org/apache/catalina/filters/Constants.java
@@ -28,6 +28,9 @@ public final class Constants {
public static final String CSRF_NONCE_SESSION_ATTR_NAME =
"org.apache.catalina.filters.CSRF_NONCE";
+public static final String CSRF_NONCE_REQUEST_ATTR_NAME =
+"org.apache.catalina.filters.CSRF_REQUEST_NONCE";
+
public static final String CSRF_NONCE_REQUEST_PARAM =
"org.apache.catalina.filters.CSRF_NONCE";
diff --git
a/java/org/apache/catalina/filters/CsrfPreventionFilter.java
b/java/org/apache/catalina/filters/CsrfPreventionFilter.java
index fff5c2fedb..501c680d74 100644
- --- a/java/org/apache/catalina/filters/CsrfPreventionFilter.java
+++ b/java/org/apache/catalina/filters/CsrfPreventionFilter.java
@@ -128,6 +128,7 @@ public class CsrfPreventionFilter extends
CsrfPreventionFilterBase {
nonceCache.add(newNonce);
+
request.setAttribute(Constants.CSRF_NONCE_REQUEST_ATTR_NAME, newNonce);
wResponse = new CsrfResponseWrapper(res, newNonce);
} else {
wResponse = response;
Feedback welcome.
- -chris
-BEGIN PGP SIGNATURE-
Re: tomcat thread incurring CPU load
HI Mark, On Wed, 13 Nov 2019 at 15:38, Mark Thomas wrote: > On 12/11/2019 19:11, M. Manna wrote: > > HI Mark, > > > > following my previous reply, we have now confirmed that it's indeed > 8.5.45 > > with APR 1.2.23 that's causing such high JVM CPU usage. > > We used took out 2 out of 50 servers from the load balancer config, > > reverted tomcat, and redeployed. With near to identical user traffic, the > > two servers are responding normally without/without traffic with 8.5.41. > > The JVM dump looks a lot better with 8.5.41. > > > > We do think that the recent changes in APR and some other tomcat jar may > > have caused compatibility issue on Windows server 2016 (64-bit) platform. > > But unfortunately, we cannot pinpoint exactly what change may have caused > > this (i.e. actual OS vs Security Updates). With this in mind, we are also > > being wary to move to 8.5.47 as we don't know if the same issue will > occur > > again. Since 8.5.41 has been packaged with previously accepted > application > > installer, we are more comfortable rolling back. > > Just to confirm, you see this high CPU usage with a clean install (no > additional web applications deployed, no configuration changes) on > Windows 2016 DataCenter (64-bit)? > > If this is the case, it should be fairly easy to reproduce. > > Mark > > We do not deploy multiple applications. In fact, Under tomcat webapps/ROOT we only have one application (ours). Each tomcat instance is hosted on a VM (total 50) and all of them are identically configured (server.xml, web.xml, logging, CPU/RAM). We have not made any other configuration change between 8.5.41 and 8.5.45. And yes, I agree with you that it's fairly easy to reproduce. Thanks, > > > > > I would appreciate if this can be looked into. > > > > On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > > > >> Hey Mark (appreciate your response in US holiday time) > >> > >> On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: > >> > >>> On November 12, 2019 12:54:53 AM UTC, "M. Manna" > >>> wrote: > Just to give an update again: > > 1) We reverted the APR to 1.2.21 - but observed no difference. > 2) We took 3 thread dumps over 1 min interval (without any user > sessions) - > All threads are tomcat's internal pool threads. > > When we checked the thread details (using fasthread.io) - we didn't > see > any > of our application stack. Since there is no user traffic, this is > coming > >>> >from tomcat internally. At this stage, we cannot really figure out > what's > the root cause. > > Any help is appreciated. > >>> > >>> Migrated from what (full version info please)? > >>> > >> from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) > >> > >>> > >>> Operating system exact version? > >>> > >> Microsoft Windows Server 2016 DataCentre (64-bit) > >> > >>> > >>> JRE vendor and exact version? > >>> > >> C:\jdk1.8.0\bin>java.exe -version > >> java version "1.8.0_162" > >> Java(TM) SE Runtime Environment (build 1.8.0_162-b12) > >> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) > >> > >> > >>> Do you see the same behavior with the latest 8.5.x and latest Tomcat > >>> Native? > >>> > >> We are using APR 1.2.23 which I can also see in latest tomcat. Due to > >> production due diligence we cannot roll to a different version that > easily. > >> Normally, we lag behind by 2 monthly releases of tomcat. We also > reverted > >> the APR to 1.2.21 (but no difference). > >> > >>> > >>> What triggers this behaviour? > >>> > >> That is quite strange. Due to US holidays, we had a low traffic on our > >> servers, and nothing has crept in to suggest that it's > application-driven. > >> We took one tomcat instance out of 50 instances and removed all user > >> sessions (i.e. no application activities or threads). Upon restart of > >> tomcat, the CPU spike lingered past the initial servlet startup period. > We > >> monitored that over 1-2 hours but there was no difference. > >> > >>> > >>> How often do you see this behaviour? > >>> > >> We took 2 sets of data > >> 1) 3 Jstack dump based on 10 seconds interval. > >> 2) 3 jstack dump based on 1 min interval. > >> > >> Both the above reveals that all background threads (http, pool etc.) > were > >> from tomcat. We didn't have any application threads lingered in those 3 > >> samples. So yes we see this almost all the time if we take samples. > >> However, when we compared with pre-production instances (with Windows > >> server R2 x64 bit), we don't see such abnormal spike. In fact, the > >> application instance doesn't incur such a big CPU spike. Whilst > composing > >> this email, I am now thinking if the APR is indeed incompatible with > >> WIndows Server R2 (or the presence of any Windows Updates) which blocks > the > >> native poll() call longer than usual. > >> > >> An example is that on Windows Server 2012 - APR poll() call takes about > >> 30% CPU time - but with Windows Server
Re: tomcat thread incurring CPU load
On 12/11/2019 19:11, M. Manna wrote: > HI Mark, > > following my previous reply, we have now confirmed that it's indeed 8.5.45 > with APR 1.2.23 that's causing such high JVM CPU usage. > We used took out 2 out of 50 servers from the load balancer config, > reverted tomcat, and redeployed. With near to identical user traffic, the > two servers are responding normally without/without traffic with 8.5.41. > The JVM dump looks a lot better with 8.5.41. > > We do think that the recent changes in APR and some other tomcat jar may > have caused compatibility issue on Windows server 2016 (64-bit) platform. > But unfortunately, we cannot pinpoint exactly what change may have caused > this (i.e. actual OS vs Security Updates). With this in mind, we are also > being wary to move to 8.5.47 as we don't know if the same issue will occur > again. Since 8.5.41 has been packaged with previously accepted application > installer, we are more comfortable rolling back. Just to confirm, you see this high CPU usage with a clean install (no additional web applications deployed, no configuration changes) on Windows 2016 DataCenter (64-bit)? If this is the case, it should be fairly easy to reproduce. Mark > > I would appreciate if this can be looked into. > > On Tue, 12 Nov 2019 at 11:27, M. Manna wrote: > >> Hey Mark (appreciate your response in US holiday time) >> >> On Tue, 12 Nov 2019 at 07:51, Mark Thomas wrote: >> >>> On November 12, 2019 12:54:53 AM UTC, "M. Manna" >>> wrote: Just to give an update again: 1) We reverted the APR to 1.2.21 - but observed no difference. 2) We took 3 thread dumps over 1 min interval (without any user sessions) - All threads are tomcat's internal pool threads. When we checked the thread details (using fasthread.io) - we didn't see any of our application stack. Since there is no user traffic, this is coming >>> >from tomcat internally. At this stage, we cannot really figure out what's the root cause. Any help is appreciated. >>> >>> Migrated from what (full version info please)? >>> >> from 8.5.41 to 8.5.45 (we migrate 3 times a year, last was in June) >> >>> >>> Operating system exact version? >>> >> Microsoft Windows Server 2016 DataCentre (64-bit) >> >>> >>> JRE vendor and exact version? >>> >> C:\jdk1.8.0\bin>java.exe -version >> java version "1.8.0_162" >> Java(TM) SE Runtime Environment (build 1.8.0_162-b12) >> Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode) >> >> >>> Do you see the same behavior with the latest 8.5.x and latest Tomcat >>> Native? >>> >> We are using APR 1.2.23 which I can also see in latest tomcat. Due to >> production due diligence we cannot roll to a different version that easily. >> Normally, we lag behind by 2 monthly releases of tomcat. We also reverted >> the APR to 1.2.21 (but no difference). >> >>> >>> What triggers this behaviour? >>> >> That is quite strange. Due to US holidays, we had a low traffic on our >> servers, and nothing has crept in to suggest that it's application-driven. >> We took one tomcat instance out of 50 instances and removed all user >> sessions (i.e. no application activities or threads). Upon restart of >> tomcat, the CPU spike lingered past the initial servlet startup period. We >> monitored that over 1-2 hours but there was no difference. >> >>> >>> How often do you see this behaviour? >>> >> We took 2 sets of data >> 1) 3 Jstack dump based on 10 seconds interval. >> 2) 3 jstack dump based on 1 min interval. >> >> Both the above reveals that all background threads (http, pool etc.) were >> from tomcat. We didn't have any application threads lingered in those 3 >> samples. So yes we see this almost all the time if we take samples. >> However, when we compared with pre-production instances (with Windows >> server R2 x64 bit), we don't see such abnormal spike. In fact, the >> application instance doesn't incur such a big CPU spike. Whilst composing >> this email, I am now thinking if the APR is indeed incompatible with >> WIndows Server R2 (or the presence of any Windows Updates) which blocks the >> native poll() call longer than usual. >> >> An example is that on Windows Server 2012 - APR poll() call takes about >> 30% CPU time - but with Windows Server 2016 it's almost always 95%. >> >> >>> >>> And anything else you think might be relevant. >>> >> >> We are using end-2-end encryption using APR (with Certificate and >> SSLConfig resource setup in server.xml). But it's survived past 3 tomcat >> upgrades without any issue. >> Except OS we don't have any obvious culprit identified at the moment. >> >> Thanks, >> >>> >>> Mark >>> Thanks, On Mon, 11 Nov 2019 at 20:57, M. Manna wrote: > Hello All, > > Any thoughts regarding this? Slightly clueless at this point, so any > direction will be appreciated. > > We are seeing the poll taking all the CPU time. We are using >
Getting more information on connection refused error
Hi All, We invoke one service which runs on a OSGI Virgo based embeddable tomcat servlet container. org.eclipse.gemini.web.tomcat_2.2.6.RELEASE.jar is the version we use. After sending some load to the service, it starts giving "connection refused error". errno: 111 (Connection refused), error: Connection refused (local port to address (), remote port 8443 to address ). I assume this is due to fact that the server socket has run out of connections and is not accepting any any more connection because this happens under heavy load testing. I would like to know how can i get more information on why connection was refused ? Will tomcat log this and what is logged under this scenario ? or does this reaches at all to tomcat and rather terminated at the OS layer itself ? Best Regards, Saurav
Re: Tomcat 8.5.xx EOL
On 13/11/2019 15:05, RICT (Ricco Truelsen) wrote: > Hi > > Any official EOL on Tomcat 8.5.x yet? > I can find official info on the Tomcat 8.0.x EOL, but not on the Tomcat 8.5.x? No. https://markmail.org/message/6wycxatwzwycmf43 Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 8.5.xx EOL
Hi Any official EOL on Tomcat 8.5.x yet? I can find official info on the Tomcat 8.0.x EOL, but not on the Tomcat 8.5.x? Best regards, Ricco
