Mark and others, On Mon, 18 Nov 2019 at 12:01, Mark Thomas <ma...@apache.org> wrote:
> OK, it looks like I can reproduce this. > > Steps to reproduce: > > - Windows 2016 Server fully patched > - Java 1.8.0u144 > - Install Tomcat 8.5.45 from windows installer > - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 > - Modify server.xml to use Http11AprProtocol on port 8080 > - Make a single request > > I then see 1 core running at 100% until the connection times out after > 20s. Make another request and a core goes back up to 100% for 20s (the > default keep-alive time out). > I have also successfully reproduced this with making a single request (sorry for not replying in the weekend). Not sure how your graph looked like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as the request hit the server. and it didn't go down at all. Thanks, > > Next steps are to try and track down the root cause. > > Mark > > > > > Mark and M, > > > > On 11/13/19 19:31, Mark Thomas wrote: > >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" > >> <manme...@gmail.com> wrote: > >>> I see this update on Windows which may have been responsible > >>> (suspicion only, haven’t rolled it back yet) > >>> > >>> > >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > > ocode-updates > >>> > >>> > >>> > > Was 8.5.45 built on Windows 10 in presence of this update ? > > > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully > >> patched at the time of the build Windows 7 64-bit VM. > > Also it doesn't matter because binaries don't include CPU microcode. > > > > It's more likely that the target system has microcode updates such as > > these that may negatively impact performance. > > > > -chris > > > >>> > >>> Thanks, > >>> > >>> On Wed, 13 Nov 2019 at 17:55, M. Manna <manme...@gmail.com> > >>> wrote: > >>> > >>>> Hi Chris, > >>>> > >>>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < > >>>> ch...@christopherschultz.net> wrote: > >>>> > >> On 11/13/19 11:20, M. Manna wrote: > >>>>>>> HI Mark, > >>>>>>> > >>>>>>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas > >>>>>>> <ma...@apache.org> wrote: > >>>>>>> > >>>>>>>> On 12/11/2019 19:11, M. Manna wrote: > >>>>>>>>> HI Mark, > >>>>>>>>> > >>>>>>>>> following my previous reply, we have now confirmed > >>>>>>>>> that it's indeed > >>>>>>>> 8.5.45 > >>>>>>>>> with APR 1.2.23 that's causing such high JVM CPU > >>>>>>>>> usage. We used took out 2 out of 50 servers from the > >>>>>>>>> load balancer config, reverted tomcat, and > >>>>>>>>> redeployed. With near to identical user traffic, the > >>>>>>>>> two servers are responding normally without/without > >>>>>>>>> traffic with 8.5.41. The JVM dump looks a lot better > >>>>>>>>> with 8.5.41. > >>>>>>>>> > >>>>>>>>> We do think that the recent changes in APR and some > >>>>>>>>> other tomcat jar may have caused compatibility issue > >>>>>>>>> on Windows server 2016 (64-bit) platform. But > >>>>>>>>> unfortunately, we cannot pinpoint exactly what change > >>>>>>>>> may have caused this (i.e. actual OS vs Security > >>>>>>>>> Updates). With this in mind, we are also being wary > >>>>>>>>> to move to 8.5.47 as we don't know if the same issue > >>>>>>>>> will > >>>>>>>> occur > >>>>>>>>> again. Since 8.5.41 has been packaged with previously > >>>>>>>>> accepted > >>>>>>>> application > >>>>>>>>> installer, we are more comfortable rolling back. > >>>>>>>> > >>>>>>>> Just to confirm, you see this high CPU usage with a > >>>>>>>> clean install (no additional web applications deployed, > >>>>>>>> no configuration changes) on Windows 2016 DataCenter > >>>>>>>> (64-bit)? > >>>>>>>> > >>>>>>>> If this is the case, it should be fairly easy to > >>>>>>>> reproduce. > >>>>>>>> > >>>>>>>> Mark > >>>>>>>> > >>>>>>>> We do not deploy multiple applications. In fact, Under > >>>>>>>> tomcat > >>>>>>> webapps/ROOT we only have one application (ours). Each > >>>>>>> tomcat instance is hosted on a VM (total 50) and all of > >>>>>>> them are identically configured (server.xml, web.xml, > >>>>>>> logging, CPU/RAM). We have not made any other > >>>>>>> configuration change between 8.5.41 and 8.5.45. And yes, > >>>>>>> I agree with you that it's fairly easy to reproduce. > > > >> I think the question is whether or not your application is > >> required > >>>> to > >> be deployed. Can you reproduce this issue with just the stock > >> applications bundled with Tomcat? > > > >>>>> > >>>>> My apologies, but our application needs to be deployed. We > >>>>> have not > >>>> (or > >>>>> didn't try in the past) to simply deploy tomcat with stock > >>>> application (in > >>>>> other words, simply starting the tomcat OOB) on our prod > >>>>> servers. This is the first time it has hit us with such > >>>>> disparity. I’ll try to investigate and get a stock > >>>>> application data. But we may not be able > >>>> to do > >>>>> that quite easily as it’s in our production. > >>>>> > >>>>> What I can see is that 3 Windows updates may have been > >>>>> responsible > >>>> for > >>>>> this, but we aren’t sure about that. I’ll let you know if we > >>>>> can get anything with the stock application instance. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> - -chris > >>>>> > >>>>> > >>>>> > >>> --------------------------------------------------------------------- > >>>>> > >>> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>> For additional commands, e-mail: > >>>>> users-h...@tomcat.apache.org > >>>>> > >>>>> > > > > > >> --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
