Hello Thomas, Thanks for your input here, what's your weapon of choice to identify this thread bar thread dump? I just downloaded jvmtop from github but that didn't seem to give me any clue at all about independent threads.
Cheers Alan -----Original Message----- From: Thomas Hoffmann (Speed4Trade GmbH) <thomas.hoffm...@speed4trade.com.INVALID> Sent: 04 February 2022 09:18 To: Tomcat Users List <users@tomcat.apache.org> Subject: AW: help with high cpu usage Hello, when I encounter high CPU usage, it's best to identify the thread Id which is eating CPU. Making a thread dump, you can then search for the thread id within this dump. This works good for long lasting threads. If the CPU eating thread changes quickly, it's harder to figure out. Greetings, Thomas ________________________________ Von: Alan F <shiva...@hotmail.com> Gesendet: Freitag, 4. Februar 2022 00:02:49 An: Tomcat Users List Betreff: Re: help with high cpu usage John thanks so much !! Will pass this on tomorrow. Cheers. ________________________________ From: john.e.gr...@wellsfargo.com.INVALID <john.e.gr...@wellsfargo.com.INVALID> Sent: 03 February 2022 22:45 To: users@tomcat.apache.org <users@tomcat.apache.org> Subject: RE: help with high cpu usage Alan, > -----Original Message----- > From: Alan F <shiva...@hotmail.com> > Sent: Thursday, February 03, 2022 2:51 PM > To: Tomcat Users List <users@tomcat.apache.org> > Subject: RE: help with high cpu usage > > My bad here are the correct dumps 10 secs apart > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0zNzY2ZmZmNy0wZDgyL > TRhZTItYmE3Mi0zMWQyYTYwN2M1ZjgudHh0&__;!!F9svGWnIaVPGSwU!- > fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG > st67ghaaLsg$ > > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS00NTE3MWUxNy1jYWRi > LTRkY2UtODBlNS1lMDk0YTJjNTg1OGEudHh0&__;!!F9svGWnIaVPGSwU!- > fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG > st67j-3O5xU$ > > > > > -----Original Message----- > From: john.e.gr...@wellsfargo.com.INVALID > <john.e.gr...@wellsfargo.com.INVALID> > Sent: 03 February 2022 19:33 > To: users@tomcat.apache.org > Subject: RE: help with high cpu usage > > Alan, > > > > -----Original Message----- > > From: Alan F <shiva...@hotmail.com> > > Sent: Thursday, February 03, 2022 12:19 PM > > To: Tomcat Users List <users@tomcat.apache.org> > > Subject: help with high cpu usage > > > > Had some issues today with one prod host. One is fine the other has > > stuck around 80%Cpu. > > Ive taken a thread dump from both hosts and would appreciate someone > > give a once over what it may be before I kill and restart. They are > > clustered nodes so running identical apps and loadbalanced by a > > hardware > balancer. > > > > Node1 is ok (relatively!) > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > > > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS1hODllYzBkZS01OGJjLTQ > > > 2ZDQtYWRhNS1kYjkxZjM2NjI1ZTAudHh0&__;!!F9svGWnIaVPGSwU!91fvMc > > RzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-mDKbTl_dmUSa0LaAolXlhipl- > > Fk2pQ$ > > > > > > Node 2 still has high CPU usage > > https://urldefense.com/v3/__https://fastthread.io/my-thread- > > > report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yN2I0YWY4Mi05OWFhL > > > TQ3YjYtOGQ2My0wMDMwZjlkNDQzNjMudHh0&__;!!F9svGWnIaVPGSwU!9 > > 1fvMcRzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn- > > mDKbTl_dmUSa0LaAolXlhipkmLFr3E$ > > Taking thread dumps was a good idea but I see very little activity on node 2. > It looks like two threads are in Hibernate's > EntityEntryContext.downgradeLocks() method, one is taking the thread > dump itself, and another is reading a request from a client, I think. > I would not expect this to add up to 80% CPU usage unless one of those > threads is stuck in a loop. Comparing multiple thread dumps taken > 5-10 seconds apart would help answer this question. How much GC is > there? Could these Hibernate queries be pulling a huge amount of data > from the DB, thus causing a lot of GC activity? > > Node 1 looks idle except for the thread taking the thread dump. > > Do you know for sure that it's the Tomcat process that is using the CPU? Of the 3 dumps from the same node, the first two are 3 hours apart, yet the two query threads seem to still be doing the same thing. (I verified that the timestamps in the dumps are different but maybe I made a mistake.) They are both making the Hibernate downgradeLocks call. This is occurring in the context of committing a transaction. That definitely makes it look like those threads are somehow either stuck or are looping. The 3rd dump has those same two threads actually making queries to Oracle. So within about 10 seconds, we have two threads committing transactions followed by making additional queries. Definitely show these to the developers if you haven't already. I didn't follow what you said about the GC. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
