On Thu, Sep 27, 2018, 6:20 PM Louis Zipes <louis.zi...@gmcr.com> wrote:
> Chris, > I looked through the log some more and I see all of the types of Thread > Statuses. Blocked, Runnable, Waiting, etc.. Any in particular that I > should concentrate on? > Louis, can you please take multiple thread dumps(at least 3) with 5sec interval? You need to look for threads which are not moving at all or moving very slowly. They could be in any state. In worst case scenario you might see some blocking or deadlock. That will give you lead on what's going on inside the container. You can use IBM's thread dump analyzer for that. Ex. > "http-bio-7005-exec-128" daemon prio=6 tid=0x0000000026466800 nid=0x40e4 > waiting for monitor entry [0x00000000432ae000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > com.demantra.applicationServer.servlets.notifications.UserNotificationHelper.execute(UserNotificationHelper.java:117) > - waiting to lock <0x000000054d652c08> (a > com.demantra.applicationServer.metaDataObjects.user.UserList) > > ODBC on the actual Window machine hosting Tomcat is Oracle in > OraClient11g_home1 (we have a 12c Oracle database) with a pool timeout set > to 60. Should I be looking to turn on some tracing on the driver? > > Thanks, Louis > > > > -----Original Message----- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Wednesday, September 26, 2018 7:30 PM > To: users@tomcat.apache.org > Subject: Re: Application hanging on Tomcat 7.0.54 > > - - - external message, proceed with caution - - - > > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Louis, > > On 9/26/18 15:56, Louis Zipes wrote: > > Problem just re-occurred and so I was able to at least get a JSTACK > > (I assume it was Tomcat since it was the Java using the most memory > > on the machine). Here is the reoccurring message. I get more hits > > on but haven't dug through all of the Google hits yet (due to > > multi-tasking) so apologies up front if there is a simple answer to > > this. > > > > "Event_Manager_1413" daemon prio=6 tid=0x0000000024856000 > > nid=0x40c4 waiting on condition [0x0000000042dae000] > > java.lang.Thread.State: TIMED_WAITING (parking) at > > sun.misc.Unsafe.park(Native Method) - parking to wait for > > <0x00000005ab45f7b8> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) > > at > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. > awaitNanos(Unknown > > Source) at java.util.concurrent.LinkedBlockingQueue.poll(Unknown > > Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown > > Source) at > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > Source) at java.lang.Thread.run(Unknown Source) > > This thread is waiting for a task, and is essentially idle. You will > have many of these on a non-busy system. > > What are the other threads doing? > > > Locked ownable synchronizers: - None > > > >> Any comments/suggestions are appreciated! > > > > Your most likely problem is database connection pool > > mismanagement: connections aren't properly released and the pool > > empties. All threads are left waiting on available database > > connections which will never be replenished. > > > > I'm using the ojdbc6.jar if that is what you are referring to or is > > there a better setting somewhere. > > ODBC? What is your database? > > - -chris > > > > -----Original Message----- From: Christopher Schultz > > [mailto:ch...@christopherschultz.net] Sent: Wednesday, September > > 26, 2018 3:46 PM To: users@tomcat.apache.org Subject: Re: > > Application hanging on Tomcat 7.0.54 > > > > - - - external message, proceed with caution - - - > > > > > > Louis, > > > > On 9/26/18 14:42, Louis Zipes wrote: > >> Hi all, Tomcat 7.0.54 running on Windows 2012 > > > >> We are running a third party application on Tomcat and today we > >> have intermittently run in issues where the application stops > >> working. The big changes in our system is that we have added > >> more end users and we are at year end so of course everyone is > >> hitting the system hard. Even if we force a log out of all users > >> and stop all background jobs then the application doesn't > >> recover. > > > >> We see no active sessions on the database (our application is > >> connecting to an Oracle database) and I see no clear error > >> messages in either our third party application logs or the Tomcat > >> logs (ex. OutofMemory). When we go to the Windows Task Manager > >> we did not see the machine's Memory max'd out but admittedly I > >> didn't look at the Java session to see if was reaching its Heap > >> Max. The only thing that we noticed was that TCP connections > >> went down right after the restart. I did open up Jconsole under > >> Java and I did force a garbage collection but that didn't seem to > >> help. > > > >> We do have an Oracle Grid Control and we did get an alert in > >> regards to Metric: [HTTP Transaction] Perceived Time per Page > >> going past thresholds but not sure if that was just an old alert > >> with and old range that was set up a long time ago or is a really > >> valid clue. Since this is PRD we had to get it back up and > >> running so all I did was increase the Tomcat Xmx Heap size and > >> restarted. I'm not really confident that is the solution since > >> as mentioned you tend to see a clear out of memory error if it > >> was too small. > > > >> So a few questions: > > > > > >> 1) Does this sound like a known issue with this earlier > >> version of Tomcat? > > > > No. > > > >> 2) Should I turn up any logging on Tomcat and if so which > >> ones? > > > > Not yet. > > > >> 3) We didn't do a JSTACK dump while it was happening. Would > >> that have been useful? > > > > Absolutely. > > > >> 4) Do we need to play around with MaxThreads and/or > >> MaxConnections. We do have maxThreads in our server.mxl but in > >> DEV when we turned it down to a value = 5 hoping to overwhelm > >> it nothing bad happened. > > > > Don't change anything, yet. > > > >> Once again, we are limited to what we could do and collect since > >> it was PRD and we needed to restart it. We restarted the Tomcat > >> service and everything is processing fine for right now. I will > >> note that that we did have that bad Windows patch that prevented > >> it from stopping and starting cleanly > >> (https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdo > w > > > >> > n) > >> but we have taken the break fix patch and the daily restarts > >> seem to be fine since then. > > > >> Any comments/suggestions are appreciated! > > > > Your most likely problem is database connection pool > > mismanagement: connections aren't properly released and the pool > > empties. All threads are left waiting on available database > > connections which will never be replenished. > > > > -chris > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > --------------------------------------- CONFIDENTIALITY NOTICE: > > This message is for intended addressee(s) only and may contain > > information that is confidential, proprietary or exempt from > > disclosure. If you are not the intended recipient, please contact > > the sender immediately. Unauthorized use or distribution is > > prohibited and may be unlawful. > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlusFlwACgkQHPApP6U8 > pFi1oRAAiPywkBdpBqHBKpxkSS4n7/3X6fKH67ZHHbL4XUVS/3T7CNLwm7LXfWcE > K4+un6R8ghhaUUtok40PZPi1yC3bNcya2ON/kG7lFPP0WLXnNRUDjNo3gkbjfY+J > eu3pgTiPdZmjvvlzsvuXWAMihwXcB5isNYSq3Xsmz30i5w84dbfT4myNZUFMDNZT > jb7TCrrZ0UzrKEYmgSoPUC66R4ckWZAP7H4+Hf33IT23TwAcWPWD+jDhqTrdD3mE > m30digNNVWbb4D1IbPbk/S+YYCh9UOHys8WvBw9sYW3+IdCkfwj3EjlSeiBC7mNj > HHTYFODalBBPPWTaaeXtLMQuPWXYPASUTUnZOVFGeuNEyicqyhWS54nJB+MglkT2 > pWaM0guPjmF3ivpCIdkyHjJB0tFW4/FTPMaywET26N2j5Xw5hH+4iP9GlwGcDMA9 > yXJs+QeaP8nvuG7zgSmq+N6nojJBne3VZR2aZT/rgakJu1eP9HywVtE3URWZy4Ur > FJUYzsRLtvAIIBRhJSQNVuIvlmMp5V0QRBfPYRh7gPnBpFJQGmBoVORbVOKxysfu > 2WMpBVGPJrsXG92hvuyK1o/S9zAadTsWqUTJ3nGlRx4bbPl3NCZqXB0RRTLJe1yb > xcBSwfMYHmxBTCpajFqLdMSEkGpDUJNf+NZ90+h7m7rGjSSoSug= > =X6UE > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------- > CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and > may contain information that is confidential, proprietary or exempt from > disclosure. If you are not the intended recipient, please contact the > sender immediately. Unauthorized use or distribution is prohibited and may > be unlawful. >
