RE: Application hanging on Tomcat 7.0.54

2018-10-01 Thread Louis Zipes 
Ok, thanks as always for the advice!  We think we figured out the user error(s) 
that caused the issue but I'm now more comfortable with pulling the Thread 
Dumps/VisualVM quicker so we can at least get those before we have to restart 
the application.

Thanks, Louis





> timeout set to 60.  Should I be looking to turn on some tracing on the
> driver?
60 what?

60 was the Pool Timeout

-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Friday, September 28, 2018 6:43 PM
To: users@tomcat.apache.org
Subject: Re: Application hanging on Tomcat 7.0.54

- - - external message, proceed with caution - - -


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/27/18 09:19, Louis Zipes wrote:
> Even though these hangs are critical to find, any plugin or
> additional code that I wish to put on our PRD server have to go
> through our SOX process so it can be cumbersome.  Are any of the
> monitoring techniques that you mention 'out of the box' with
> Tomcat 7.0.54 or JDK 1.7.  I can certainly open the JCONSOLE in the
> java bin folder  but I don't have those nice Spring Boot Add-ons
> documented in say
> https://stackoverflow.com/questions/36587023/how-to-debug-log-tomcat-j
dbc-connection-pools-connections

You
>
>
don't need Spring Boot or anything like that. jconsole (or, better
yet VisualVM) can certainly observe your connection pool status as well
as runtime thread state, etc.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluurl0ACgkQHPApP6U8
pFjcdBAAqDBUm3gIasaDXdo0b0ZSyL56ADyTNa4c3hDdXLy4KJQjhYEltm23JktE
VCdTDdZj7T/dYAqyJvNXugak4f4SkBS5BhGyHBK+mfLmT8H3rqNNi+lUHHLlPHJn
O03YZfNz/HtYpajSgReaFm8SQQ/9RYhtlyVA0ZH+6UH1RVsGAxnqNAaUReCMyR1r
sN4lhwzft/xQCJ+riQ+rtrgeCpQJaZts88Bph8PFQ9CMKu1tWDU4RvgqyxMJ3uzm
Ciq4wQ5zU8ke5nDuiLryfli7ODj07pDgt0HJTfVggbeNi8psNps2We+s0sjLFwbs
ldw27q00H7vJsgyds0FDtbqoGQ1C8WkmfO/L6G47VxO55Jps8O4nbaaweRNtDWJO
N/RuBU2UvvzVgKuKtzBNXnKF7mIEssSLnb0OyY7ANuQBwhy/FmcHWey79VLYvsKt
ZD9DzCKE0ekJLg1Pfc2bMQsiDI/O9lFB/6JeknLs7fde1b9j3+8DeL/kUDHEt4go
lG2rs44/2GqTfNdhBEU8ERIaxretstK/ms3/zLoeVYO7VQypHeeagDHH7TOfztbR
hlYX5egZRCs0VT61gp4A2BONhbSWcldFbrUa8zRXnYLPe0WZMUEcmDsVt7YXv85x
2UJtZL6jFawEr91L8bjB9bKPBQFdilJFok1o/iib1ltXhXsTItE=
=IVM1
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

---
CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and may 
contain information that is confidential, proprietary or exempt from 
disclosure. If you are not the intended recipient, please contact the sender 
immediately. Unauthorized use or distribution is prohibited and may be unlawful.

Re: Application hanging on Tomcat 7.0.54

2018-09-28 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/27/18 09:19, Louis Zipes wrote:
> Even though these hangs are critical to find, any plugin or 
> additional code that I wish to put on our PRD server have to go 
> through our SOX process so it can be cumbersome.  Are any of the 
> monitoring techniques that you mention 'out of the box' with
> Tomcat 7.0.54 or JDK 1.7.  I can certainly open the JCONSOLE in the
> java bin folder  but I don't have those nice Spring Boot Add-ons
> documented in say 
> https://stackoverflow.com/questions/36587023/how-to-debug-log-tomcat-j
dbc-connection-pools-connections

You
>
> 
don't need Spring Boot or anything like that. jconsole (or, better
yet VisualVM) can certainly observe your connection pool status as well
as runtime thread state, etc.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluurl0ACgkQHPApP6U8
pFjcdBAAqDBUm3gIasaDXdo0b0ZSyL56ADyTNa4c3hDdXLy4KJQjhYEltm23JktE
VCdTDdZj7T/dYAqyJvNXugak4f4SkBS5BhGyHBK+mfLmT8H3rqNNi+lUHHLlPHJn
O03YZfNz/HtYpajSgReaFm8SQQ/9RYhtlyVA0ZH+6UH1RVsGAxnqNAaUReCMyR1r
sN4lhwzft/xQCJ+riQ+rtrgeCpQJaZts88Bph8PFQ9CMKu1tWDU4RvgqyxMJ3uzm
Ciq4wQ5zU8ke5nDuiLryfli7ODj07pDgt0HJTfVggbeNi8psNps2We+s0sjLFwbs
ldw27q00H7vJsgyds0FDtbqoGQ1C8WkmfO/L6G47VxO55Jps8O4nbaaweRNtDWJO
N/RuBU2UvvzVgKuKtzBNXnKF7mIEssSLnb0OyY7ANuQBwhy/FmcHWey79VLYvsKt
ZD9DzCKE0ekJLg1Pfc2bMQsiDI/O9lFB/6JeknLs7fde1b9j3+8DeL/kUDHEt4go
lG2rs44/2GqTfNdhBEU8ERIaxretstK/ms3/zLoeVYO7VQypHeeagDHH7TOfztbR
hlYX5egZRCs0VT61gp4A2BONhbSWcldFbrUa8zRXnYLPe0WZMUEcmDsVt7YXv85x
2UJtZL6jFawEr91L8bjB9bKPBQFdilJFok1o/iib1ltXhXsTItE=
=IVM1
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Application hanging on Tomcat 7.0.54

2018-09-28 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/27/18 08:49, Louis Zipes wrote:
> I looked through the log some more and I see all of the types of 
> Thread Statuses.  Blocked, Runnable, Waiting, etc..  Any in 
> particular that I should concentrate on?
The thread state doesn't always tell the whole story. Sometimes, it's
shown as RUNNABLE but it really means it's blocked. The stack trace is
more useful.

> 
> Ex. "http-bio-7005-exec-128" daemon prio=6 tid=0x26466800
> nid=0x40e4 waiting for monitor entry [0x432ae000] 
> java.lang.Thread.State: BLOCKED (on object monitor) at
> com.demantra.applicationServer.servlets.notifications.UserNotification
Helper.execute(UserNotificationHelper.java:117)
>
> 
- - waiting to lock <0x00054d652c08> (a
com.demantra.applicationServer.metaDataObjects.user.UserList)

That certainly looks like it's waiting on something. If, as Suvendu
suggests, you take more than one thread-dump in a row -- maybe 10
seconds between dumps -- and compare them, you can detect whether or
not those threads just happen to be waiting one time, or if they are
waiting forever.

> ODBC on the actual Window machine hosting Tomcat is Oracle in 
> OraClient11g_home1  (we have a 12c Oracle database) with a pool 
> timeout set to 60.  Should I be looking to turn on some tracing on 
> the driver?
60 what?

Enabling tracing is likely to generate a LOT of logging output. You
can probably discover a lot from thread dumps without changing anything.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluurbsACgkQHPApP6U8
pFgf/BAAkI/Bqy/mqcguHVaZYY1x3dOmIsGoVvIo2jKi6PUqQOdsPCwb1FR97Eoq
jtyuAiOk9bUnDUDE6l/1fNV8e2z6QHrVCGWog+3sjejpJGv4Dkfslm+wW1PsQU+/
fsd3ihtWRSyAqSNhfKuVB394yBubKuVqvHGYK6rRjyRMZ/b6JAEGOEW6/B3QBUV/
bB3TfYPZ1ih6lBrCIHaHYlFwwcVIVqqT7YJkAurdYdopqke9WY+fkgNO3i855zmV
yujcHPMjC8Y7PW9rw5VOlevYBsuxS3jf/62iXyj4fSbhUzpxrRAk0qc/dukzioFF
rgXs9VY3sJBvcmGt9bLfMiPwB9l52a/lloIiNYhitZ6ASaEKYAQqukpBwqL86NuC
8aSUNAzCt80Tr1DZZLP46KnzCfHDWocobF42locqHhe3iJ7PahfsQVVKYJ1cwyga
70qy+oBf3fzBTWq2LrA+a3TaOyOO7SVSY+NmKMS+7FNiicR18ace50PAHzjBN5eB
hOCaqeMdJh05Ip7u95VjBUqxlnc+fNli7D1rRy/jXOO/T3eqWg5nMUyrGxSVxomx
C4CU6hrc44dQeSZg2jXkgzlKOLGPmeOO+T8nPZ26Mdg2dlbbuSvSE0mydJvZzAao
Y1xmLJJDe6ciMYMkSP3JKC1bjylrRhZAZ+MC4C2iJKY564/mWQQ=
=uG5n
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Application hanging on Tomcat 7.0.54

2018-09-27 Thread Suvendu Sekhar Mondal 
On Thu, Sep 27, 2018, 6:20 PM Louis Zipes  wrote:

> Chris,
> I looked through the log some more and I see all of the types of Thread
> Statuses.  Blocked, Runnable, Waiting, etc..  Any in particular that I
> should concentrate on?
>

Louis, can you please take multiple thread dumps(at least 3) with 5sec
interval? You need to look for threads which are not moving at all or
moving very slowly. They could be in any state. In worst case scenario you
might see some blocking or deadlock. That will give you lead on what's
going on inside the container. You can use IBM's thread dump analyzer for
that.

Ex.
> "http-bio-7005-exec-128" daemon prio=6 tid=0x26466800 nid=0x40e4
> waiting for monitor entry [0x432ae000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.demantra.applicationServer.servlets.notifications.UserNotificationHelper.execute(UserNotificationHelper.java:117)
> - waiting to lock <0x00054d652c08> (a
> com.demantra.applicationServer.metaDataObjects.user.UserList)
>
> ODBC on the actual Window machine hosting Tomcat is Oracle in
> OraClient11g_home1  (we have a 12c Oracle database) with a pool timeout set
> to 60.  Should I be looking to turn on some tracing on the driver?
>
> Thanks, Louis
>
>
>
> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Wednesday, September 26, 2018 7:30 PM
> To: users@tomcat.apache.org
> Subject: Re: Application hanging on Tomcat 7.0.54
>
> - - - external message, proceed with caution - - -
>
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Louis,
>
> On 9/26/18 15:56, Louis Zipes wrote:
> > Problem just re-occurred and so I was able to at least get a JSTACK
> > (I assume it was Tomcat since it was the Java using the most memory
> > on the machine).  Here is the reoccurring message.  I get more hits
> > on but haven't dug through all of the Google hits yet (due to
> > multi-tasking) so apologies up front if there is a simple answer to
> > this.
> >
> > "Event_Manager_1413" daemon prio=6 tid=0x24856000
> > nid=0x40c4 waiting on condition [0x42dae000]
> > java.lang.Thread.State: TIMED_WAITING (parking) at
> > sun.misc.Unsafe.park(Native Method) - parking to wait for
> > <0x0005ab45f7b8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >
> >
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> > at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> awaitNanos(Unknown
> > Source) at java.util.concurrent.LinkedBlockingQueue.poll(Unknown
> > Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
> > Source) at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source) at java.lang.Thread.run(Unknown Source)
>
> This thread is waiting for a task, and is essentially idle. You will
> have many of these on a non-busy system.
>
> What are the other threads doing?
>
> > Locked ownable synchronizers: - None
> >
> >> Any comments/suggestions are appreciated!
> >
> > Your most likely problem is database connection pool
> > mismanagement: connections aren't properly released and the pool
> > empties. All threads are left waiting on available database
> > connections which will never be replenished.
> >
> > I'm using the ojdbc6.jar if that is what you are referring to or is
> > there a better setting somewhere.
>
> ODBC? What is your database?
>
> - -chris
>
>
> > -Original Message- From: Christopher Schultz
> > [mailto:ch...@christopherschultz.net] Sent: Wednesday, September
> > 26, 2018 3:46 PM To: users@tomcat.apache.org Subject: Re:
> > Application hanging on Tomcat 7.0.54
> >
> > - - - external message, proceed with caution - - -
> >
> >
> > Louis,
> >
> > On 9/26/18 14:42, Louis Zipes wrote:
> >> Hi all, Tomcat 7.0.54 running on Windows 2012
> >
> >> We are running a third party application on Tomcat and today we
> >> have intermittently run in issues where the application stops
> >> working.  The big changes in our system is that we have added
> >> more end users and we are at year end so of course everyone is
> >> hitting the system hard. Even if we force a log out of all users
> >> and stop all background jobs then the application doesn't
> >> recover.
> >
> >> We see no active sessions on the database (our application is
> >> connectin

RE: Application hanging on Tomcat 7.0.54

2018-09-27 Thread Louis Zipes 
Hi Guido,
Even though these hangs are critical to find, any plugin or additional code 
that I wish to put on our PRD server have to go through our SOX process so it 
can be cumbersome.  Are any of the monitoring techniques that you mention 'out 
of the box' with Tomcat 7.0.54 or JDK 1.7.  I can certainly open the JCONSOLE 
in the java bin folder  but I don't have those nice Spring Boot Add-ons 
documented in say 
https://stackoverflow.com/questions/36587023/how-to-debug-log-tomcat-jdbc-connection-pools-connections

I assume I would need something additional to do the following or not true:

>>The Tomcat JDBC Connection Pool for instance provide a MBean view for every 
>>connection with it's parameters. You may live watch the number of active, 
>>idle, ... connections (with a "well hidden" chart feature: Double-Click to 
>>expand some value representations; this here to a chart)

Thank you!

-Original Message-
From: Jäkel, Guido [mailto:g.jae...@dnb.de]
Sent: Thursday, September 27, 2018 2:29 AM
To: 'Tomcat Users List'
Subject: RE: Application hanging on Tomcat 7.0.54

- - - external message, proceed with caution - - -


Dear Louis,

I would recommend to use a tool like JVisualVM (with Plugins*) to take a look 
on this things while it's still running or have blocked. You may live watch 
things like running threads or the Java heap occupation or investigate JVM, 
Java and Tomcat parameters (and even run some actions) via JMX with 
"MBean-Explorer". You may trigger to generate a heap dump or force FullGC and 
even do CPU or Memory profiling.

The Tomcat JDBC Connection Pool for instance provide a MBean view for every 
connection with it's parameters. You may live watch the number of active, idle, 
... connections (with a "well hidden" chart feature: Double-Click to expand 
some value representations; this here to a chart)

Tomcat (i.e. Catalina) have something like a running request scoreboard, also. 
You may take and identify from that all long running or blocked requests. On 
the MBean, it's at 
"Catalina|RequestProcessor|||currentURI" and 
corresponding parameters.


(*) You would have to install some plugins inside the JVisualVM Tool, e.g. 
"Visual GC", "VisualVM Buffer Monitor", "Thread Inspector", "VisualVM BMeans"

Greetings

Guido

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

---
CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and may 
contain information that is confidential, proprietary or exempt from 
disclosure. If you are not the intended recipient, please contact the sender 
immediately. Unauthorized use or distribution is prohibited and may be unlawful.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Application hanging on Tomcat 7.0.54

2018-09-27 Thread Louis Zipes 
Chris,
I looked through the log some more and I see all of the types of Thread 
Statuses.  Blocked, Runnable, Waiting, etc..  Any in particular that I should 
concentrate on?

Ex.
"http-bio-7005-exec-128" daemon prio=6 tid=0x26466800 nid=0x40e4 
waiting for monitor entry [0x432ae000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
com.demantra.applicationServer.servlets.notifications.UserNotificationHelper.execute(UserNotificationHelper.java:117)
- waiting to lock <0x00054d652c08> (a 
com.demantra.applicationServer.metaDataObjects.user.UserList)

ODBC on the actual Window machine hosting Tomcat is Oracle in 
OraClient11g_home1  (we have a 12c Oracle database) with a pool timeout set to 
60.  Should I be looking to turn on some tracing on the driver?

Thanks, Louis



-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Wednesday, September 26, 2018 7:30 PM
To: users@tomcat.apache.org
Subject: Re: Application hanging on Tomcat 7.0.54

- - - external message, proceed with caution - - -


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/26/18 15:56, Louis Zipes wrote:
> Problem just re-occurred and so I was able to at least get a JSTACK
> (I assume it was Tomcat since it was the Java using the most memory
> on the machine).  Here is the reoccurring message.  I get more hits
> on but haven't dug through all of the Google hits yet (due to
> multi-tasking) so apologies up front if there is a simple answer to
> this.
>
> "Event_Manager_1413" daemon prio=6 tid=0x24856000
> nid=0x40c4 waiting on condition [0x42dae000]
> java.lang.Thread.State: TIMED_WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x0005ab45f7b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
awaitNanos(Unknown
> Source) at java.util.concurrent.LinkedBlockingQueue.poll(Unknown
> Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
> Source) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) at java.lang.Thread.run(Unknown Source)

This thread is waiting for a task, and is essentially idle. You will
have many of these on a non-busy system.

What are the other threads doing?

> Locked ownable synchronizers: - None
>
>> Any comments/suggestions are appreciated!
>
> Your most likely problem is database connection pool
> mismanagement: connections aren't properly released and the pool
> empties. All threads are left waiting on available database
> connections which will never be replenished.
>
> I'm using the ojdbc6.jar if that is what you are referring to or is
> there a better setting somewhere.

ODBC? What is your database?

- -chris


> -Original Message- From: Christopher Schultz
> [mailto:ch...@christopherschultz.net] Sent: Wednesday, September
> 26, 2018 3:46 PM To: users@tomcat.apache.org Subject: Re:
> Application hanging on Tomcat 7.0.54
>
> - - - external message, proceed with caution - - -
>
>
> Louis,
>
> On 9/26/18 14:42, Louis Zipes wrote:
>> Hi all, Tomcat 7.0.54 running on Windows 2012
>
>> We are running a third party application on Tomcat and today we
>> have intermittently run in issues where the application stops
>> working.  The big changes in our system is that we have added
>> more end users and we are at year end so of course everyone is
>> hitting the system hard. Even if we force a log out of all users
>> and stop all background jobs then the application doesn't
>> recover.
>
>> We see no active sessions on the database (our application is
>> connecting to an Oracle database) and I see no clear error
>> messages in either our third party application logs or the Tomcat
>> logs (ex. OutofMemory).  When we go to the Windows Task Manager
>> we did not see the machine's Memory max'd out but admittedly I
>> didn't look at the Java session to see if was reaching its Heap
>> Max.  The only thing that we noticed was that TCP connections
>> went down right after the restart.  I did open up Jconsole under
>> Java and I did force a garbage collection but that didn't seem to
>> help.
>
>> We do have an Oracle Grid Control and we did get an alert in
>> regards to Metric: [HTTP Transaction] Perceived Time per Page
>> going past thresholds but not sure if that was just an old alert
>> with and old range that was set up a long time ago or is a really
>> valid clue.Since this is PRD we

RE: Application hanging on Tomcat 7.0.54

2018-09-27 Thread Jäkel , Guido 
Dear Louis,

I would recommend to use a tool like JVisualVM (with Plugins*) to take a look 
on this things while it's still running or have blocked. You may live watch 
things like running threads or the Java heap occupation or investigate JVM, 
Java and Tomcat parameters (and even run some actions) via JMX with 
"MBean-Explorer". You may trigger to generate a heap dump or force FullGC and 
even do CPU or Memory profiling.

The Tomcat JDBC Connection Pool for instance provide a MBean view for every 
connection with it's parameters. You may live watch the number of active, idle, 
... connections (with a "well hidden" chart feature: Double-Click to expand 
some value representations; this here to a chart)

Tomcat (i.e. Catalina) have something like a running request scoreboard, also. 
You may take and identify from that all long running or blocked requests. On 
the MBean, it's at 
"Catalina|RequestProcessor|||currentURI" and 
corresponding parameters.


(*) You would have to install some plugins inside the JVisualVM Tool, e.g. 
"Visual GC", "VisualVM Buffer Monitor", "Thread Inspector", "VisualVM BMeans"

Greetings

Guido

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Application hanging on Tomcat 7.0.54

2018-09-26 Thread Guang Chao 
On Thu, Sep 27, 2018 at 3:56 AM Louis Zipes  wrote:

> Problem just re-occurred and so I was able to at least get a JSTACK  (I
> assume it was Tomcat since it was the Java using the most memory on the
> machine).  Here is the reoccurring message.  I get more hits on but haven't
> dug through all of the Google hits yet (due to multi-tasking) so apologies
> up front if there is a simple answer to this.
>
> "Event_Manager_1413" daemon prio=6 tid=0x24856000 nid=0x40c4
> waiting on condition [0x42dae000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0005ab45f7b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
> Source)
> at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
>
Do you lock from memory or database?  Make sure locking sequence is
consistent all across the application.  Meaning if you have

Lock A
Lock B
Lock C

You dont want another logic that locks something in another order.  E.g.

Lock B
Lock A
Lock C

Because that will cause deadlocks.



>Locked ownable synchronizers:
> - None
>
> > Any comments/suggestions are appreciated!
>
> Your most likely problem is database connection pool mismanagement:
> connections aren't properly released and the pool empties. All threads
> are left waiting on available database connections which will never be
> replenished.
>
> I'm using the ojdbc6.jar if that is what you are referring to or is there
> a better setting somewhere.
>

What db connection pool are you using? Are you letting tomcat manage it via
Context.xml and get connection via JNDI?  Or are you managing inside the
application?


>
> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Wednesday, September 26, 2018 3:46 PM
> To: users@tomcat.apache.org
> Subject: Re: Application hanging on Tomcat 7.0.54
>
> - - - external message, proceed with caution - - -
>
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Louis,
>
> On 9/26/18 14:42, Louis Zipes wrote:
> > Hi all, Tomcat 7.0.54 running on Windows 2012
> >
> > We are running a third party application on Tomcat and today we
> > have intermittently run in issues where the application stops
> > working.  The big changes in our system is that we have added more
> > end users and we are at year end so of course everyone is hitting
> > the system hard. Even if we force a log out of all users and stop
> > all background jobs then the application doesn't recover.
> >
> > We see no active sessions on the database (our application is
> > connecting to an Oracle database) and I see no clear error messages
> > in either our third party application logs or the Tomcat logs (ex.
> > OutofMemory).  When we go to the Windows Task Manager we did not
> > see the machine's Memory max'd out but admittedly I didn't look at
> > the Java session to see if was reaching its Heap Max.  The only
> > thing that we noticed was that TCP connections went down right
> > after the restart.  I did open up Jconsole under Java and I did
> > force a garbage collection but that didn't seem to help.
> >
> > We do have an Oracle Grid Control and we did get an alert in
> > regards to Metric: [HTTP Transaction] Perceived Time per Page going
> > past thresholds but not sure if that was just an old alert with and
> > old range that was set up a long time ago or is a really valid
> > clue.Since this is PRD we had to get it back up and running so
> > all I did was increase the Tomcat Xmx Heap size and restarted.  I'm
> > not really confident that is the solution since as mentioned you
> > tend to see a clear out of memory error if it was too small.
> >
> > So a few questions:
> >
> >
> > 1) Does this sound like a known issue with this earlier version
> > of Tomcat?
>
> No.
>
> > 2) Should I turn up any logging on Tomcat and if so which
> > ones?
>
> Not yet.
>
> > 3) We didn't do a JSTACK dump while it was happening.  Would
> > that have been useful?
>
> Absolutely.
>
> > 4) Do we need to play around with MaxThreads and/or
> > MaxConnectio

Re: Application hanging on Tomcat 7.0.54

2018-09-26 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/26/18 15:56, Louis Zipes wrote:
> Problem just re-occurred and so I was able to at least get a JSTACK
> (I assume it was Tomcat since it was the Java using the most memory
> on the machine).  Here is the reoccurring message.  I get more hits
> on but haven't dug through all of the Google hits yet (due to
> multi-tasking) so apologies up front if there is a simple answer to
> this.
> 
> "Event_Manager_1413" daemon prio=6 tid=0x24856000
> nid=0x40c4 waiting on condition [0x42dae000] 
> java.lang.Thread.State: TIMED_WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x0005ab45f7b8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
> 
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
awaitNanos(Unknown
> Source) at java.util.concurrent.LinkedBlockingQueue.poll(Unknown
> Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
> Source) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) at java.lang.Thread.run(Unknown Source)

This thread is waiting for a task, and is essentially idle. You will
have many of these on a non-busy system.

What are the other threads doing?

> Locked ownable synchronizers: - None
> 
>> Any comments/suggestions are appreciated!
> 
> Your most likely problem is database connection pool
> mismanagement: connections aren't properly released and the pool
> empties. All threads are left waiting on available database
> connections which will never be replenished.
> 
> I'm using the ojdbc6.jar if that is what you are referring to or is
> there a better setting somewhere.

ODBC? What is your database?

- -chris


> -Original Message- From: Christopher Schultz
> [mailto:ch...@christopherschultz.net] Sent: Wednesday, September
> 26, 2018 3:46 PM To: users@tomcat.apache.org Subject: Re:
> Application hanging on Tomcat 7.0.54
> 
> - - - external message, proceed with caution - - -
> 
> 
> Louis,
> 
> On 9/26/18 14:42, Louis Zipes wrote:
>> Hi all, Tomcat 7.0.54 running on Windows 2012
> 
>> We are running a third party application on Tomcat and today we 
>> have intermittently run in issues where the application stops 
>> working.  The big changes in our system is that we have added
>> more end users and we are at year end so of course everyone is
>> hitting the system hard. Even if we force a log out of all users
>> and stop all background jobs then the application doesn't
>> recover.
> 
>> We see no active sessions on the database (our application is 
>> connecting to an Oracle database) and I see no clear error
>> messages in either our third party application logs or the Tomcat
>> logs (ex. OutofMemory).  When we go to the Windows Task Manager
>> we did not see the machine's Memory max'd out but admittedly I
>> didn't look at the Java session to see if was reaching its Heap
>> Max.  The only thing that we noticed was that TCP connections
>> went down right after the restart.  I did open up Jconsole under
>> Java and I did force a garbage collection but that didn't seem to
>> help.
> 
>> We do have an Oracle Grid Control and we did get an alert in 
>> regards to Metric: [HTTP Transaction] Perceived Time per Page
>> going past thresholds but not sure if that was just an old alert
>> with and old range that was set up a long time ago or is a really
>> valid clue.Since this is PRD we had to get it back up and
>> running so all I did was increase the Tomcat Xmx Heap size and
>> restarted.  I'm not really confident that is the solution since
>> as mentioned you tend to see a clear out of memory error if it
>> was too small.
> 
>> So a few questions:
> 
> 
>> 1) Does this sound like a known issue with this earlier
>> version of Tomcat?
> 
> No.
> 
>> 2) Should I turn up any logging on Tomcat and if so which 
>> ones?
> 
> Not yet.
> 
>> 3) We didn't do a JSTACK dump while it was happening.  Would 
>> that have been useful?
> 
> Absolutely.
> 
>> 4) Do we need to play around with MaxThreads and/or 
>> MaxConnections.  We do have maxThreads in our server.mxl but in
>> DEV when we turned it down to a value = 5  hoping to overwhelm
>> it nothing bad happened.
> 
> Don't change anything, yet.
> 
>> Once again, we are limited to what we could do and collect since
>> it was PRD and we needed to resta

RE: Application hanging on Tomcat 7.0.54

2018-09-26 Thread Louis Zipes 
Problem just re-occurred and so I was able to at least get a JSTACK  (I assume 
it was Tomcat since it was the Java using the most memory on the machine).  
Here is the reoccurring message.  I get more hits on but haven't dug through 
all of the Google hits yet (due to multi-tasking) so apologies up front if 
there is a simple answer to this.

"Event_Manager_1413" daemon prio=6 tid=0x24856000 nid=0x40c4 waiting on 
condition [0x42dae000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0005ab45f7b8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown
 Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
- None

> Any comments/suggestions are appreciated!

Your most likely problem is database connection pool mismanagement:
connections aren't properly released and the pool empties. All threads
are left waiting on available database connections which will never be
replenished.

I'm using the ojdbc6.jar if that is what you are referring to or is there a 
better setting somewhere.

-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Wednesday, September 26, 2018 3:46 PM
To: users@tomcat.apache.org
Subject: Re: Application hanging on Tomcat 7.0.54

- - - external message, proceed with caution - - -


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/26/18 14:42, Louis Zipes wrote:
> Hi all, Tomcat 7.0.54 running on Windows 2012
>
> We are running a third party application on Tomcat and today we
> have intermittently run in issues where the application stops
> working.  The big changes in our system is that we have added more
> end users and we are at year end so of course everyone is hitting
> the system hard. Even if we force a log out of all users and stop
> all background jobs then the application doesn't recover.
>
> We see no active sessions on the database (our application is
> connecting to an Oracle database) and I see no clear error messages
> in either our third party application logs or the Tomcat logs (ex.
> OutofMemory).  When we go to the Windows Task Manager we did not
> see the machine's Memory max'd out but admittedly I didn't look at
> the Java session to see if was reaching its Heap Max.  The only
> thing that we noticed was that TCP connections went down right
> after the restart.  I did open up Jconsole under Java and I did
> force a garbage collection but that didn't seem to help.
>
> We do have an Oracle Grid Control and we did get an alert in
> regards to Metric: [HTTP Transaction] Perceived Time per Page going
> past thresholds but not sure if that was just an old alert with and
> old range that was set up a long time ago or is a really valid
> clue.Since this is PRD we had to get it back up and running so
> all I did was increase the Tomcat Xmx Heap size and restarted.  I'm
> not really confident that is the solution since as mentioned you
> tend to see a clear out of memory error if it was too small.
>
> So a few questions:
>
>
> 1) Does this sound like a known issue with this earlier version
> of Tomcat?

No.

> 2) Should I turn up any logging on Tomcat and if so which
> ones?

Not yet.

> 3) We didn't do a JSTACK dump while it was happening.  Would
> that have been useful?

Absolutely.

> 4) Do we need to play around with MaxThreads and/or
> MaxConnections.  We do have maxThreads in our server.mxl but in DEV
> when we turned it down to a value = 5  hoping to overwhelm it
> nothing bad happened.

Don't change anything, yet.

> Once again, we are limited to what we could do and collect since it
> was PRD and we needed to restart it.  We restarted the Tomcat
> service and everything is processing fine for right now.  I will
> note that that we did have that bad Windows patch that prevented it
> from stopping and starting cleanly
> (https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdow
n)
> but we have taken the break fix patch and the daily restarts seem
> to be fine since then.
>
> Any comments/suggestions are appreciated!

Your most likely problem is database connection pool mismanagement:
connections aren't properly released and the pool empties. All threads
are left waiting on available database connections which will never be
replenished.

- -

Re: Application hanging on Tomcat 7.0.54

2018-09-26 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Louis,

On 9/26/18 14:42, Louis Zipes wrote:
> Hi all, Tomcat 7.0.54 running on Windows 2012
> 
> We are running a third party application on Tomcat and today we
> have intermittently run in issues where the application stops
> working.  The big changes in our system is that we have added more
> end users and we are at year end so of course everyone is hitting
> the system hard. Even if we force a log out of all users and stop
> all background jobs then the application doesn't recover.
> 
> We see no active sessions on the database (our application is
> connecting to an Oracle database) and I see no clear error messages
> in either our third party application logs or the Tomcat logs (ex.
> OutofMemory).  When we go to the Windows Task Manager we did not
> see the machine's Memory max'd out but admittedly I didn't look at
> the Java session to see if was reaching its Heap Max.  The only
> thing that we noticed was that TCP connections went down right
> after the restart.  I did open up Jconsole under Java and I did
> force a garbage collection but that didn't seem to help.
> 
> We do have an Oracle Grid Control and we did get an alert in
> regards to Metric: [HTTP Transaction] Perceived Time per Page going
> past thresholds but not sure if that was just an old alert with and
> old range that was set up a long time ago or is a really valid
> clue.Since this is PRD we had to get it back up and running so
> all I did was increase the Tomcat Xmx Heap size and restarted.  I'm
> not really confident that is the solution since as mentioned you
> tend to see a clear out of memory error if it was too small.
> 
> So a few questions:
> 
> 
> 1) Does this sound like a known issue with this earlier version
> of Tomcat?

No.

> 2) Should I turn up any logging on Tomcat and if so which
> ones?

Not yet.

> 3) We didn't do a JSTACK dump while it was happening.  Would
> that have been useful?

Absolutely.

> 4) Do we need to play around with MaxThreads and/or
> MaxConnections.  We do have maxThreads in our server.mxl but in DEV
> when we turned it down to a value = 5  hoping to overwhelm it
> nothing bad happened.

Don't change anything, yet.

> Once again, we are limited to what we could do and collect since it
> was PRD and we needed to restart it.  We restarted the Tomcat
> service and everything is processing fine for right now.  I will
> note that that we did have that bad Windows patch that prevented it
> from stopping and starting cleanly
> (https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdow
n)
> but we have taken the break fix patch and the daily restarts seem
> to be fine since then.
> 
> Any comments/suggestions are appreciated!

Your most likely problem is database connection pool mismanagement:
connections aren't properly released and the pool empties. All threads
are left waiting on available database connections which will never be
replenished.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlur4fgACgkQHPApP6U8
pFhq/RAAwZixHqbHxYdX3VCrTfvz0tnOmu7W4sbeFqhExV+M6NVL2LK1RO26eTq5
OJB2o/RheCeajWHxqiJQY4ERMTOyyqZYCsRG9L901heW2UAW122zeX7hhXDB1IMo
qIBVYQalg1j5e2Lw9MqT3ISj6U/GNA6VlirTAHGtuEBBpKXXyb6KtmOgpjHjjXS7
mTYni2iTHO/NaGrS519alFPMBnF4Wq5NzRcLewNMqj9Nbx2uu3Suu95DhJ+WIIep
DOmyy4UGHxGx2QqNUnVWMHApGGjFD4pTWIBwzcbbsL56kZDxRGsF0SsB0VR/jSMY
HdI71RgjpQFSyar1rcCTPCRP97KnUC+oaKhn+i2jBMSRxs94GEOqk4LuKNo39HKP
tXEMkYl0o/CR9QaFDjncy9M2M3/o50ooBvYTOu0SjmyZO+ab9tJpaXbnlf2ChxJI
AWbhBGmwJ6kF5FvVbmzujV7EEF2YCRMBpWo2zjNd6zWvX9OWEOXrOaHuX7iBPCga
YqEMQWyS7XWiBG7AI8+ka5x4s/oxWsbn/6pCdDXhfxl5p5jv7ajm7LbLXGut1N6a
uV5hmLLgZywF68AYe6X3GWv6mygXMBYABZxEA6klWE7HpIzvLxmxu0+vFlYR8qsb
FMAacJ/FYJ9nGRuMqG+V2Edr5U//JvrWqy4raPIvwXGtX+FsqEc=
=Zavb
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Application hanging on Tomcat 7.0.54

2018-09-26 Thread Louis Zipes 
Hi all,
Tomcat 7.0.54 running on Windows 2012

We are running a third party application on Tomcat and today we have 
intermittently run in issues where the application stops working.  The big 
changes in our system is that we have added more end users and we are at year 
end so of course everyone is hitting the system hard. Even if we force a log 
out of all users and stop all background jobs then the application doesn't 
recover.

We see no active sessions on the database (our application is connecting to an 
Oracle database) and I see no clear error messages in either our third party 
application logs or the Tomcat logs (ex. OutofMemory).  When we go to the 
Windows Task Manager we did not see the machine's Memory max'd out but 
admittedly I didn't look at the Java session to see if was reaching its Heap 
Max.  The only thing that we noticed was that TCP connections went down right 
after the restart.  I did open up Jconsole under Java and I did force a garbage 
collection but that didn't seem to help.

We do have an Oracle Grid Control and we did get an alert in regards to Metric: 
[HTTP Transaction] Perceived Time per Page going past thresholds but not sure 
if that was just an old alert with and old range that was set up a long time 
ago or is a really valid clue.Since this is PRD we had to get it back up 
and running so all I did was increase the Tomcat Xmx Heap size and restarted.  
I'm not really confident that is the solution since as mentioned you tend to 
see a clear out of memory error if it was too small.

So a few questions:


1) Does this sound like a known issue with this earlier version of Tomcat?

2) Should I turn up any logging on Tomcat and if so which ones?

3) We didn't do a JSTACK dump while it was happening.  Would that have been 
useful?

4) Do we need to play around with MaxThreads and/or MaxConnections.  We do 
have maxThreads in our server.mxl but in DEV when we turned it down to a value 
= 5  hoping to overwhelm it nothing bad happened.

Once again, we are limited to what we could do and collect since it was PRD and 
we needed to restart it.  We restarted the Tomcat service and everything is 
processing fine for right now.  I will note that that we did have that bad 
Windows patch that prevented it from stopping and starting cleanly  
(https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdown) but we 
have taken the break fix patch and the daily restarts seem to be fine since 
then.

Any comments/suggestions are appreciated!

Thanks, Louis




LOUIS ZIPES
SOFTWARE DEVELOPER ANALYST IV
O: 781-418-3257
louis.zi...@keurig.com
Keurig Dr Pepper
Visit us at www.KeurigDrPepper.com


---
CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and may 
contain information that is confidential, proprietary or exempt from 
disclosure. If you are not the intended recipient, please contact the sender 
immediately. Unauthorized use or distribution is prohibited and may be unlawful.