Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
All,
One of dev instances this morning is screaming because I have hundreds
of active sessions. I checked, and it looks like the background
processor thread for the context is not running, therefore no sessions
are expiring.
Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
Here are the still-running threads:
catalina-exec-946
Attach Listener
MySQL Statement Cancellation Timer
catalina-exec-338
catalina-exec-337
catalina-exec-5
catalina-exec-4
catalina-exec-3
http-nio-127.0.0.1-8217-Acceptor-0
http-nio-127.0.0.1-8217-ClientPoller-1
http-nio-127.0.0.1-8217-ClientPoller-0
ajp-bio-8215-AsyncTimeout
ajp-bio-8215-Acceptor-0
Memcached IO over {MemcachedConnection to localhost/127.0.0.1:11211}
Abandoned connection cleanup thread
Thread-3
NioBlockingSelector.BlockPoller-1
GC Daemon
Service Thread
C2 CompilerThread1
C2 CompilerThread0
Signal Dispatcher
Finalizer
Reference Handler
main
VM Thread
GC task thread#0 (ParallelGC)
GC task thread#1 (ParallelGC)
VM Periodic Task Thread
I tried invoking backgroundProcess on my Manager bean via JMX but it
didn't seem to actually clean anything up. Is there any hope of
recovery without bouncing the web application?
Also, it would be nice to get a notification if the thread is dying
due to some exception. I searched-though catalina.out and found
nothing relevant. Any suggestions?
Thanks,
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkIp2AAoJEBzwKT+lPKRYxOUQAJVMmWhZEimBNZiSwcgP2lNA
PET0guRv9x8b9A75UcLjPU1E69OCZ1AxpsT0W8eKLhzI8ehmv/COb6JDIIW79cuL
ksOSXeFwsPQfv/f0rB2byDmIrMVBg+053PLU3v3nWWp0VnLsdrr2oGfdvyjUmdtE
WuqhmGtVqzpOuZZixq3xi2sYWwmwATA1DdsTWllat6mdbgC2289S9lv8w7a/SXO0
f3aYW+AFxnX3nPN4ldbpbrUVOUfxKHXlMdjf4Rg1Z/aTIWNrInlJSAgiUjD6hY5P
rm/F83/zbsCfm7mRDPVjw425SQ3tpYX7K6p7bSoPkCwip9brURUyuIz4aSkLv27e
fO+rgm8P3/jdUmyfh6UT+mB5li2uBS6vPHu87lrIH1c5o65x3/+S0FgNA5Q5tJNC
GYti74FNwwga+g7cOMsItnZrOExm7xnpDifctGkMV3V7S2mG66iCgDVD7vk4B8F/
tBOSR+i1OP6RMwGLbnr2OW/QFKTnksN8jBlAFF4osXIPf1QrcUwLJA7o2z3ynqlR
ykHaolvZ2HU25gieAUfKy7ZRj7dJ61aAP3Ak7Rpr9Krno0GxlKTS53JeYg1mNGsp
4goBt8jdvyMMMA8ZcV7HfAUfPAbJn/2ehmkCLBqzMu0Un3OeRADvZZrI308wTPSG
q9r0JhsDD/BZUhE47Iwj
=Az1U
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
2014-06-05 19:19 GMT+04:00 Christopher Schultz ch...@christopherschultz.net:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
All,
One of dev instances this morning is screaming because I have hundreds
of active sessions. I checked, and it looks like the background
processor thread for the context is not running, therefore no sessions
are expiring.
Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
Here are the still-running threads:
catalina-exec-946
Attach Listener
MySQL Statement Cancellation Timer
catalina-exec-338
catalina-exec-337
catalina-exec-5
catalina-exec-4
catalina-exec-3
http-nio-127.0.0.1-8217-Acceptor-0
http-nio-127.0.0.1-8217-ClientPoller-1
http-nio-127.0.0.1-8217-ClientPoller-0
ajp-bio-8215-AsyncTimeout
ajp-bio-8215-Acceptor-0
Memcached IO over {MemcachedConnection to localhost/127.0.0.1:11211}
Abandoned connection cleanup thread
Thread-3
NioBlockingSelector.BlockPoller-1
GC Daemon
Service Thread
C2 CompilerThread1
C2 CompilerThread0
Signal Dispatcher
Finalizer
Reference Handler
main
VM Thread
GC task thread#0 (ParallelGC)
GC task thread#1 (ParallelGC)
VM Periodic Task Thread
I tried invoking backgroundProcess on my Manager bean via JMX but it
didn't seem to actually clean anything up. Is there any hope of
recovery without bouncing the web application?
Also, it would be nice to get a notification if the thread is dying
due to some exception. I searched-though catalina.out and found
nothing relevant. Any suggestions?
In general, dying threads are handled via
Thread.setUncaughtExceptionHandler(...)
and
ThreadGroup.uncaughtException(...)
The default behaviour in JRE it so log to System.err. So catalina.out
is the place where I expect it to be written.
(It is not nice as it bypasses logging framework, but that is how it
is currently).
A unusual culprit is OutOfMemoryError.
I tried invoking backgroundProcess on my Manager bean via JMX but it
didn't seem to actually clean anything up. Is there any hope of
recovery without bouncing the web application?
The background thread is started via threadStart() which is called
from ContainerBase.startInternal() / StandardContext.startInternal().
So I see no hope of recovering unless the container that owns the
thread is stopped/started. I think that by default the thread is
started by Engine, so Engine needs to be bounced here.
BTW, javadoc for ContainerBase.backgroundProcess() says This method
will be invoked inside the classloading context of this container. I
think that is not true when it is being invoked via JMX.
Best regards,
Konstantin Kolinko
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Konstantin,
On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
2014-06-05 19:19 GMT+04:00 Christopher Schultz
ch...@christopherschultz.net:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256
All,
One of dev instances this morning is screaming because I have
hundreds of active sessions. I checked, and it looks like the
background processor thread for the context is not running,
therefore no sessions are expiring.
Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
Here are the still-running threads:
catalina-exec-946 Attach Listener MySQL Statement
Cancellation Timer catalina-exec-338 catalina-exec-337
catalina-exec-5 catalina-exec-4 catalina-exec-3
http-nio-127.0.0.1-8217-Acceptor-0
http-nio-127.0.0.1-8217-ClientPoller-1
http-nio-127.0.0.1-8217-ClientPoller-0
ajp-bio-8215-AsyncTimeout ajp-bio-8215-Acceptor-0 Memcached
IO over {MemcachedConnection to localhost/127.0.0.1:11211}
Abandoned connection cleanup thread Thread-3
NioBlockingSelector.BlockPoller-1 GC Daemon Service Thread
C2 CompilerThread1 C2 CompilerThread0 Signal Dispatcher
Finalizer Reference Handler main VM Thread GC task
thread#0 (ParallelGC) GC task thread#1 (ParallelGC) VM
Periodic Task Thread
I tried invoking backgroundProcess on my Manager bean via JMX
but it didn't seem to actually clean anything up. Is there any
hope of recovery without bouncing the web application?
Also, it would be nice to get a notification if the thread is
dying due to some exception. I searched-though catalina.out and
found nothing relevant. Any suggestions?
In general, dying threads are handled via
Thread.setUncaughtExceptionHandler(...) and
ThreadGroup.uncaughtException(...)
The default behaviour in JRE it so log to System.err. So
catalina.out is the place where I expect it to be written.
That's what I would have thought, too. I didn't see anything,
unfortunately.
(It is not nice as it bypasses logging framework, but that is how
it is currently).
A unusual culprit is OutOfMemoryError.
I think you mean a usual culprit and I agree: this has happened in
the past where the background processor thread dies due to OOME.
I tried invoking backgroundProcess on my Manager bean via JMX
but it didn't seem to actually clean anything up. Is there any
hope of recovery without bouncing the web application?
The background thread is started via threadStart() which is called
from ContainerBase.startInternal() /
StandardContext.startInternal().
So I see no hope of recovering unless the container that owns the
thread is stopped/started. I think that by default the thread is
started by Engine, so Engine needs to be bounced here.
The JMX method backgroundProcess should call
ManagerBase.backgroundProcess which should go ahead and expire the
sessions. I don't see a reason why it wouldn't work. Unfortunately,
Tomcat's Manager wasn't in debug-log mode or I would have been able to
see what happened when I invoked backgroundProcess.
BTW, javadoc for ContainerBase.backgroundProcess() says This
method will be invoked inside the classloading context of this
container. I think that is not true when it is being invoked via
JMX.
This is ManagerBase.backgroundProcess, which does not have the same
documentation. While ContainerBase.backgroundProcess would usually
invoke ManagerBase.backgroundProcess, using JMX goes directly to
ManagerBase.
I'm still curious why invoking ManagerBase.backgroundProcess via JMX
had no effect. I'm not sure what the count and
processExpiresFrequency values were at the time, so my calls could
have been rejected due to background-processing-throttling.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkL/LAAoJEBzwKT+lPKRYSn0P/Rs3TRlD0W7YTGuYuxIbeIoc
9CODVusNETHiUU0zrY4cJwZFXhBkfJ/ev8h9+ErEIsB854KJQNJ67F1o2iNmr3Nm
qyWHldZjaYSdns3zHKhEkhyhEutIFfM8KHMDE0azc7GtKEkKNs8en8MVuBPITTWY
JYtjh8fTaxm+rfjuLp/vRRB+Oj9GYQncuLihCvWPUzWMr+sRM0/baRaWAKP3d4M6
4EC7r2Pi3HuqMbUEYt5Vh3O3x/5BOdLsU/tu616L9fGC6aiNF2w3KUIw5pECdPS7
9vIIsXhsGQZ2+Y1TpnN/lPojItTtak6hdweMNGAKberxe816YjEdTrwkeue6MiLD
J9U8ArQLI58CJCcfM2My+dyaPea53VsBKPkYbcDpO+YM1sqko9JkJTsv1fpzeKVn
NtiCQNRq7UoaIJ7u3seTXn7piJ8ahrD+FpkLXxOiLwQ7lLNKtR4b+14smwKDrI/h
gUobA92SQNgluLetnpxAvcve1sjySSVs0RN1rpw1GPTDrh/cOWIYkdiiUzZn58lf
Ro1FyotAy9JHVDCFJpcwwIzaVty/6cJpJHI+2Ukl4aMHJRjFClioDb6Gx/wBnk7I
SNiCj1R3EKxbvrTSPruHYTSdREtQHWQYpDj6odfuNtupY65K86iS4fkNTv08CYlt
y64Igz0yrJPwqCdgAPL4
=kOau
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
2014-06-05 23:06 GMT+04:00 Christopher Schultz ch...@christopherschultz.net:
On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
2014-06-05 19:19 GMT+04:00 Christopher Schultz
ch...@christopherschultz.net:
I tried invoking backgroundProcess on my Manager bean via JMX
but it didn't seem to actually clean anything up. Is there any
hope of recovery without bouncing the web application?
Also, it would be nice to get a notification if the thread is
dying due to some exception. I searched-though catalina.out and
found nothing relevant. Any suggestions?
In general, dying threads are handled via
Thread.setUncaughtExceptionHandler(...) and
ThreadGroup.uncaughtException(...)
The default behaviour in JRE it so log to System.err. So
catalina.out is the place where I expect it to be written.
That's what I would have thought, too. I didn't see anything,
unfortunately.
(It is not nice as it bypasses logging framework, but that is how
it is currently).
A unusual culprit is OutOfMemoryError.
I think you mean a usual culprit and I agree: this has happened in
the past where the background processor thread dies due to OOME.
Yes, a typo. I meant a usual one.
I tried invoking backgroundProcess on my Manager bean via JMX
but it didn't seem to actually clean anything up. Is there any
hope of recovery without bouncing the web application?
The background thread is started via threadStart() which is called
from ContainerBase.startInternal() /
StandardContext.startInternal().
So I see no hope of recovering unless the container that owns the
thread is stopped/started. I think that by default the thread is
started by Engine, so Engine needs to be bounced here.
The JMX method backgroundProcess should call
ManagerBase.backgroundProcess which should go ahead and expire the
sessions. I don't see a reason why it wouldn't work. Unfortunately,
Tomcat's Manager wasn't in debug-log mode or I would have been able to
see what happened when I invoked backgroundProcess.
1. If you have Manager webapp installed, it has API to expire sessions
(on the list page in HTML gui)
2. I think logging level can be manipulated via JMX.
BTW, javadoc for ContainerBase.backgroundProcess() says This
method will be invoked inside the classloading context of this
container. I think that is not true when it is being invoked via
JMX.
This is ManagerBase.backgroundProcess, which does not have the same
documentation. While ContainerBase.backgroundProcess would usually
invoke ManagerBase.backgroundProcess, using JMX goes directly to
ManagerBase.
OK.
I'm still curious why invoking ManagerBase.backgroundProcess via JMX
had no effect. I'm not sure what the count and
processExpiresFrequency values were at the time, so my calls could
have been rejected due to background-processing-throttling.
//ManagerBase.java
@Override
public void backgroundProcess() {
count = (count + 1) % processExpiresFrequency;
if (count == 0)
processExpires();
}
processExpiresFrequency should be readable via JMX.
Default is '6', so you have to call the method 6 times in a row.
Best regards,
Konstantin Kolinko
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess via JMX has no effect
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Konstantin,
On 6/5/14, 4:27 PM, Konstantin Kolinko wrote:
2014-06-05 23:06 GMT+04:00 Christopher Schultz
ch...@christopherschultz.net:
On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
2014-06-05 19:19 GMT+04:00 Christopher Schultz
ch...@christopherschultz.net:
I tried invoking backgroundProcess on my Manager bean via
JMX but it didn't seem to actually clean anything up. Is
there any hope of recovery without bouncing the web
application?
Also, it would be nice to get a notification if the thread
is dying due to some exception. I searched-though
catalina.out and found nothing relevant. Any suggestions?
In general, dying threads are handled via
Thread.setUncaughtExceptionHandler(...) and
ThreadGroup.uncaughtException(...)
The default behaviour in JRE it so log to System.err. So
catalina.out is the place where I expect it to be written.
That's what I would have thought, too. I didn't see anything,
unfortunately.
(It is not nice as it bypasses logging framework, but that is
how it is currently).
A unusual culprit is OutOfMemoryError.
I think you mean a usual culprit and I agree: this has happened
in the past where the background processor thread dies due to
OOME.
Yes, a typo. I meant a usual one.
I tried invoking backgroundProcess on my Manager bean via
JMX but it didn't seem to actually clean anything up. Is
there any hope of recovery without bouncing the web
application?
The background thread is started via threadStart() which is
called from ContainerBase.startInternal() /
StandardContext.startInternal().
So I see no hope of recovering unless the container that owns
the thread is stopped/started. I think that by default the
thread is started by Engine, so Engine needs to be bounced
here.
The JMX method backgroundProcess should call
ManagerBase.backgroundProcess which should go ahead and expire
the sessions. I don't see a reason why it wouldn't work.
Unfortunately, Tomcat's Manager wasn't in debug-log mode or I
would have been able to see what happened when I invoked
backgroundProcess.
1. If you have Manager webapp installed, it has API to expire
sessions (on the list page in HTML gui)
2. I think logging level can be manipulated via JMX.
I hadn't thought of that. Next time. ;)
BTW, javadoc for ContainerBase.backgroundProcess() says This
method will be invoked inside the classloading context of this
container. I think that is not true when it is being invoked
via JMX.
This is ManagerBase.backgroundProcess, which does not have the
same documentation. While ContainerBase.backgroundProcess would
usually invoke ManagerBase.backgroundProcess, using JMX goes
directly to ManagerBase.
OK.
I'm still curious why invoking ManagerBase.backgroundProcess via
JMX had no effect. I'm not sure what the count and
processExpiresFrequency values were at the time, so my calls
could have been rejected due to
background-processing-throttling.
//ManagerBase.java @Override public void backgroundProcess() {
count = (count + 1) % processExpiresFrequency; if (count == 0)
processExpires(); }
processExpiresFrequency should be readable via JMX.
Default is '6', so you have to call the method 6 times in a row.
Aah. I gave up after 3. Next time.
Thanks for the info!
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkOEWAAoJEBzwKT+lPKRY9hwP/Azf+FdBNpEl/V29SjuVfAb/
GTkEwzcVyxFh0uzr1wydcjdw8vE/SeVYcL0++dWmjzV6qGPeVLY0/wbrASBfJckc
Cka8gbLpSnIaVkrGqEAqWvfisSXYY2UGpCh2raOzgRW8/kqKcwriNrxRyZ7vPXXo
v4Ql+J4mjBC+XN4RRqHnz//l0bRBuwYGUgfeew7Tv5hO32PfGV2GS6FtgsbceIP7
GDRpqFoBo9xQtmgYHvHIVYrRD7zSBFcf72YZdY23PIlQum6TdvxWjkt0KipNrRE8
N7mDBprddiuRv1q6vNjWJbkz+y+YROd4PlHpK/sCxhdQMfvpl4SpNXp2Uo0b6cBB
KiFjJ6i9HUcEbaHrRZXCS+UoOxBOhQM/WBRtPXv/Vy+k7hIyjtfs1QCojrwnQQcl
EMff7iDdukk+m2FlnK3SdbxJALGtVh1Y+Q3eYa+X3IqBoz7/RQDt1tBUjpm4eKkF
G+cg7/+JubJJh+2TMOHzi7YFsBo978TMMO3rmu+Bl3E4YHayzt0RMfde2V0Dj4jD
WJCQOFz/JYHRjZoeY/lJSx0Y9kydfNeqF/aQ5hlF1+c2PKVYVkwp5NG5H48yXjpo
E3HcDV0nSV69VmbWQKiZpGn74B5J9tHTLQ7KYFJRH0BpyOk+QZc0Zzh2zTtAnbyq
tRA2omkbjAifP0PA0IMg
=jS15
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
