-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
> 2014-06-05 19:19 GMT+04:00 Christopher Schultz
> <ch...@christopherschultz.net>:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> All,
>>
>> One of dev instances this morning is screaming because I have
>> hundreds of active sessions. I checked, and it looks like the
>> background processor thread for the context is not running,
>> therefore no sessions are expiring.
>>
>> Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
>>
>> Here are the still-running threads:
>>
>> "catalina-exec-946" "Attach Listener" "MySQL Statement
>> Cancellation Timer" "catalina-exec-338" "catalina-exec-337"
>> "catalina-exec-5" "catalina-exec-4" "catalina-exec-3"
>> "http-nio-127.0.0.1-8217-Acceptor-0"
>> "http-nio-127.0.0.1-8217-ClientPoller-1"
>> "http-nio-127.0.0.1-8217-ClientPoller-0"
>> "ajp-bio-8215-AsyncTimeout" "ajp-bio-8215-Acceptor-0" "Memcached
>> IO over {MemcachedConnection to localhost/127.0.0.1:11211}"
>> "Abandoned connection cleanup thread" "Thread-3"
>> "NioBlockingSelector.BlockPoller-1" "GC Daemon" "Service Thread"
>> "C2 CompilerThread1" "C2 CompilerThread0" "Signal Dispatcher"
>> "Finalizer" "Reference Handler" "main" "VM Thread" "GC task
>> thread#0 (ParallelGC)" "GC task thread#1 (ParallelGC)" "VM
>> Periodic Task Thread"
>>
>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
>>
>> Also, it would be nice to get a notification if the thread is
>> dying due to some exception. I searched-though catalina.out and
>> found nothing relevant. Any suggestions?
>
> In general, dying threads are handled via
> Thread.setUncaughtExceptionHandler(...) and
> ThreadGroup.uncaughtException(...)
>
> The default behaviour in JRE it so log to System.err. So
> catalina.out is the place where I expect it to be written.
That's what I would have thought, too. I didn't see anything,
unfortunately.
> (It is not nice as it bypasses logging framework, but that is how
> it is currently).
>
> A unusual culprit is OutOfMemoryError.
I think you mean "a usual culprit" and I agree: this has happened in
the past where the background processor thread dies due to OOME.
>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
>
> The background thread is started via threadStart() which is called
> from ContainerBase.startInternal() /
> StandardContext.startInternal().
>
> So I see no hope of recovering unless the container that owns the
> thread is stopped/started. I think that by default the thread is
> started by Engine, so Engine needs to be bounced here.
The JMX method "backgroundProcess" should call
ManagerBase.backgroundProcess which should go ahead and expire the
sessions. I don't see a reason why it wouldn't work. Unfortunately,
Tomcat's Manager wasn't in debug-log mode or I would have been able to
see what happened when I invoked backgroundProcess.
> BTW, javadoc for ContainerBase.backgroundProcess() says "This
> method will be invoked inside the classloading context of this
> container." I think that is not true when it is being invoked via
> JMX.
This is ManagerBase.backgroundProcess, which does not have the same
documentation. While ContainerBase.backgroundProcess would usually
invoke ManagerBase.backgroundProcess, using JMX goes directly to
ManagerBase.
I'm still curious why invoking ManagerBase.backgroundProcess via JMX
had no effect. I'm not sure what the "count" and
"processExpiresFrequency" values were at the time, so my calls could
have been rejected due to background-processing-throttling.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkL/LAAoJEBzwKT+lPKRYSn0P/Rs3TRlD0W7YTGuYuxIbeIoc
9CODVusNETHiUU0zrY4cJwZFXhBkfJ/ev8h9+ErEIsB854KJQNJ67F1o2iNmr3Nm
qyWHldZjaYSdns3zHKhEkhyhEutIFfM8KHMDE0azc7GtKEkKNs8en8MVuBPITTWY
JYtjh8fTaxm+rfjuLp/vRRB+Oj9GYQncuLihCvWPUzWMr+sRM0/baRaWAKP3d4M6
4EC7r2Pi3HuqMbUEYt5Vh3O3x/5BOdLsU/tu616L9fGC6aiNF2w3KUIw5pECdPS7
9vIIsXhsGQZ2+Y1TpnN/lPojItTtak6hdweMNGAKberxe816YjEdTrwkeue6MiLD
J9U8ArQLI58CJCcfM2My+dyaPea53VsBKPkYbcDpO+YM1sqko9JkJTsv1fpzeKVn
NtiCQNRq7UoaIJ7u3seTXn7piJ8ahrD+FpkLXxOiLwQ7lLNKtR4b+14smwKDrI/h
gUobA92SQNgluLetnpxAvcve1sjySSVs0RN1rpw1GPTDrh/cOWIYkdiiUzZn58lf
Ro1FyotAy9JHVDCFJpcwwIzaVty/6cJpJHI+2Ukl4aMHJRjFClioDb6Gx/wBnk7I
SNiCj1R3EKxbvrTSPruHYTSdREtQHWQYpDj6odfuNtupY65K86iS4fkNTv08CYlt
y64Igz0yrJPwqCdgAPL4
=kOau
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
