Christopher
________________________________________
From: Christopher Schultz [ch...@christopherschultz.net]
Sent: Friday, February 28, 2014 11:38 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 2/27/14, 5:40 PM, Konstantin Kolinko wrote:
> 2014-02-28 2:06 GMT+04:00 Isaac Gonzalez
> <igonza...@autoreturn.com>:
>> Hi Christopher(and Konstantin), attached is a couple of thread
>> dumps of when we experienced the issue again today. I also
>> noticed we get this message right before the problem occurs: Feb
>> 27, 2014 12:47:15 PM
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>> create new native thread) executing
>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>> terminating thread
>
> That explains why a connection cannot be accepted.
>
> I wonder are you hitting an "ulimit" limit, or there is just not
> enough free memory to allocate stack area for a new thread (which
> size is set by -Xss option to java executable).
>
> Your thread dumps contain 149 threads each.
While it does explain why one (Tomcat) server would become
unresponsive, it doesn't really explain why the entire cluster would
become unresponsive.
Issac, are you using sticky-sessions or anything like that, or does
your load-balancing mod_jk configuration choose arbitrarily between a
backend server? You initially gave an abridged configuration, so I
can't tell.
As you indicate below, I am not clustering. There is only one
backend tomcat.
>> After the "split", did both Tomcats appear to lock-up
>> simultaneously, or did only one of them have a problem and the
>> other one stayed up?
>
> Isaac: They all appear to lock-up simultaneously, if users try to
> access that JK mount point.
>
> [...]
>
> Isaac: I am not load-balancing the tomcat servers...I only have
> one...I do "load balance" the apache front end servers via dns
> round-robin....
Oh, you only have a single back-end server? Well, then that why they
"all" go down at once, so you seem to have found your problem: the
server itself is going down because you don't have enough resources to
keep it up.
Indeed I have! Seems like I underallocated server memory...the machine
had only 8 Gigs with 7 tomcat instances running that all had up to a maximum of
2 gigs of maximum memory heap size, plus OS stuff running, RabbitMQ, and other
things. I am wondering though if something else could be the underlying root
cause of this issue, or was I simply under allocating memory..such as
connections not being closed, either by the client mod_jk connector, or the db
connector...We'll see in the next few days I guess....
thanks again Chris, you and Konstantin pointed me to the issue...
-Isaac
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTEOWeAAoJEBzwKT+lPKRYUxYQAItxQZTOWjubWA32fDwGhcGy
Y4cuBDU7OZiw/I4oQTCEY0Uk62eKf6g0dXQL4VqM8j4jz2VLhP10uPDA3c/+CBhU
kpVJg2ifLD0YJmUqnxuTHYIdwwvPrPa/PNCAWYJCcDXE1DVEz8HLAoYlQY5oJ5Cf
P+wtYTgWqJyzddtB2sjB+YQcVj+83aWkfKuipednSdm0utCPP5PQzPiF7agoP9qt
vDB0preG0GFQXTShYqMRKeEt3hu+BdLXugp7kJA5KDMEcSbWyPzefxWl0CtKhcJB
d/ntEtoYbR0gWGO4Qajio6NVmw9TWzBf4spbg8scBz8ijE314VNsw6mdT9F55TZZ
43iYSnDAK1dNfs7guqAAk7z5Gf+fChy28zFmOm0lSzs1/o5HHvJFqKse94hzjJW6
R4uCUVktbvoJPfot6zoG3ofsYm+PVcibPOj4Xh0m7nBPKvqTZ5BVyeLBRR/E+KRy
O4jJ0DshRnhy3qL9l5uO6h7miIb+GMwjpc3A6lbbITHVDKspaq8kll+m6sAn1ppV
Z8PnNysSMTGHY6azjMisZlp4b/i9r+Nc+HabtbjRrt1StfuWrHTeIwQ46n7XcMrr
biATXUpo94kM2eGJecP0jtyBrgYwkaz22NtbW3i2l47XQKa/dhhP6IzlgzK8Fmki
eKiBf5+iNyhc9dB3adQ2
=Uchz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
