-----Original Message----- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, March 07, 2014 8:18 AM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 3/6/14, 7:39 AM, Daniel Mikusa wrote: > On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez <igonza...@autoreturn.com> > wrote: > >> >> >> -----Original Message----- From: Daniel Mikusa >> [mailto:dmik...@gopivotal.com] Sent: Tuesday, March 04, 2014 >> 12:42 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk >> connections after server runs for a couple of days >> >> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <igonza...@autoreturn.com> >> wrote: >> >>> Dan, >>> >>> ________________________________________ From: Daniel Mikusa >>> [dmik...@gopivotal.com] Sent: Tuesday, March 04, 2014 6:20 AM >>> To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk >>> connections after server runs for a couple of days >>> >>> On Mar 4, 2014, at 6:32 AM, Rainer Jung <rainer.j...@kippdata.de> >>> wrote: >>> >>>> On 27.02.2014 23:06, Isaac Gonzalez wrote: >>>>> Hi Christopher(and Konstantin), attached is a couple of thread >>>>> dumps of when we experienced the issue again today. >>>>> I also noticed we get this message right before the problem >>>>> occurs: Feb 27, 2014 12:47:15 PM >>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable >>>>> run SEVERE: Caught exception (java.lang.OutOfMemoryError: >>>>> unable to create new native thread) executing >>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, >>>>> terminating thread >>>> >>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native >>>> memory needed by the process plus thread stacks. Not unlikely, that >>>> you ran out of memory address space for a 32 bit process. >>>> >>>> The only fixes would then be: >>>> >>>> - switch to a 64 bit system >>>> >>>> - reduce heap if the app can work with less >>>> >>>> - improve performance or eliminate bottlenecks so that the app >>>> works with less threads >>>> >>>> - limit you connector thread pool size. That will still mean that >>>> if requests begin to queue because of performance problems, the web >>>> server can't create additional connections, but you won't get in an >>>> irregular situation as you experience now. In that case you would >>>> need to configure a low idle timeout for the connections on the JK >>>> and TC side. >>> >>> It may also be possible to lower the thread stack size with the -Xss >>> option. >>> >>> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would >>> lowering it to 64k be a bit too low? What sort of repercussions >>> would we run into? Very helpful information by the way. >> >> It depends on your apps, so you'll need to test and see. If you go >> too low, you'll get StackOverflow exceptions. If you see those, just >> gradually increase until they go away. >> >> Dan >> >> >>> >>> -Isaac >>> >>> >>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#thread >>> s_ >>> >>> oom >>> >>> Might buy you some room for a few additional threads. >>> >>> Dan >>> >>>> >>>> Regards, >>>> >>>> Rainer >>>> >>>> >>>> ------------------------------------------------------------------- >>>> -- >>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: >>>> users-h...@tomcat.apache.org >>>> >>> >>> >>> -------------------------------------------------------------------- >>> - >>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> -------------------------------------------------------------------- >>> - >>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> Ok so the problem just happened again just now. Dan, Can you >> elaborate on how to configure limiting the connector thread pool >> size. I am also going to lower the thread stack size as you >> recommended. It seems like this problem creeps up when we have a >> hiccup in connectivity at our data center. Perhaps I also need to >> lower the idle timeout some more between tomcat and mod_jk. They are >> also between a firewall by the way, so I can configure a timeout >> between the two there as well. We aren't experiencing too many idle >> disconnects there. > > See maxConnections / maxThreads on the Connector tag. > > http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Impl > ementation > > or Executor if you’re using an executor. > > http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html >... and you definitely *should* be using a manually-configured Executor. >- -chris Ok so move to a manually configured executor and set the thread stack size to be smaller than the 1024k default sounds like it should definitely help out a bit. -Isaac --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
