Re: Tomcat Crashed when the concurrent Users reached 150
Thanks a Lot Mark On Mon, Dec 9, 2019 at 2:22 PM Mark Thomas wrote: > On 09/12/2019 06:41, Jayaram Ponnusamy wrote: > > Thanks for your Valuable Comments: > > We are using Apache Tomcat/8.0.50 and Mod_JK mod_jk/1.2.28 > > > > The Above configuration made by some consultant. We are in the > situation to > > correct all the things to make it for high availability. > > Also we are in the process of upgrading the apache to 2.4.25 with mod_jk/ > > 1.2.42. And Event MPM instead of Prefork. > > > > Please let us know do we need to use worker MPM also or EVENT MPM is > fine? > > If Event MPM with below Configuration then how many concurrent users can > > access the site and what are the relevant changes needs to be done on > > Tomcat Side? > > On Tomcat Servers We have 16GB RAM with 8CPU [Two Nodes] > > > > # event MPM > > > > StartServers 1 > > ServerLimit 7 > > MinSpareThreads 250 > > MaxSpareThreads 2500 > > ThreadsPerChild 500 > > ThreadLimit 500 > > MaxRequestWorkers 3500 > > MaxConnectionsPerChild 0 > > > > > > > > # Tomcat Configuration > > > URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" > > minSpareThreads="10" connectionTimeout="-1" /> > > > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > > connectionTimeout="2" acceptCount="2000" /> > > > > > > # Tomcat Configuration > > -Xms2048M -Xmx6144M -XX:PermSize=1024m -XX:MaxPermSize=2048m > > > > > > > > > > On Thu, Dec 5, 2019 at 7:27 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Jayaram, > > > > On 12/5/19 03:19, Jayaram Ponnusamy wrote: > We are using apache 2.2.21 on RHEL6.9 and in Backend we are using > Tomcat AppServer with JavaBased CMS. Our Server Architecture is > *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL > Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* > > > > That version of Apache httpd is pretty old and unsupported. RHEL 6.9 > > is also out of support (if I'm reading their Wikipedia page > > correctly). You may want to upgrade everything while you are looking > > at all this. > > > We are using Apache to connect Tomcat by MOD_JK and mapping the > URL (Attached VirtualHost.conf), and No applications/code are > running on WebServers. > > 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two > WebServers (4 CPU & 8GB RAM on Each Servers) > > *PROBLEM:* When the Child Process count is reached / crossed 200 > like below then site is not accessible > > > > Do you mean that new connections from httpd -> Tomcat cannot be made? > > > when the child processes reached 300 or More than site crashed > > > > Please define "crashed". Got an exception? JVM crash (with hs_pid* > > file)? Kernel panic? > > > and then we bring the site back by restarting Tomcat & Apache > HTTP. > > > > Do you need to restart httpd? Or only Tomcat? Or will restarting httpd > > work? > > > Kindly please check my configuration and help to allow at-least > 500 concurrent users. > > *CONFIGURATION:* * httpd.conf* StartServers > 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 > StartServers 6 MaxClients 230 > MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 > MaxRequestsPerChild 0 > > > > Which MPM are you actually using? You have listed both prefork and > > worker. AFAICT you have to pick one at runtime. > > > > For prefork, the maximum number of connections you should expect to be > > made to each Tomcat backend is 3500 (the value of MaxClients). > > > > For worker, the maximum number of connections you should expect to be > > made to each Tomcat backend is 230. > > > *Tomcat:* redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" > maxThreads="600" minSpareThreads="10" connectionTimeout="-1" /> > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > connectionTimeout="2" acceptCount="2000" /> > > > > Note that AJP connections are expected to be persistent; they don't > > close when the request has completed. They don't even really do "keep > > alive" in the same sense as HTTP keepalive. It's more like "always > > keepalive". Once an httpd process opens a connection to a Tomcat > > instance, that connection will remain open for quite a while as long > > as requests continue to be made. > > > > If you are using prefork MPM, the the version of Tomcat you are using > > is critical to understanding what is happening, here. If you are using > > a BIO-based connector, then you will need to increase the number of > > threads on each Tomcat server from 600 to 3500. > > > > If you are using the worker MPM, then your Tomcat should be able to > > handle the 230 maximum connections configured there. > > TL;DR - Try switching to the NIO AJP connector on Tomcat. > > Take a look at this session I uploaded from TomcatCon London. You > probably want to start around
Re: Tomcat Crashed when the concurrent Users reached 150
On 09/12/2019 06:41, Jayaram Ponnusamy wrote: > Thanks for your Valuable Comments: > We are using Apache Tomcat/8.0.50 and Mod_JK mod_jk/1.2.28 > > The Above configuration made by some consultant. We are in the situation to > correct all the things to make it for high availability. > Also we are in the process of upgrading the apache to 2.4.25 with mod_jk/ > 1.2.42. And Event MPM instead of Prefork. > > Please let us know do we need to use worker MPM also or EVENT MPM is fine? > If Event MPM with below Configuration then how many concurrent users can > access the site and what are the relevant changes needs to be done on > Tomcat Side? > On Tomcat Servers We have 16GB RAM with 8CPU [Two Nodes] > > # event MPM > > StartServers 1 > ServerLimit 7 > MinSpareThreads 250 > MaxSpareThreads 2500 > ThreadsPerChild 500 > ThreadLimit 500 > MaxRequestWorkers 3500 > MaxConnectionsPerChild 0 > > > > # Tomcat Configuration > URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" > minSpareThreads="10" connectionTimeout="-1" /> > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > connectionTimeout="2" acceptCount="2000" /> > > > # Tomcat Configuration > -Xms2048M -Xmx6144M -XX:PermSize=1024m -XX:MaxPermSize=2048m > > > > > On Thu, Dec 5, 2019 at 7:27 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Jayaram, > > On 12/5/19 03:19, Jayaram Ponnusamy wrote: We are using apache 2.2.21 on RHEL6.9 and in Backend we are using Tomcat AppServer with JavaBased CMS. Our Server Architecture is *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* > > That version of Apache httpd is pretty old and unsupported. RHEL 6.9 > is also out of support (if I'm reading their Wikipedia page > correctly). You may want to upgrade everything while you are looking > at all this. > We are using Apache to connect Tomcat by MOD_JK and mapping the URL (Attached VirtualHost.conf), and No applications/code are running on WebServers. 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two WebServers (4 CPU & 8GB RAM on Each Servers) *PROBLEM:* When the Child Process count is reached / crossed 200 like below then site is not accessible > > Do you mean that new connections from httpd -> Tomcat cannot be made? > when the child processes reached 300 or More than site crashed > > Please define "crashed". Got an exception? JVM crash (with hs_pid* > file)? Kernel panic? > and then we bring the site back by restarting Tomcat & Apache HTTP. > > Do you need to restart httpd? Or only Tomcat? Or will restarting httpd > work? > Kindly please check my configuration and help to allow at-least 500 concurrent users. *CONFIGURATION:* * httpd.conf* StartServers 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 StartServers 6 MaxClients 230 MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 MaxRequestsPerChild 0 > > Which MPM are you actually using? You have listed both prefork and > worker. AFAICT you have to pick one at runtime. > > For prefork, the maximum number of connections you should expect to be > made to each Tomcat backend is 3500 (the value of MaxClients). > > For worker, the maximum number of connections you should expect to be > made to each Tomcat backend is 230. > *Tomcat:* >>> redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" minSpareThreads="10" connectionTimeout="-1" /> >>> URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" connectionTimeout="2" acceptCount="2000" /> > > Note that AJP connections are expected to be persistent; they don't > close when the request has completed. They don't even really do "keep > alive" in the same sense as HTTP keepalive. It's more like "always > keepalive". Once an httpd process opens a connection to a Tomcat > instance, that connection will remain open for quite a while as long > as requests continue to be made. > > If you are using prefork MPM, the the version of Tomcat you are using > is critical to understanding what is happening, here. If you are using > a BIO-based connector, then you will need to increase the number of > threads on each Tomcat server from 600 to 3500. > > If you are using the worker MPM, then your Tomcat should be able to > handle the 230 maximum connections configured there. TL;DR - Try switching to the NIO AJP connector on Tomcat. Take a look at this session I uploaded from TomcatCon London. You probably want to start around 35:00 and the topic of thread exhaustion. https://youtu.be/2QYWp1k5QQM Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Crashed when the concurrent Users reached 150
Thanks for your Valuable Comments: We are using Apache Tomcat/8.0.50 and Mod_JK mod_jk/1.2.28 The Above configuration made by some consultant. We are in the situation to correct all the things to make it for high availability. Also we are in the process of upgrading the apache to 2.4.25 with mod_jk/ 1.2.42. And Event MPM instead of Prefork. Please let us know do we need to use worker MPM also or EVENT MPM is fine? If Event MPM with below Configuration then how many concurrent users can access the site and what are the relevant changes needs to be done on Tomcat Side? On Tomcat Servers We have 16GB RAM with 8CPU [Two Nodes] # event MPM StartServers 1 ServerLimit 7 MinSpareThreads 250 MaxSpareThreads 2500 ThreadsPerChild 500 ThreadLimit 500 MaxRequestWorkers 3500 MaxConnectionsPerChild 0 # Tomcat Configuration # Tomcat Configuration -Xms2048M -Xmx6144M -XX:PermSize=1024m -XX:MaxPermSize=2048m On Thu, Dec 5, 2019 at 7:27 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Jayaram, > > On 12/5/19 03:19, Jayaram Ponnusamy wrote: > > We are using apache 2.2.21 on RHEL6.9 and in Backend we are using > > Tomcat AppServer with JavaBased CMS. Our Server Architecture is > > *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL > > Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* > > That version of Apache httpd is pretty old and unsupported. RHEL 6.9 > is also out of support (if I'm reading their Wikipedia page > correctly). You may want to upgrade everything while you are looking > at all this. > > > We are using Apache to connect Tomcat by MOD_JK and mapping the > > URL (Attached VirtualHost.conf), and No applications/code are > > running on WebServers. > > > > 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two > > WebServers (4 CPU & 8GB RAM on Each Servers) > > > > *PROBLEM:* When the Child Process count is reached / crossed 200 > > like below then site is not accessible > > Do you mean that new connections from httpd -> Tomcat cannot be made? > > > when the child processes reached 300 or More than site crashed > > Please define "crashed". Got an exception? JVM crash (with hs_pid* > file)? Kernel panic? > > > and then we bring the site back by restarting Tomcat & Apache > > HTTP. > > Do you need to restart httpd? Or only Tomcat? Or will restarting httpd > work? > > > Kindly please check my configuration and help to allow at-least > > 500 concurrent users. > > > > *CONFIGURATION:* * httpd.conf* StartServers > > 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 > > StartServers 6 MaxClients 230 > > MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 > > MaxRequestsPerChild 0 > > Which MPM are you actually using? You have listed both prefork and > worker. AFAICT you have to pick one at runtime. > > For prefork, the maximum number of connections you should expect to be > made to each Tomcat backend is 3500 (the value of MaxClients). > > For worker, the maximum number of connections you should expect to be > made to each Tomcat backend is 230. > > > *Tomcat:* > redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" > > maxThreads="600" minSpareThreads="10" connectionTimeout="-1" /> > > > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > > connectionTimeout="2" acceptCount="2000" /> > > Note that AJP connections are expected to be persistent; they don't > close when the request has completed. They don't even really do "keep > alive" in the same sense as HTTP keepalive. It's more like "always > keepalive". Once an httpd process opens a connection to a Tomcat > instance, that connection will remain open for quite a while as long > as requests continue to be made. > > If you are using prefork MPM, the the version of Tomcat you are using > is critical to understanding what is happening, here. If you are using > a BIO-based connector, then you will need to increase the number of > threads on each Tomcat server from 600 to 3500. > > If you are using the worker MPM, then your Tomcat should be able to > handle the 230 maximum connections configured there. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3pL98ACgkQHPApP6U8 > pFiuAxAAjeYpDrMjjNgFvnC4Y6QS6TXN2dOJ3BuREKXjMitxoIWooNbcJuQ8uhVP > LbG+2Q9LyhfNxwrWm+urNZOaJBhSrJYjBoZfBG94LUPw7TcmBy24J5H4YjT00i4A > zwfDutOIUxmBBSrLoPT+updAnYEosR0S/ghYbtf7d/0xgjeYfdCINhVj5g2M0jyh > aahVO+bQ/qdaUzrJLdr2oka3ee5e5dpUr9LnextqXjpCJ+/vIN1h3pX0a4q+i9jD > Z55KjydFz2rloliOq4HvOlZcZqufN5QUlcZ5JELh06LurFVuuhQ6uM3r0ykIM2i3 > tMg/stXQL9HVCuo2T6eIfqW12COJWV+9uYe8urb9czPt9lCIyCI9L85HfyGdvnj8 > rvDnSF45ZwedcNRwgfXgECWV2dm3EnuNrK0ca5PyFESH9IE54Zhh/Mer/pslCz9B > wFdQJMBqqsrDEwDqifWjcobzJjTbB7olx1okq/MJ+AWntvSB+/afsNAC3zO33z2A > T/2ysW3MpnBCsjauxLyFXUQchLSv83laSDezrBOg3mZt1wZk3imW4XTJf7BBlsO8 >
Re: Tomcat Crashed when the concurrent Users reached 150
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jayaram, On 12/5/19 03:19, Jayaram Ponnusamy wrote: > We are using apache 2.2.21 on RHEL6.9 and in Backend we are using > Tomcat AppServer with JavaBased CMS. Our Server Architecture is > *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL > Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* That version of Apache httpd is pretty old and unsupported. RHEL 6.9 is also out of support (if I'm reading their Wikipedia page correctly). You may want to upgrade everything while you are looking at all this. > We are using Apache to connect Tomcat by MOD_JK and mapping the > URL (Attached VirtualHost.conf), and No applications/code are > running on WebServers. > > 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two > WebServers (4 CPU & 8GB RAM on Each Servers) > > *PROBLEM:* When the Child Process count is reached / crossed 200 > like below then site is not accessible Do you mean that new connections from httpd -> Tomcat cannot be made? > when the child processes reached 300 or More than site crashed Please define "crashed". Got an exception? JVM crash (with hs_pid* file)? Kernel panic? > and then we bring the site back by restarting Tomcat & Apache > HTTP. Do you need to restart httpd? Or only Tomcat? Or will restarting httpd work? > Kindly please check my configuration and help to allow at-least > 500 concurrent users. > > *CONFIGURATION:* * httpd.conf* StartServers > 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 > StartServers 6 MaxClients 230 > MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 > MaxRequestsPerChild 0 Which MPM are you actually using? You have listed both prefork and worker. AFAICT you have to pick one at runtime. For prefork, the maximum number of connections you should expect to be made to each Tomcat backend is 3500 (the value of MaxClients). For worker, the maximum number of connections you should expect to be made to each Tomcat backend is 230. > *Tomcat:* redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" > maxThreads="600" minSpareThreads="10" connectionTimeout="-1" /> > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > connectionTimeout="2" acceptCount="2000" /> Note that AJP connections are expected to be persistent; they don't close when the request has completed. They don't even really do "keep alive" in the same sense as HTTP keepalive. It's more like "always keepalive". Once an httpd process opens a connection to a Tomcat instance, that connection will remain open for quite a while as long as requests continue to be made. If you are using prefork MPM, the the version of Tomcat you are using is critical to understanding what is happening, here. If you are using a BIO-based connector, then you will need to increase the number of threads on each Tomcat server from 600 to 3500. If you are using the worker MPM, then your Tomcat should be able to handle the 230 maximum connections configured there. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3pL98ACgkQHPApP6U8 pFiuAxAAjeYpDrMjjNgFvnC4Y6QS6TXN2dOJ3BuREKXjMitxoIWooNbcJuQ8uhVP LbG+2Q9LyhfNxwrWm+urNZOaJBhSrJYjBoZfBG94LUPw7TcmBy24J5H4YjT00i4A zwfDutOIUxmBBSrLoPT+updAnYEosR0S/ghYbtf7d/0xgjeYfdCINhVj5g2M0jyh aahVO+bQ/qdaUzrJLdr2oka3ee5e5dpUr9LnextqXjpCJ+/vIN1h3pX0a4q+i9jD Z55KjydFz2rloliOq4HvOlZcZqufN5QUlcZ5JELh06LurFVuuhQ6uM3r0ykIM2i3 tMg/stXQL9HVCuo2T6eIfqW12COJWV+9uYe8urb9czPt9lCIyCI9L85HfyGdvnj8 rvDnSF45ZwedcNRwgfXgECWV2dm3EnuNrK0ca5PyFESH9IE54Zhh/Mer/pslCz9B wFdQJMBqqsrDEwDqifWjcobzJjTbB7olx1okq/MJ+AWntvSB+/afsNAC3zO33z2A T/2ysW3MpnBCsjauxLyFXUQchLSv83laSDezrBOg3mZt1wZk3imW4XTJf7BBlsO8 ffjIWAZ/BMp92+tXqpEtFBySGolGC4lOFU9vXrt9Kn3qCv2+fv0H8pCQTzsL+Nuf +CpEXnX8KnZtr97B/4M2LfRDhd8k9FZsrCU/mAEZUpKv89ZGWxI= =l2rL -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Crashed when the concurrent Users reached 150
On 05/12/2019 08:19, Jayaram Ponnusamy wrote: > Hello, > We are using apache 2.2.21 on RHEL6.9 and in Backend we are using Tomcat > AppServer with JavaBased CMS. > Our Server Architecture is *(F5 [For Load Balance] -> Apache WebServer [For > Redirection & URL Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* > > > We are using Apache to connect Tomcat by MOD_JK and mapping the URL > (Attached VirtualHost.conf), and No applications/code are running on > WebServers. > > 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) > 2. Two WebServers (4 CPU & 8GB RAM on Each Servers) Tomcat version? mod_jk version? Mark > > *PROBLEM:* > When the Child Process count is reached / crossed 200 like below then site > is not accessible when the child processes reached 300 or More than site > crashed and then we bring the site back by restarting Tomcat & Apache HTTP. > Kindly please check my configuration and help to allow at-least 500 > concurrent users. > > *CONFIGURATION:* > * httpd.conf* > > StartServers 80 > ServerLimit 3500 > MaxClients 3500 > MaxRequestsPerChild 0 > > > StartServers 6 > MaxClients 230 > MinSpareThreads 75 > MaxSpareThreads 150 > ThreadsPerChild 25 > MaxRequestsPerChild 0 > > > *Tomcat:* > URIEncoding="UTF-8" emptySessionPath="true" maxThreads="600" > minSpareThreads="10" connectionTimeout="-1" /> > URIEncoding="UTF-8" maxThreads="600" minSpareThreads="100" > connectionTimeout="2" acceptCount="2000" /> > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat Crashed when the concurrent Users reached 150
Hello, We are using apache 2.2.21 on RHEL6.9 and in Backend we are using Tomcat AppServer with JavaBased CMS. Our Server Architecture is *(F5 [For Load Balance] -> Apache WebServer [For Redirection & URL Mapping] -> Tomcat AppServer [Running a Java Based CMS]).* We are using Apache to connect Tomcat by MOD_JK and mapping the URL (Attached VirtualHost.conf), and No applications/code are running on WebServers. 1. Two TomcatServers (8 CPU & 16GB RAM on Each Servers) 2. Two WebServers (4 CPU & 8GB RAM on Each Servers) *PROBLEM:* When the Child Process count is reached / crossed 200 like below then site is not accessible when the child processes reached 300 or More than site crashed and then we bring the site back by restarting Tomcat & Apache HTTP. Kindly please check my configuration and help to allow at-least 500 concurrent users. *CONFIGURATION:* * httpd.conf* StartServers 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 StartServers 6 MaxClients 230 MinSpareThreads 75 MaxSpareThreads 150 ThreadsPerChild 25 MaxRequestsPerChild 0 *Tomcat:* -- -- *Thanks & Regards,* *Jayaram Ponnusamy* Mobile: +974-5512-3315