Re: tuning to accommodate Apache 2.4 event mpm

[Suvendu Sekhar Mondal]

John,

On Fri, Apr 28, 2017 at 11:48 AM, Rainer Jung  wrote:
> Am 27.04.2017 um 22:21 schrieb Mark Thomas:
>>
>> On 27/04/17 20:08, John Cartwright - NOAA Federal wrote:
>>>
>>> Thanks for your reply Mark!  My sysadmin tells me just that we're
>>> using "the defaults" for event_mpm.  However we are still using the
>>> BIO AJP connector:
>>>
>>> >> maxThreads="300" minSpareThreads="25"
>>> maxSpareThreads="75" connectionTimeout="2"/>
>>>
>>> Is there a way I can tell from the Tomcat side when there is thread
>>> starvation?
>>
>>
>> You'd need to look at the number of active connections. If it gets to
>> 300 then you likely have a problem.
>>
>> netstat might be the least invasive way of doing that. Other options
>> include the Manager app, JMX and thread dumps.
>>
>> Switching to NIO will allow you to server more connections with the same
>> number of threads.
>
>
> I assume you are using the latest mod_jk version?
>
> The current version of mod_jk also counts the total connection number (sum
> of connections from all Apache child processes to your Tomcat) and makes it
> available via the status worker for monitoring. The counting was a bit buggy
> in older versions, but should be correct in the latest one.
>
> Event MPM and mod_jk: the relevant phase in Apache is the handler phase,
> where the actual request forwarding is done by mod_jk. Things do not really
> need to change in the handler phase for proxying modules with event MPM.
> mod_jk does not need to know about the MPM or behave specific, the
> forwarding runs synchronously. By default mod_jk uses as many connections
> per Apache process as you have worker threads per Apache process configured.
> On the Tomcat side with BIO you need as many threads for the AJP connector
> as all you Apache fronting that Tomcat might have threads in total. Using
> the NIO connector on the Tomcat side is indeed better than BIO especially
> when you expect lots of concurrency (many Apache threads configured). Make
> sure you also use a recent Tomcat patch level then.
>
> What messages did you get in your mod_jk error log?

I also have the same question. Please check mod_jk log entries in the
time frame when you are getting 503s. It must have some clues.

We had similar type of problem a year ago(although we were using
mod_proxy_http). Later we found that backend Tomcat servers were not
responding to httpd's ping. As a result httpd was throwing 503s
sporadically. In our case long GC pauses were causing the problem. We
tuned GC and problem went away completely. You can check if you are
having something like that or not. Again you need to correlate 503
occurrences with GC logs.

https://tomcat.apache.org/connectors-doc/reference/workers.html

>
> For the mod_jk config, a good starter is the config files we bundle with the
> source download of mod_jk. You can also find them under
>
> http://svn.apache.org/viewvc/tomcat/jk/trunk/conf/
>
> These files set timeouts etc.
>
> Regards,
>
> Rainer
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[Rainer Jung]

Am 27.04.2017 um 22:21 schrieb Mark Thomas:

On 27/04/17 20:08, John Cartwright - NOAA Federal wrote:

Thanks for your reply Mark!  My sysadmin tells me just that we're
using "the defaults" for event_mpm.  However we are still using the
BIO AJP connector:



Is there a way I can tell from the Tomcat side when there is thread starvation?


You'd need to look at the number of active connections. If it gets to
300 then you likely have a problem.

netstat might be the least invasive way of doing that. Other options
include the Manager app, JMX and thread dumps.

Switching to NIO will allow you to server more connections with the same
number of threads.


I assume you are using the latest mod_jk version?

The current version of mod_jk also counts the total connection number 
(sum of connections from all Apache child processes to your Tomcat) and 
makes it available via the status worker for monitoring. The counting 
was a bit buggy in older versions, but should be correct in the latest one.


Event MPM and mod_jk: the relevant phase in Apache is the handler phase, 
where the actual request forwarding is done by mod_jk. Things do not 
really need to change in the handler phase for proxying modules with 
event MPM. mod_jk does not need to know about the MPM or behave 
specific, the forwarding runs synchronously. By default mod_jk uses as 
many connections per Apache process as you have worker threads per 
Apache process configured. On the Tomcat side with BIO you need as many 
threads for the AJP connector as all you Apache fronting that Tomcat 
might have threads in total. Using the NIO connector on the Tomcat side 
is indeed better than BIO especially when you expect lots of concurrency 
(many Apache threads configured). Make sure you also use a recent Tomcat 
patch level then.


What messages did you get in your mod_jk error log?

For the mod_jk config, a good starter is the config files we bundle with 
the source download of mod_jk. You can also find them under


http://svn.apache.org/viewvc/tomcat/jk/trunk/conf/

These files set timeouts etc.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

John,

(Grr... my initial reply didn't go to the list)

On 4/27/17 12:47 PM, John Cartwright - NOAA Federal wrote:
> Hello All,
> 
> We recently switched to Apache 2.4 and it's event mpm and are now 
> finding that some of our Tomcat 7 webapps are failing sporadically 
> when accessed via Apache - generally reporting 503s back to the 
> browser.  The webapps work reliably when accessed directly from 
> Tomcat.
> 
> We're using mod_jk for the connector - can someone please direct me
> to resources on tuning Tomcat settings to match those of the Apache
> event mpm or suggest what I should be tuning?

Can you post your settings related to the number of processes and
threads allowed on each httpd instance, as well as how many httpd
instances you have?

Also, please post your AJP  configuration as well.

Premature answer: you haven't counted correctly and your httpd instances
are trying to make more connections than Tomcat is configured to accept.

- -chris

-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJZAmHzAAoJEBzwKT+lPKRYFE8P/idSPbzSNEiqWQ6dPklZcDDs
58sj6uAsy5vLmFOt110LksTFG2ZqsO7ZbUH6b5R2QxPV5xJaJgO6fBN5gUddRtzj
JwOZ6mOAldelkQYnBfhuN82+155bfYx6RRAyX7N0MZrJ050+KibZxzWWxCvcIxtn
G4JwEdzK54mqqexWzbrC/1gge/Wf7Ju61tBNpVYfPst/5CuUIMuP+a3QvsiZRW8q
9VHS15I5Ah/Asgd37ikowQwJWBXV6hSvTYSWs7dhitKroj3nqV9+nPoYFZFuLfTI
G9cEZnKdeHPUjUpjtDzbXsiG8x3gjY4V0sJ614nZR2xFUzqZjvVCP0PPnIUgOoUE
f7EbdQOEfchPKH8U7nUany7nykpRrkMLEvpku7ZyKlnpaN4q1xyYk94XKhlakm+b
tpx2SFv3ETXaniYYUPeVJaKQkK20TmDZJYvpyH5FzeD/UdcuUkVvhfcrBeF4zM0p
/pXacE7mcnCdlBiaagkcCr9tfNGUxNikepB7/qq+/D5c4y23zxLsEC+HBaXwX6s5
tbvEI7/rZ5sQc3MsFWi1g0bvO4flgzpSG/T8PjxaULgFXAToxDzrMLGUXG71YLhK
Plw2MqjWRHCK1Lcj53q1/h42Z4txZt7YkJ1Wscx8cuZTpA8Zk6nkNWe3z95g+r1+
yFCmIJbdC/Df4xdrndLO
=0ssB
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[Mark Thomas]

On 27/04/17 20:08, John Cartwright - NOAA Federal wrote:
> Thanks for your reply Mark!  My sysadmin tells me just that we're
> using "the defaults" for event_mpm.  However we are still using the
> BIO AJP connector:
> 
>  maxThreads="300" minSpareThreads="25"
> maxSpareThreads="75" connectionTimeout="2"/>
> 
> Is there a way I can tell from the Tomcat side when there is thread 
> starvation?

You'd need to look at the number of active connections. If it gets to
300 then you likely have a problem.

netstat might be the least invasive way of doing that. Other options
include the Manager app, JMX and thread dumps.

Switching to NIO will allow you to server more connections with the same
number of threads.

Mark


> 
> --john
> 
> 
> On Thu, Apr 27, 2017 at 12:34 PM, Mark Thomas  wrote:
>> On 27/04/17 17:47, John Cartwright - NOAA Federal wrote:
>>> Hello All,
>>>
>>> We recently switched to Apache 2.4 and it's event mpm and are now
>>> finding that some of our Tomcat 7 webapps are failing sporadically
>>> when accessed via Apache - generally reporting 503s back to the
>>> browser.  The webapps work reliably when accessed directly from
>>> Tomcat.
>>>
>>> We're using mod_jk for the connector - can someone please direct me to
>>> resources on tuning Tomcat settings to match those of the Apache event
>>> mpm or suggest what I should be tuning?
>>
>> Might you be using the BIO AJP connector on the Tomcat end? If so,
>> switching to the NIO AJP connector should solve most if not all of the
>> issues.
>>
>> (I suspect the problem is thread starvation although I can't find good
>> docs on how the mod_jk connection pools work on the httpd end with the
>> event worker.)
>>
>> Mark
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[John Cartwright - NOAA Federal]

Thanks for your reply Mark!  My sysadmin tells me just that we're
using "the defaults" for event_mpm.  However we are still using the
BIO AJP connector:



Is there a way I can tell from the Tomcat side when there is thread starvation?

--john


On Thu, Apr 27, 2017 at 12:34 PM, Mark Thomas  wrote:
> On 27/04/17 17:47, John Cartwright - NOAA Federal wrote:
>> Hello All,
>>
>> We recently switched to Apache 2.4 and it's event mpm and are now
>> finding that some of our Tomcat 7 webapps are failing sporadically
>> when accessed via Apache - generally reporting 503s back to the
>> browser.  The webapps work reliably when accessed directly from
>> Tomcat.
>>
>> We're using mod_jk for the connector - can someone please direct me to
>> resources on tuning Tomcat settings to match those of the Apache event
>> mpm or suggest what I should be tuning?
>
> Might you be using the BIO AJP connector on the Tomcat end? If so,
> switching to the NIO AJP connector should solve most if not all of the
> issues.
>
> (I suspect the problem is thread starvation although I can't find good
> docs on how the mod_jk connection pools work on the httpd end with the
> event worker.)
>
> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[Mark Thomas]

On 27/04/17 17:47, John Cartwright - NOAA Federal wrote:
> Hello All,
> 
> We recently switched to Apache 2.4 and it's event mpm and are now
> finding that some of our Tomcat 7 webapps are failing sporadically
> when accessed via Apache - generally reporting 503s back to the
> browser.  The webapps work reliably when accessed directly from
> Tomcat.
> 
> We're using mod_jk for the connector - can someone please direct me to
> resources on tuning Tomcat settings to match those of the Apache event
> mpm or suggest what I should be tuning?

Might you be using the BIO AJP connector on the Tomcat end? If so,
switching to the NIO AJP connector should solve most if not all of the
issues.

(I suspect the problem is thread starvation although I can't find good
docs on how the mod_jk connection pools work on the httpd end with the
event worker.)

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

tuning to accommodate Apache 2.4 event mpm

[John Cartwright - NOAA Federal]

Hello All,

We recently switched to Apache 2.4 and it's event mpm and are now
finding that some of our Tomcat 7 webapps are failing sporadically
when accessed via Apache - generally reporting 503s back to the
browser.  The webapps work reliably when accessed directly from
Tomcat.

We're using mod_jk for the connector - can someone please direct me to
resources on tuning Tomcat settings to match those of the Apache event
mpm or suggest what I should be tuning?

Thanks!

--john

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org