Am 22.10.2018 um 00:43 schrieb Jerry Malcolm:
On 10/21/2018 7:27 AM, Rainer Jung wrote:
Plus:
mod_http2 uses additional threads per Apache httpd process. Look for
H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
default it seems you get ThreadsPerChild normal http worker threads
plus ThreadsPerChild HTTP/2 worker threads.
So the old-style assumption of mod_jk, that it is enough to provide as
many backend connections per httpd process, as normal httpd worker
threads are configured (ThreadsPerChild), does no longer apply for
HTTP/2.
You could try increasing the mod_jk pool size (connection_pool_size)
to 2*ThreadsPerChild, but you might also want to consider switching to
mod_proxy.
Regards,
Rainer
Rainer,
I've done some reading up on mpm, http/2, etc. But it's given me more
questions than answers. I need a bit of education help sorting this out.
Back to the original problem. I need to understand these mod_jk log
entries (there are hundreds of them...):
[Sun Oct 21 17:15:19 2018] [10116:5456] [warn]
ajp_get_endpoint::jk_ajp_common.c (3229): Unable to get the free
endpoint for worker worker1 from 64 slots
That means the size of the per process connection pool used to connect
to Tomcat "worker1" is 64 and all 64 connections were busy when another
one was needed. Probably this 64 was not configured by you but instead
auto-sized via ThreadsPerChild (MPM sizing). From what I understand
about the HTTP/2 implementation by mod_http2, you should try in doubling
the number of endpoints by setting
worker.worker1.connection_pool_size=128 in workers.properties without
changing ThreadsPerChild. Some info on the param can be found at
http://tomcat.apache.org/connectors-doc/reference/workers.html.
What you need to keep in mind: it is also possible, that the real
culprit is, that already 64 connections from one Apache process to the
backend were busy. That could mean, that the requests became too slow in
Tomcat or more likely in your application running in Tomcat. So I would
suggest you also take thread dumps, in this case of Tomcat worker1, when
the above message happens frequently to find out, what those 64
concurrent requests are doing (that use up all 64 possible connections).
[Sun Oct 21 17:15:19 2018] [10116:5456] [error] jk_handler::mod_jk.c
(2744): Could not get endpoint for worker=worker1
[Sun Oct 21 17:15:19 2018] [10116:5456] [info] jk_handler::mod_jk.c
(2806): Service error=0 for worker=worker1
These two are just follow-up messages of the above.
a) what does a 'free endpoint' represent? Is it an httpd thread that it
can't get? Is it a connection to Tomcat? Something else?
Connection from a specific Apache httpd process to Tomcat. These
endpoints=connections are pooled per Apache process. The number of
connections can be configured via worker.WORKER_NAME.connection_pool_size.
b) it says it has 64 of whatever it's looking for.... what is a 'slot'?
Just a synonym for endpoint. By slot in that message is meant an item in
the pool.
I currently have no mpm enabled. I am running on WAMP (May2018
download). From what I can tell after much digging, Apache Windows
does not use prefork, worker, or event mpm and rather has it's own WinNT
mpm. The default config file has only two parms for WinNT. But this mpm
config file is commented out by default. I'll enable it.
Ah, Windows. So unlike Unix/Linux we do only have a single child
process, which does all the work. The number 64 is the default number of
threads in this child for Windows.
<IfModule mpm_winnt_module>
ThreadsPerChild 150
MaxConnectionsPerChild 0
</IfModule>
OK, after that you have 150 threads in the one and only Apache httpd
child process, that does all the work. mod_jk will automatically also
set the number of endpoints (slots) = the connection_pool_size to 150,
if not configured by you otherwise.
You suggested increasing connection_pool_size. Is there a way to see
what pool size is set to by default? The docs talk about iis, iPlanet,
etc defaults. But it's not clear what it's set to for my default
config. Is there some kind of config dump option for mod_jk?
Configure a status worker. It has plenty of interesting info. For
monitoring it is enough to configure a read-only status worker. See
http://tomcat.apache.org/connectors-doc/reference/status.html.
You also suggested moving to mod_proxy. You really lost me there. I
found info on how to config proxy, etc. But I couldn't find anything
about how proxy differs from non-proxy. What problem does it solve? And
how would it potentially fix my issue?
mod_jk and mod_proxy are simply two different ways of connecting Apache
httpd to backend Tomcats. mod_jk uses a special protocol named AJP13
(and an AJP connector in Tomcat which understands that protocol).
mod_proxy simply uses HTTP or HTTPS and the normal HTTP/HTTPS connector
in Tomcat. mod_jk is provided by the Tomcat project and configured
mostly via workers.properties, mod_proxy is provided as part of Apache
httpd and configured via "normal" Proxy... httpd directives.
Sorry for all the questions. But I'm very concerned about just shooting
in the dark making all kinds of changes when I really don't know if and
how it's helping or hurting. Any education to help me understand what
the fundamental problem is with 'endpoints' in the mod_jk log will be
greatly appreciated.
Now that I understand that you are on Windows, the default size of 64
woker threads in total for Apache httpd is not that big. So i suggest
- enable the explicit config with 150 as you already did
- increase connection_pool_size to 300 if you are actually using HTTP/2
- enable a read-only status worker for monitoring. The number of busy
endpoints can be seen under the busy column, the max number since start
or statistics reset under max_busy. There are also the number of
connections and max number of connections since start/statistics reset.
The difference between busy and number of connections is, that a
connection is counted in busy only, if it is currently being used for a
request/response.
- If you are till getting errors take a few thread dumps of your backend
Tomcats during the time you get errors and provide them to the mailing
list, possibly after redacting things there, that you don't want public.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
