AW: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
> your httpd nodes (I guess just 1) Just to clarify. We have one apache-httpd (and one tomcat) running. >tomcat tomcat-8.5.28 >connector-impl I'd say we are using nio ... 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.16] using APR version [1.6.3]. 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 09-Aug-2018 19:54:31.185 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.0.2n 7 Dec 2017] 09-Aug-2018 19:54:31.608 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"] ... 09-Aug-2018 19:55:36.046 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] ... -Ursprüngliche Nachricht- Von: Christopher Schultz Gesendet: Montag, 17. September 2018 20:36 An: users@tomcat.apache.org Betreff: Re: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/17/18 02:54, Clemens Wyss DEV wrote: >> How many httpd nodes? > 1 Ok. >> Which MPM? > Tried worker and event These should have roughly the same semantics. >> Per MPM, what is MaxClients, etc.? > MaxRequestWorkers 512 ThreadsPerChild32 So the maximum number of simultaneous connections your httpd nodes (I guess just 1) will try to send to the backend Tomcat node will be 512 based upon MaxRequestWorkers. >> How many Tomcat nodes? > 1 > >> Tomcat connector configuration? > maxThreads="400" protocol="AJP/1.3" /> So you can have 512 AJP connections from httpd connecting to Tomcat which has 400 threads and, depending upon which connector is actually in use, you may be limited to 400 connections or 10,000. Based upon your situation, I'd say you are using Java BIO and are thus limited to 400 connections. What version of Tomcat are you using? What connector eventually gets used? There are several: Java BIO Java NIO Java NIO2 APR You can tell from the startup messages what kind of connector you get; it's shown when the connector starts and emits an INFO message saying that it's configured. - -chris > -Ursprüngliche Nachricht- Von: Christopher Schultz > Gesendet: Sonntag, 16. September > 2018 20:42 An: users@tomcat.apache.org Betreff: Re: > jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... > > Clemens, > > On 9/15/18 06:50, Clemens Wyss DEV wrote: >> Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] >> [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): >> Could not get endpoint for worker=testAPJ" > >> errors in our md_jk.log. Worker properties are as follwos: > >> ... worker.list=testAPJ > >> worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 >> worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 >> worker.testAJP.connection_pool_timeout=600 ... > >> At that point Apache seems to be stuck/struggling (but our tomcat >> does not seem to be under pressure). Restarting Apache solves the >> issue ... till it pops up again ... > >> What is happening? What needs tob e tuned? > >> Apache 2.4.34, tried both event- and worker-MPM > > How many httpd nodes? Which MPM? Per MPM, what is MaxClients, etc.? > > How many Tomcat nodes? Tomcat connector configuration? > > Usually, this happens because you are overwhelming your Tomcat nodes > with connections from the web server. Remember that every Tomcat node > needs to be able to handle the maximum number connections from *all > web servers simultaneously*. That number is probably way more than > your Tomcat connector is configured to handle. > > -chris > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluf8/sACgkQHPApP6U8 pFh5Dw/+KPqqu13CiyyjvZx2QrCFyQuxW4m2iXW8TK2AEzRR3G0jlefFZgrJWpfb TwZwr57XEE3cKo7l2R8YcFOcJARfkZRsx7D8dHvqsmDpL9zlHBCnZ+PVEBxla1Qk oyh1n0KYJMO98sHaoZghKjYJ8b5GALe2XeGhCf/LTZvdkM+3bmY2S64x4vXuDcfG 47VGjWu0xzjFQJ4INd2bnRTXbXvScGmdv8qbHX2EmT/PjsAmzeo+hxiOBWbPJG6K b+Mpf1Nb9SpcwBRLHLy4/xX2yo34vrkiLcmvBO7HUoIPCWpcLFAdtmFLSBWStBK1
Re: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/17/18 02:54, Clemens Wyss DEV wrote: >> How many httpd nodes? > 1 Ok. >> Which MPM? > Tried worker and event These should have roughly the same semantics. >> Per MPM, what is MaxClients, etc.? > MaxRequestWorkers 512 ThreadsPerChild32 So the maximum number of simultaneous connections your httpd nodes (I guess just 1) will try to send to the backend Tomcat node will be 512 based upon MaxRequestWorkers. >> How many Tomcat nodes? > 1 > >> Tomcat connector configuration? > maxThreads="400" protocol="AJP/1.3" /> So you can have 512 AJP connections from httpd connecting to Tomcat which has 400 threads and, depending upon which connector is actually in use, you may be limited to 400 connections or 10,000. Based upon your situation, I'd say you are using Java BIO and are thus limited to 400 connections. What version of Tomcat are you using? What connector eventually gets used? There are several: Java BIO Java NIO Java NIO2 APR You can tell from the startup messages what kind of connector you get; it's shown when the connector starts and emits an INFO message saying that it's configured. - -chris > -Ursprüngliche Nachricht- Von: Christopher Schultz > Gesendet: Sonntag, 16. September > 2018 20:42 An: users@tomcat.apache.org Betreff: Re: > jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... > > Clemens, > > On 9/15/18 06:50, Clemens Wyss DEV wrote: >> Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] >> [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): >> Could not get endpoint for worker=testAPJ" > >> errors in our md_jk.log. Worker properties are as follwos: > >> ... worker.list=testAPJ > >> worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 >> worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 >> worker.testAJP.connection_pool_timeout=600 ... > >> At that point Apache seems to be stuck/struggling (but our tomcat >> does not seem to be under pressure). Restarting Apache solves the >> issue ... till it pops up again ... > >> What is happening? What needs tob e tuned? > >> Apache 2.4.34, tried both event- and worker-MPM > > How many httpd nodes? Which MPM? Per MPM, what is MaxClients, > etc.? > > How many Tomcat nodes? Tomcat connector configuration? > > Usually, this happens because you are overwhelming your Tomcat > nodes with connections from the web server. Remember that every > Tomcat node needs to be able to handle the maximum number > connections from *all web servers simultaneously*. That number is > probably way more than your Tomcat connector is configured to > handle. > > -chris > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluf8/sACgkQHPApP6U8 pFh5Dw/+KPqqu13CiyyjvZx2QrCFyQuxW4m2iXW8TK2AEzRR3G0jlefFZgrJWpfb TwZwr57XEE3cKo7l2R8YcFOcJARfkZRsx7D8dHvqsmDpL9zlHBCnZ+PVEBxla1Qk oyh1n0KYJMO98sHaoZghKjYJ8b5GALe2XeGhCf/LTZvdkM+3bmY2S64x4vXuDcfG 47VGjWu0xzjFQJ4INd2bnRTXbXvScGmdv8qbHX2EmT/PjsAmzeo+hxiOBWbPJG6K b+Mpf1Nb9SpcwBRLHLy4/xX2yo34vrkiLcmvBO7HUoIPCWpcLFAdtmFLSBWStBK1 m29fcvr0Eg1UREajDpYKICrUfArWUMKZZv4OEkrZaWr93vbPV2bMnJ/qC2k7lQIt k5mdag4L5vEonACQv7o9cstAEW5Q6FN9mwHvwyZ42n97CgA/pBDbNIIElD5l/Y4s o6jm9xUmc+6BUiR4Wq9WxBlVeziqV1RBJaExDr6CNpJ6FVRgHBuC/UdEIeqa6Stv PwoKT3dZJaXDmBeXFbY9024b6eKkR3T0y1murM1V/udL+ZfBWAAGRW2+/FNVLPhx e3JxMactH7Dt3MTTRRRtOHgSOaIMtH+yNt6iw4yzJQDgAwtnHTWt/XjR+D/K7nux POkNME2yMstqx/hvwmXgxb7mQusvYKHCb9Bh+K+rraPrm6Vo1xc= =45Ix -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [OT] Tomcat Clustering Support
-Original Message-
From: Christopher Schultz
Sent: Tuesday, August 28, 2018 3:35 PM
To: users@tomcat.apache.org
Subject: Re: [OT] Tomcat Clustering Support
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Scott,
I'm no Tomcat-clustering expert, but...
On 8/28/18 13:59, Scott Evans wrote:
> channelSendOptions="10" channelStartOptions="3">
>
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true" mapSendOptions="10" />
>
> className="org.apache.catalina.tribes.group.GroupChannel">
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.1.0.13" port="45522" frequency="500" dropTime="15000"
> soTimeout="1" domain="mercer"/> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="10.255.250.34" port="4003" selectorTimeout="8000"
> maxThreads="25" />
>
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter
">
>
>
className="org.apache.catalina.tribes.transport.nio.PooledParallelSend
er"
>
>
timeout="8000" poolSize="25"
> /> className="org.apache.catalina.tribes.group.interceptors.TcpPingInterc
eptor"/>
>
>
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDet
ector"/>
>
>
className="org.apache.catalina.tribes.group.interceptors.MessageDispat
ch15Interceptor"/>
>
>
>
> className="org.apache.catalina.tribes.group.interceptors.StaticMembers
hipInterceptor">
>
> className="org.apache.catalina.tribes.membership.StaticMember"
> domain="mercer" uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,1,0}"/>
>
> className="org.apache.catalina.tribes.membership.StaticMember"
> port="4001" host="10.255.250.35" domain="mercer"
> uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,0,0}" />
>
> className="org.apache.catalina.tribes.membership.StaticMember"
> port="4002" host="10.255.250.35" domain="mercer"
> uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,0,1}" />
>
> className="org.apache.catalina.tribes.membership.StaticMember"
> port="4004" host="10.255.250.34" domain="mercer"
> uniqueId="{5,6,7,0,1,2,3,4,0,0,0,0,0,0,1,1}" />
>
>
>
>
>
>
> /> filter=".*javax\.faces\.resource.*"/> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/"
> watchDir="/tmp/war-listen/" watchEnabled="false"/>
>
>
>
>
It looks like you have both multicast AND static membership enabled.
Keichi's presentation on Clustering at ApacheCon Miami (2017) has a slide
(it's slide #38 here:
https://events.static.linuxfound.org/sites/events/files/slides/TomcatClu
ster_3.pdf)
that says that using static-membership requires that you disable multicast.
Also, just confirming that you have two Tomcat nodes on one IP address
(10.255.250.35, ports 4001 and 4002).
Can you post a thread dump of a deadlock situation? Only the deadlocked
threads should really be necessary to post. Can you replicate the deadlock
without using your own full application? That is, can you create a simple
application that can be used to reproduce this on a similarly-configured
test instance (cluster) of Tomcat nodes?
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluFo/MACgkQHPApP6U8
pFhG4xAAkUb/Zj9HtwRle8xypc8hrmLfiifo9acIKbb1H3k/2VfYW3EjGqVRzV6c
E5iGf3JFlnsDEsMIi/oSTObe/aJ15y6z1qfCpud1BRSvi1yHr8jf6W+/M4/QcMNk
JerBmsx8dgoLteVq34xEld678NftgufaHpd3z5y3HnqfX0MoJkCOaYH5lUbA5MpI
61vEngWnWsLvFyTcf+h9PnkxsH5CdA0A9Hjsg56MESAyGoEZ1Jx1MkrIooFLOHVx
sgxciUIosQy5wqIbpZrZMteB1T6gFSvVsoTCu2ogubJUU216xt3XEezVtksL9Kfc
+1GbaDeMb65W6GlUU9W61TPb4Id/2mcQ2oUyQERctvIib7GoTcpLJFSHkKlp81GL
vS3L4siQkSv1M6pIvAtnAJEVPogBgYJXnSVOObpGAmyaDkJt8k1OSCDWqVPmLfUm
mIlhDGBtngxl0pEM1juLFC2ulaOGS8Vjn5VGZgXDXZVQ6xVmqBDfl9o6x+IB+KDT
beOGXQKveI18K0qPjxfVtF9OyVgfeLoOzVw2AXAD8QBXorWPlEt53sbInv2r/a3l
UOKGvxxGpeqmzAtEwm0GxrJsDrfJ2tTp0eIDA94n7d3tuG+zoOgOFaMxXcryieyj
XXl+4+DjD7YxVAXNfUslP7eYglHh1SdJVc8/MwlH0g0fARY74/o=
=eKZv
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Hi Chris,
Just getting back to this. Thanks for the info on static membership and
multicast, we may give that a try. We've also finally been able to get a
thread dump of a recent deadlock situation that occurred which is good news.
To answer your other questions, you are correct that two Nodes are on one IP
and we do not have a way to create a simple program to reproduce the issue.
It is just randomly occurring, though very infrequently.
For next steps, due to the sensitive nature and voluminous amount of the
data contained in the thread dump, we would ask that we somehow take this
offline so it's not shared on the forums. Please let me know what our
options would be for that.
Thanks
-
To
AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
>How many httpd nodes? 1 >Which MPM? Tried worker and event >Per MPM, what is MaxClients, etc.? Listen 80 Listen 443 ServerRoot "/usr/local/apache2" ServerName localhost:80 PidFile logs/httpd.pid Timeout 10 UseCanonicalName Off KeepAlive On ServerTokens Prod MaxKeepAliveRequests 100 KeepAliveTimeout 1 TraceEnable off HostnameLookups Off ServerSignature Off ServerLimit16 MaxRequestWorkers 512 ThreadLimit32 ThreadsPerChild32 MaxConnectionsPerChild 0 >How many Tomcat nodes? 1 >Tomcat connector configuration? Thx for any advice - Clemens -Ursprüngliche Nachricht- Von: Christopher Schultz Gesendet: Sonntag, 16. September 2018 20:42 An: users@tomcat.apache.org Betreff: Re: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/15/18 06:50, Clemens Wyss DEV wrote: > Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] > [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): Could not > get endpoint for worker=testAPJ" > > errors in our md_jk.log. Worker properties are as follwos: > > ... worker.list=testAPJ > > worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 > worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 > worker.testAJP.connection_pool_timeout=600 ... > > At that point Apache seems to be stuck/struggling (but our tomcat does > not seem to be under pressure). Restarting Apache solves the issue ... > till it pops up again ... > > What is happening? What needs tob e tuned? > > Apache 2.4.34, tried both event- and worker-MPM How many httpd nodes? Which MPM? Per MPM, what is MaxClients, etc.? How many Tomcat nodes? Tomcat connector configuration? Usually, this happens because you are overwhelming your Tomcat nodes with connections from the web server. Remember that every Tomcat node needs to be able to handle the maximum number connections from *all web servers simultaneously*. That number is probably way more than your Tomcat connector is configured to handle. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlueo+oACgkQHPApP6U8 pFhwVw/+M48YAmMzOBmHelQyzyFQFOZj5hEy/bUVyDmTgJ6xWfiRxSgJRViRK6lW DxAs1YFdLdam/RZg9uraE/KKc01zAtNKPAbr134Vkz4/gVMbCs1Ndc1rlvb2Etjb dOX7JajPOgLZa8m0hIu4KitkhbkXBJmqT2zQOmLF1KZmyNmaxkgknQux3K5mn5P9 p/ml2N9FTaxh1PYVnQWPKbLlwBBRlxHrTMqeLLpy5rZh8x41sSCdNQI2eLOopWAZ 1bRkOFMMCdT66yp5VcPTMNCKBL98GWsewTsRhkO5EB54IoL/x/uIYkx7BuHpU/C4 lt9dSYECqzp1HZJBkJKAUxsGRCH2hcTQJ75g+X4QfhCFv6ZKQTy+j8cE6nvwXk5D T9krmy9flmKccmFq+EPwt8mIcC0aZQ9TRmCBW5+HRQh/LTf4BexA0pDq4Yi2zxc2 9PkZSaTGkFLUGpLDH5qIwNhH78LhHT1BjwWxkeupgLAscEIb1znjZhwiJ87zd/7/ HHKXttaIUJ2E4s1zptkg1m8dNuLS7GCv/GuO1TER2Nk/4BtIrHE1QojNZj0Logll B65vvtHS7UdIarfF1DU1Jq7mh+fJ7PZseFgPI2WukwhUndNqo+AZ8ipnJEEAKklS hxsPGW0vuRzyqXV0IHB7rQMwAEArnq8yEi0Mwj7WagPQmKrxDdE= =7/Kx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
