Re: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
Am 15.09.2018 um 12:50 schrieb Clemens Wyss DEV: Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): Could not get endpoint for worker=testAPJ" errors in our md_jk.log. Worker properties are as follwos: ... worker.list=testAPJ worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 worker.testAJP.connection_pool_timeout=600 ... At that point Apache seems to be stuck/struggling (but our tomcat does not seem to be under pressure). Restarting Apache solves the issue ... till it pops up again ... What is happening? What needs tob e tuned? Apache 2.4.34, tried both event- and worker-MPM Assuming this is mod_jk 1.2.44? Are there more setting for worker testAPJ? Normally mod_jk creates as many local connection structures (named endpoints) in each Apache httpd child process, as that process has worker threads. When an httpd worker thread wants to talk to tomcat, it retrieves such an endpoint and uses it to create and handle the commnunication. The error you observe means, that all endpoints were already in use. Since we create as many structures as there are worker threads - everything is per httpd process, this should not happen (and I don't remember any case were it did happen). Ideas what could go wrong: - setting the worker property connection_pool_size or the deprecated cachesize for worker testAPJ to a smaller value than your httpd ThreadsPerChild (32 from your config snippet). If not set, mod_jk automatically detects the number of httpd worker threads - setting connection_acquire_timeout to a small value. By default it is equals to retries*retry_interval which in turn by default is equals to 2*100 milliseconds. mod_jk will retry getting an endpoint before it shows you error message "retries" times with a sleep pause of "retry_interval" milliseconds but no longer than connection_acquire_timeout milliseconds. - retrieving and endpoint must acquire a lock first. On some platforms locking can lead to problems like false positives in deadlock detection. But i think this can't happen here since the code doesn't check the return value of the locking. - memory shortage leading to failing allocations (not likely but possible) Do you see any other log messages? Any ones in the httpd error log or especially the mod_jk log? There should be a WARN message of type "Unable to get the free endpoint for worker %s from %u slots" but maybe more before that final problem happens? What do you see with JkLogLevel info? Does the problem happen under high load or when your backend gets slow? What does "netstat -anp | grep 8009" show when the hang occurs? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
> your httpd nodes (I guess just 1) Just to clarify. We have one apache-httpd (and one tomcat) running. >tomcat tomcat-8.5.28 >connector-impl I'd say we are using nio ... 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.16] using APR version [1.6.3]. 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 09-Aug-2018 19:54:31.181 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 09-Aug-2018 19:54:31.185 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.0.2n 7 Dec 2017] 09-Aug-2018 19:54:31.608 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"] ... 09-Aug-2018 19:55:36.046 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] ... -Ursprüngliche Nachricht- Von: Christopher Schultz Gesendet: Montag, 17. September 2018 20:36 An: users@tomcat.apache.org Betreff: Re: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/17/18 02:54, Clemens Wyss DEV wrote: >> How many httpd nodes? > 1 Ok. >> Which MPM? > Tried worker and event These should have roughly the same semantics. >> Per MPM, what is MaxClients, etc.? > MaxRequestWorkers 512 ThreadsPerChild32 So the maximum number of simultaneous connections your httpd nodes (I guess just 1) will try to send to the backend Tomcat node will be 512 based upon MaxRequestWorkers. >> How many Tomcat nodes? > 1 > >> Tomcat connector configuration? > maxThreads="400" protocol="AJP/1.3" /> So you can have 512 AJP connections from httpd connecting to Tomcat which has 400 threads and, depending upon which connector is actually in use, you may be limited to 400 connections or 10,000. Based upon your situation, I'd say you are using Java BIO and are thus limited to 400 connections. What version of Tomcat are you using? What connector eventually gets used? There are several: Java BIO Java NIO Java NIO2 APR You can tell from the startup messages what kind of connector you get; it's shown when the connector starts and emits an INFO message saying that it's configured. - -chris > -Ursprüngliche Nachricht- Von: Christopher Schultz > Gesendet: Sonntag, 16. September > 2018 20:42 An: users@tomcat.apache.org Betreff: Re: > jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... > > Clemens, > > On 9/15/18 06:50, Clemens Wyss DEV wrote: >> Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] >> [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): >> Could not get endpoint for worker=testAPJ" > >> errors in our md_jk.log. Worker properties are as follwos: > >> ... worker.list=testAPJ > >> worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 >> worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 >> worker.testAJP.connection_pool_timeout=600 ... > >> At that point Apache seems to be stuck/struggling (but our tomcat >> does not seem to be under pressure). Restarting Apache solves the >> issue ... till it pops up again ... > >> What is happening? What needs tob e tuned? > >> Apache 2.4.34, tried both event- and worker-MPM > > How many httpd nodes? Which MPM? Per MPM, what is MaxClients, etc.? > > How many Tomcat nodes? Tomcat connector configuration? > > Usually, this happens because you are overwhelming your Tomcat nodes > with connections from the web server. Remember that every Tomcat node > needs to be able to handle the maximum number connections from *all > web servers simultaneously*. That number is probably way more than > your Tomcat connector is configured to handle. > > -chris > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluf8/sACgkQHPApP6U8 pFh5Dw/+KPqqu13CiyyjvZ
Re: AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/17/18 02:54, Clemens Wyss DEV wrote: >> How many httpd nodes? > 1 Ok. >> Which MPM? > Tried worker and event These should have roughly the same semantics. >> Per MPM, what is MaxClients, etc.? > MaxRequestWorkers 512 ThreadsPerChild32 So the maximum number of simultaneous connections your httpd nodes (I guess just 1) will try to send to the backend Tomcat node will be 512 based upon MaxRequestWorkers. >> How many Tomcat nodes? > 1 > >> Tomcat connector configuration? > maxThreads="400" protocol="AJP/1.3" /> So you can have 512 AJP connections from httpd connecting to Tomcat which has 400 threads and, depending upon which connector is actually in use, you may be limited to 400 connections or 10,000. Based upon your situation, I'd say you are using Java BIO and are thus limited to 400 connections. What version of Tomcat are you using? What connector eventually gets used? There are several: Java BIO Java NIO Java NIO2 APR You can tell from the startup messages what kind of connector you get; it's shown when the connector starts and emits an INFO message saying that it's configured. - -chris > -Ursprüngliche Nachricht- Von: Christopher Schultz > Gesendet: Sonntag, 16. September > 2018 20:42 An: users@tomcat.apache.org Betreff: Re: > jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... > > Clemens, > > On 9/15/18 06:50, Clemens Wyss DEV wrote: >> Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] >> [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): >> Could not get endpoint for worker=testAPJ" > >> errors in our md_jk.log. Worker properties are as follwos: > >> ... worker.list=testAPJ > >> worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 >> worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 >> worker.testAJP.connection_pool_timeout=600 ... > >> At that point Apache seems to be stuck/struggling (but our tomcat >> does not seem to be under pressure). Restarting Apache solves the >> issue ... till it pops up again ... > >> What is happening? What needs tob e tuned? > >> Apache 2.4.34, tried both event- and worker-MPM > > How many httpd nodes? Which MPM? Per MPM, what is MaxClients, > etc.? > > How many Tomcat nodes? Tomcat connector configuration? > > Usually, this happens because you are overwhelming your Tomcat > nodes with connections from the web server. Remember that every > Tomcat node needs to be able to handle the maximum number > connections from *all web servers simultaneously*. That number is > probably way more than your Tomcat connector is configured to > handle. > > -chris > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluf8/sACgkQHPApP6U8 pFh5Dw/+KPqqu13CiyyjvZx2QrCFyQuxW4m2iXW8TK2AEzRR3G0jlefFZgrJWpfb TwZwr57XEE3cKo7l2R8YcFOcJARfkZRsx7D8dHvqsmDpL9zlHBCnZ+PVEBxla1Qk oyh1n0KYJMO98sHaoZghKjYJ8b5GALe2XeGhCf/LTZvdkM+3bmY2S64x4vXuDcfG 47VGjWu0xzjFQJ4INd2bnRTXbXvScGmdv8qbHX2EmT/PjsAmzeo+hxiOBWbPJG6K b+Mpf1Nb9SpcwBRLHLy4/xX2yo34vrkiLcmvBO7HUoIPCWpcLFAdtmFLSBWStBK1 m29fcvr0Eg1UREajDpYKICrUfArWUMKZZv4OEkrZaWr93vbPV2bMnJ/qC2k7lQIt k5mdag4L5vEonACQv7o9cstAEW5Q6FN9mwHvwyZ42n97CgA/pBDbNIIElD5l/Y4s o6jm9xUmc+6BUiR4Wq9WxBlVeziqV1RBJaExDr6CNpJ6FVRgHBuC/UdEIeqa6Stv PwoKT3dZJaXDmBeXFbY9024b6eKkR3T0y1murM1V/udL+ZfBWAAGRW2+/FNVLPhx e3JxMactH7Dt3MTTRRRtOHgSOaIMtH+yNt6iw4yzJQDgAwtnHTWt/XjR+D/K7nux POkNME2yMstqx/hvwmXgxb7mQusvYKHCb9Bh+K+rraPrm6Vo1xc= =45Ix -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
>How many httpd nodes? 1 >Which MPM? Tried worker and event >Per MPM, what is MaxClients, etc.? Listen 80 Listen 443 ServerRoot "/usr/local/apache2" ServerName localhost:80 PidFile logs/httpd.pid Timeout 10 UseCanonicalName Off KeepAlive On ServerTokens Prod MaxKeepAliveRequests 100 KeepAliveTimeout 1 TraceEnable off HostnameLookups Off ServerSignature Off ServerLimit16 MaxRequestWorkers 512 ThreadLimit32 ThreadsPerChild32 MaxConnectionsPerChild 0 >How many Tomcat nodes? 1 >Tomcat connector configuration? Thx for any advice - Clemens -Ursprüngliche Nachricht- Von: Christopher Schultz Gesendet: Sonntag, 16. September 2018 20:42 An: users@tomcat.apache.org Betreff: Re: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ... -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/15/18 06:50, Clemens Wyss DEV wrote: > Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] > [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): Could not > get endpoint for worker=testAPJ" > > errors in our md_jk.log. Worker properties are as follwos: > > ... worker.list=testAPJ > > worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 > worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 > worker.testAJP.connection_pool_timeout=600 ... > > At that point Apache seems to be stuck/struggling (but our tomcat does > not seem to be under pressure). Restarting Apache solves the issue ... > till it pops up again ... > > What is happening? What needs tob e tuned? > > Apache 2.4.34, tried both event- and worker-MPM How many httpd nodes? Which MPM? Per MPM, what is MaxClients, etc.? How many Tomcat nodes? Tomcat connector configuration? Usually, this happens because you are overwhelming your Tomcat nodes with connections from the web server. Remember that every Tomcat node needs to be able to handle the maximum number connections from *all web servers simultaneously*. That number is probably way more than your Tomcat connector is configured to handle. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlueo+oACgkQHPApP6U8 pFhwVw/+M48YAmMzOBmHelQyzyFQFOZj5hEy/bUVyDmTgJ6xWfiRxSgJRViRK6lW DxAs1YFdLdam/RZg9uraE/KKc01zAtNKPAbr134Vkz4/gVMbCs1Ndc1rlvb2Etjb dOX7JajPOgLZa8m0hIu4KitkhbkXBJmqT2zQOmLF1KZmyNmaxkgknQux3K5mn5P9 p/ml2N9FTaxh1PYVnQWPKbLlwBBRlxHrTMqeLLpy5rZh8x41sSCdNQI2eLOopWAZ 1bRkOFMMCdT66yp5VcPTMNCKBL98GWsewTsRhkO5EB54IoL/x/uIYkx7BuHpU/C4 lt9dSYECqzp1HZJBkJKAUxsGRCH2hcTQJ75g+X4QfhCFv6ZKQTy+j8cE6nvwXk5D T9krmy9flmKccmFq+EPwt8mIcC0aZQ9TRmCBW5+HRQh/LTf4BexA0pDq4Yi2zxc2 9PkZSaTGkFLUGpLDH5qIwNhH78LhHT1BjwWxkeupgLAscEIb1znjZhwiJ87zd/7/ HHKXttaIUJ2E4s1zptkg1m8dNuLS7GCv/GuO1TER2Nk/4BtIrHE1QojNZj0Logll B65vvtHS7UdIarfF1DU1Jq7mh+fJ7PZseFgPI2WukwhUndNqo+AZ8ipnJEEAKklS hxsPGW0vuRzyqXV0IHB7rQMwAEArnq8yEi0Mwj7WagPQmKrxDdE= =7/Kx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Clemens, On 9/15/18 06:50, Clemens Wyss DEV wrote: > Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] > [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): Could > not get endpoint for worker=testAPJ" > > errors in our md_jk.log. Worker properties are as follwos: > > ... worker.list=testAPJ > > worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 > worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 > worker.testAJP.connection_pool_timeout=600 ... > > At that point Apache seems to be stuck/struggling (but our tomcat > does not seem to be under pressure). Restarting Apache solves the > issue ... till it pops up again ... > > What is happening? What needs tob e tuned? > > Apache 2.4.34, tried both event- and worker-MPM How many httpd nodes? Which MPM? Per MPM, what is MaxClients, etc.? How many Tomcat nodes? Tomcat connector configuration? Usually, this happens because you are overwhelming your Tomcat nodes with connections from the web server. Remember that every Tomcat node needs to be able to handle the maximum number connections from *all web servers simultaneously*. That number is probably way more than your Tomcat connector is configured to handle. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlueo+oACgkQHPApP6U8 pFhwVw/+M48YAmMzOBmHelQyzyFQFOZj5hEy/bUVyDmTgJ6xWfiRxSgJRViRK6lW DxAs1YFdLdam/RZg9uraE/KKc01zAtNKPAbr134Vkz4/gVMbCs1Ndc1rlvb2Etjb dOX7JajPOgLZa8m0hIu4KitkhbkXBJmqT2zQOmLF1KZmyNmaxkgknQux3K5mn5P9 p/ml2N9FTaxh1PYVnQWPKbLlwBBRlxHrTMqeLLpy5rZh8x41sSCdNQI2eLOopWAZ 1bRkOFMMCdT66yp5VcPTMNCKBL98GWsewTsRhkO5EB54IoL/x/uIYkx7BuHpU/C4 lt9dSYECqzp1HZJBkJKAUxsGRCH2hcTQJ75g+X4QfhCFv6ZKQTy+j8cE6nvwXk5D T9krmy9flmKccmFq+EPwt8mIcC0aZQ9TRmCBW5+HRQh/LTf4BexA0pDq4Yi2zxc2 9PkZSaTGkFLUGpLDH5qIwNhH78LhHT1BjwWxkeupgLAscEIb1znjZhwiJ87zd/7/ HHKXttaIUJ2E4s1zptkg1m8dNuLS7GCv/GuO1TER2Nk/4BtIrHE1QojNZj0Logll B65vvtHS7UdIarfF1DU1Jq7mh+fJ7PZseFgPI2WukwhUndNqo+AZ8ipnJEEAKklS hxsPGW0vuRzyqXV0IHB7rQMwAEArnq8yEi0Mwj7WagPQmKrxDdE= =7/Kx -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
jk_handler::mod_jk.c (2917): Could not get endpoint for worker ...
Hi all, we are seeing quite a few: "[Mon Sep 10 15:19:46 2018] [27562:140532026529536] [error] jk_handler::mod_jk.c (2917): Could not get endpoint for worker=testAPJ" errors in our md_jk.log. Worker properties are as follwos: ... worker.list=testAPJ worker.testAPJ.port=8009 worker.testAPJ.host=127.0.0.1 worker.testAPJ.type=ajp13 worker.testAPJ.socket_keepalive=1 worker.testAJP.connection_pool_timeout=600 ... At that point Apache seems to be stuck/struggling (but our tomcat does not seem to be under pressure). Restarting Apache solves the issue ... till it pops up again ... What is happening? What needs tob e tuned? Apache 2.4.34, tried both event- and worker-MPM Thx in advance Clemens
