Hi, I am debugging a mod_jk load-balancing configuration which has been used a lot, but for two nodes only. Currently, we made a change for more nodes, and we are facing problem.
Original idea was to have one PROD and DR servers such that all requests are handled by PROD and if PROD goes down DR takes over. In order to do that we used activation=disabled for DR worker, such that: worker.list=jkstatus,lbhierarchy worker.jkstatus.type=status worker.lbhierarchy.type=lb worker.lbhierarchy.balance_workers=hierarchy-1,hierarchy-2 worker.hierarchy-1.type=ajp13 worker.hierarchy-1.host=host1 worker.hierarchy-1.port=8009 worker.hierarchy-1.socket_timeout=0 worker.hierarchy-1.socket_keepalive=False worker.hierarchy-1.retries=2 worker.hierarchy-1.connection_pool_timeout=0 worker.hierarchy-1.lbfactor=1 worker.hierarchy-1.redirect=hierarchy-2 worker.hierarchy-2.type=ajp13 worker.hierarchy-2.host=host2 worker.hierarchy-2.port=8009 worker.hierarchy-2.socket_timeout=0 worker.hierarchy-2.socket_keepalive=False worker.hierarchy-2.retries=2 worker.hierarchy-2.connection_pool_timeout=0 worker.hierarchy-2.lbfactor=1 worker.hierarchy-2.activation=disabled However, the current demand is to have four servers which are chained. Basically, we try to have one PROD and 3 DR servers. Each DR is activated when the previous worker goes down. Therefore, we prepared configuration like this: worker.list=jkstatus,lbhierarchy,lb2hierarchy worker.jkstatus.type=status worker.lbhierarchy.type=lb worker.lbhierarchy.balance_workers=hierarchy-1,hierarchy-2,hierarchy-3,hierarchy-4 worker.hierarchy-1.type=ajp13 worker.hierarchy-1.host=host worker.hierarchy-1.port=8009 worker.hierarchy-1.socket_timeout=0 worker.hierarchy-1.socket_keepalive=False worker.hierarchy-1.retries=2 worker.hierarchy-1.connection_pool_timeout=0 worker.hierarchy-1.lbfactor=1 worker.hierarchy-1.redirect=hierarchy-2 worker.hierarchy-2.type=ajp13 worker.hierarchy-2.host=host worker.hierarchy-2.port=8010 worker.hierarchy-2.socket_timeout=0 worker.hierarchy-2.socket_keepalive=False worker.hierarchy-2.retries=2 worker.hierarchy-2.connection_pool_timeout=0 worker.hierarchy-2.lbfactor=1 worker.hierarchy-2.activation=disabled worker.hierarchy-2.redirect=hierarchy-3 worker.hierarchy-3.type=ajp13 worker.hierarchy-3.host=host worker.hierarchy-3.port=8011 worker.hierarchy-3.socket_timeout=0 worker.hierarchy-3.socket_keepalive=False worker.hierarchy-3.retries=2 worker.hierarchy-3.connection_pool_timeout=0 worker.hierarchy-3.lbfactor=1 worker.hierarchy-3.activation=disabled worker.hierarchy-3.redirect=hierarchy-4 worker.hierarchy-4.type=ajp13 worker.hierarchy-4.host=host12 worker.hierarchy-4.port=10603 worker.hierarchy-4.socket_timeout=0 worker.hierarchy-4.socket_keepalive=False worker.hierarchy-4.retries=2 worker.hierarchy-4.connection_pool_timeout=0 worker.hierarchy-4.lbfactor=1 worker.hierarchy-4.activation=disabled worker.hierarchy-4.redirect=hierarchy-1 Initially, 3 servers are disabled and redirect is specified. Problem occurs when hierarchy-3 worker goes down because hierarchy-4 never gets initiated and mod_jk log says that: All tomcat instances failed, no more workers left However, workers list is like this: worker.lbhierarchy.balance_workers=hierarchy-1,hierarchy-2,hierarchy-3,hierarchy-4 Which means we have got more than 3 workers hence we have more workers left... Here is log where I tried to take down workers one by one: [Tue Oct 18 09:51:30.623 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] jk_open_socket::jk_connect.c (817): connect to HOST:8009 failed (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (hierarchy-1) Failed opening socket to (HOST:8009) (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [error] ajp_send_request::jk_ajp_common.c (1728): (hierarchy-1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-1) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [error] ajp_service::jk_ajp_common.c (2794): (hierarchy-1) connecting to tomcat failed (rc=-3, errors=5, client_errors=0). [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1595): service failed, worker hierarchy-1 is in error state [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] jk_open_socket::jk_connect.c (817): connect to HOST:8010 failed (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (hierarchy-2) Failed opening socket to (HOST:8010) (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [error] ajp_send_request::jk_ajp_common.c (1728): (hierarchy-2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Tue Oct 18 09:51:30.724 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1) [Tue Oct 18 09:51:30.824 2016] [31890:139909801125856] [info] jk_open_socket::jk_connect.c (817): connect to HOST:8010 failed (errno=111) [Tue Oct 18 09:51:30.824 2016] [31890:139909801125856] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (hierarchy-2) Failed opening socket to (HOST:8010) (errno=111) [Tue Oct 18 09:51:30.824 2016] [31890:139909801125856] [error] ajp_send_request::jk_ajp_common.c (1728): (hierarchy-2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Tue Oct 18 09:51:30.824 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2) [Tue Oct 18 09:51:30.824 2016] [31890:139909801125856] [error] ajp_service::jk_ajp_common.c (2794): (hierarchy-2) connecting to tomcat failed (rc=-3, errors=5, client_errors=0). [Tue Oct 18 09:51:30.825 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1595): service failed, worker hierarchy-2 is in error state [Tue Oct 18 09:51:30.825 2016] [31890:139909801125856] [info] jk_open_socket::jk_connect.c (817): connect to HOST:8011 failed (errno=111) [Tue Oct 18 09:51:30.825 2016] [31890:139909801125856] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (hierarchy-3) Failed opening socket to (HOST:8011) (errno=111) [Tue Oct 18 09:51:30.825 2016] [31890:139909801125856] [error] ajp_send_request::jk_ajp_common.c (1728): (hierarchy-3) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Tue Oct 18 09:51:30.825 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-3) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1) [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [info] jk_open_socket::jk_connect.c (817): connect to HOST:8011 failed (errno=111) [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1068): (hierarchy-3) Failed opening socket to (HOST:8011) (errno=111) [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [error] ajp_send_request::jk_ajp_common.c (1728): (hierarchy-3) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [info] ajp_service::jk_ajp_common.c (2773): (hierarchy-3) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2) [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [error] ajp_service::jk_ajp_common.c (2794): (hierarchy-3) connecting to tomcat failed (rc=-3, errors=5, client_errors=0). [Tue Oct 18 09:51:30.925 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1595): service failed, worker hierarchy-3 is in error state [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1664): All tomcat instances failed, no more workers left (attempt=0, retry=1) [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1664): All tomcat instances failed, no more workers left (attempt=1, retry=1) [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1664): All tomcat instances failed, no more workers left (attempt=2, retry=1) [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1664): All tomcat instances failed, no more workers left (attempt=3, retry=1) [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] service::jk_lb_worker.c (1675): All tomcat instances are busy or in error state [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [error] service::jk_lb_worker.c (1680): All tomcat instances failed, no more workers left [Tue Oct 18 09:51:31.026 2016] [31890:139909801125856] [info] jk_handler::mod_jk.c (2991): Service error=0 for worker=lbhierarchy I tested similar configuration for 5 workers, and I never got over worker 3.. We are using : Server Version: Apache/2.2.15 (Unix) mod_jk/1.2.41 JK Version: mod_jk/1.2.41 I don't know whether is this a bug or did I miss something in documentation? Thanks for a help! Miloš Kozák Application Operations AO Front Office Support Commerz Systems Prague ________________________________ Právní informace: Tento e-mail a všechny připojené soubory jsou důvěrné a mohou být chráněny zákonem a jsou určeny pouze oprávněným adresátům. Pokud jste obdržel/a tento e-mail omylem, oznamte to, prosím, neprodleně jeho odesílateli a pak jej vymažte. Tento e-mail ani připojené soubory nejsou návrhem na uzavření smlouvy ani jeho přijetím, ledaže jsou tak výslovně označeny. Commerzbank neručí za bezchybný a úplný přenos zasílaných informací, ani za zpoždění nebo přerušení přenosu a ani za škody způsobené použitím nebo důvěrou v tyto informace. Legal information: This e-mail message and all attached files are confidential, may be protected under the law, and are intended for authorized recipients only. If you have received this e-mail message in error, please notify the sender immediately and subsequently delete the message. This e-mail message and the attached files do not constitute a proposal to enter into an agreement or the acceptance of such a proposal, unless expressly designated as such. COMMERZBANK gives no guarantee that the transmitted information is error-free and complete. COMMERZBANK will not be held liable for any transmission delay or interruption, nor for any loss incurred due to the use of, or reliance on, the transmitted information. COMMERZBANK Aktiengesellschaft, pobočka Praha, se sídlem Praha 2, Jugoslávská 1, 120 21, IČO: 47 61 09 21, zapsaná v Obchodním rejstříku u Městského soudu v Praze, v oddílu A, vložka 7341, pobočka COMMERZBANK Aktiengesellschaft, Frankfurt nad Mohanem, Německo. Povinné údaje: http://www.commerzbank.de/pflichtangaben COMMERZBANK Aktiengesellschaft, pobočka Praha, with its registered office: Praha 2, Jugoslávská 1, 120 21, registered number: 47610921, entered in the Commercial Register at the Municipal Court Prague, Section A, Entry 7341, branch of COMMERZBANK Aktiengesellschaft, Frankfurt am Main, Germany. Mandatory information: http://www.commerzbank.com/pflichtangaben
