AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
redirectPort=8444 request.useSecret=true request.secret=x maxThreads=1 tcpNoDelay=true / Greetings Alexander -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Montag, 1. Februar 2010 11:51 An: Tomcat Users List Betreff: Re: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable On 01/02/2010 07:40, Alexander Diedler wrote: 44 requests currently being processed, 1656 idle workers I doubt you have that many AJP threads available on Tomcat. AJP connector config in server.xml? mod_jk config in httpd.conf? Looks like my guess was right. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
Manager.xml Articlefinder ROOT.xml Probe.xml Manager.xml -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Montag, 1. Februar 2010 22:27 An: Tomcat Users List Betreff: Re: AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/1/2010 3:54 PM, André Warnier wrote: worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1 worker2 Should this not be : worker.loadbalancer.balance_workers=worker1,worker2 http://tomcat.apache.org/connectors-doc/reference/workers.html balance_workers-A comma separated list of workers that the load balancer need to manage. Good observation. I tried to look at the mod_jk code to figure out how the string parsing was done, and I have to admit that I can't understand any of that code. :( Certainly using a comma-separated list would help. I'd be interested in seeing the JkMount directives, too, and seeing if both nodes are getting any traffic. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktnRxoACgkQ9CaO5/Lv0PCs9ACgr8eqCglq7pp/miiT5FoIqevo vv8An38AXScIm52q8l0cgCjNzCCoOz4c =xUPB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
. -Ursprüngliche Nachricht- Von: Alexander Diedler [mailto:adied...@tecracer.de] Gesendet: Montag, 1. Februar 2010 08:40 An: Tomcat Users List Betreff: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable Hello, Server Version: Apache/2.2.14 (Win32) mod_jk/1.2.28 Server Built: Sep 28 2009 22:41:08 Server loaded APR Version: 1.3.9 Compiled with APR Version: 1.3.9 Server loaded APU Version: 1.3.9 Compiled with APU Version: 1.3.9 Module Magic Number: 20051115:23 Hostname/port: XXX.XXX.XXX.XXX:80 Timeouts: connection: 300keep-alive: 5 MPM Name: WinNT MPM Information: Max Daemons: 64 Threaded: yes Forked: no Server Architecture: 32-bit Server Root: D:/tecracer/software/Apache Config File: D:/tecracer/software/Apache/conf/httpd.conf Server Built With: -D APACHE_MPM_DIR=server/mpm/winnt -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT=/apache -D SUEXEC_BIN=/apache/bin/suexec -D DEFAULT_ERRORLOG=logs/error.log -D AP_TYPES_CONFIG_FILE=conf/mime.types -D SERVER_CONFIG_FILE=conf/httpd.conf Server Version: Apache/2.2.14 (Win32) mod_jk/1.2.28 Server Built: Sep 28 2009 22:41:08 Current Time: Monday, 01-Feb-2010 08:38:14 W. Europe Standard Time Restart Time: Monday, 01-Feb-2010 04:58:09 W. Europe Standard Time Parent Server Generation: 0 Server uptime: 3 hours 40 minutes 5 seconds Total accesses: 28126 - Total Traffic: 340.1 MB 2.13 requests/sec - 26.4 kB/second - 12.4 kB/request 44 requests currently being processed, 1656 idle workers Listing Load Balancing Worker (1 Worker) [Hide] [S|E|R] Worker Status for loadbalancer Type Sticky Sessions Force Sticky Sessions Retries LB Method Locking Recover Wait Time Error Escalation Time Max Reply Timeouts [Hide] lb True False 3 Busyness Optimistic 60 30 0 Good Degraded Bad/Stopped Busy Max Busy Next Maintenance Last Reset [Hide] 2 0 0 0 70 74/196 13277 Balancer Members [Hide] Name Type Hostname Address:Port Connection Pool Timeout Connect Timeout Prepost Timeout Reply Timeout Retries Recovery Options Max Packet Size [Hide] worker1 ajp13 k0.domain.local xxx.xxx.x.78:8010 0 0 0 0 2 0 8192 worker2 ajp13 k1.domain.local xxx.xxx.x.77:8010 0 0 0 0 2 0 8192 Name Act State D F M V Acc Err CE RE Wr Rd Busy Max Route RR Cd Rs LR LE [S|E|R] worker1 ACT OK 0 100 1 0 425 (1/sec) 0 0 0 264K (908 /sec) 1.6M (5.3K/sec) 0 5 worker1 0/0 298 [S|E|R] worker2 ACT OK 0 100 1 0 2257 (0/sec) 112 24 0 1.3M (104 /sec) 17M (1.3K/sec) 0 11 worker2 0/0 13277 Greetings Alexander Diedler -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Sonntag, 31. Januar 2010 18:55 An: Tomcat Users List Betreff: Re: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable On 31/01/2010 17:50, Alexander Diedler wrote: Hello, We spend a lot of time to investigate the mistake, but I would not be found by us. We have two Virtual Machines with Windows 2003 R2 Standard Edition and on every node there is a Apache 2.2.14 and Tomcat 6.0.20 (and tcnative installed) with mod_jk 1.2.28 (all 32 Bit) installed. After a period (hours up to a day) we receive on every application a 503 error through the Webserver. The Apache seems to work fine, jkmanager, htdocs, server-info and server-status can be opened and the apps seems can be accessed by HTTP Connector Port 8080. Is there are any idea, what is the problem in this case? Any suggestions, what we can improve to avoid this errors. Hard to tell with out any configuration info. At a guess, more httpd threads than AJP threads and unused connections not being closed quickly enough. Try fixing your timeouts or use JkOptions +DisableReuse Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
On 01/02/2010 07:40, Alexander Diedler wrote: 44 requests currently being processed, 1656 idle workers I doubt you have that many AJP threads available on Tomcat. AJP connector config in server.xml? mod_jk config in httpd.conf? Looks like my guess was right. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
Hi Mark, I attach my current settings: In httpd.conf: ThreadLimit 1920 ThreadsPerChild 1920 MaxRequestsPerChild 1 Worker.properties worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1 worker2 worker.loadbalancer.sticky_session=true worker.loadbalancer.sticky_session_force=false worker.loadbalancer.method=Busyness worker.loadbalancer.retries=4 worker.loadbalancer.secret= server.xml: Connector port=8010 protocol=AJP/1.3 connectionTimeout=3 redirectPort=8444 request.useSecret=true request.secret=X maxThreads=1 tcpNoDelay=true / Cluster className=org.apache.catalina.ha.tcp.SimpleTcpCluster channelSendOptions=8 Manager className=org.apache.catalina.ha.session.DeltaManager expireSessionsOnShutdown=false notifyListenersOnReplication=true/ Channel className=org.apache.catalina.tribes.group.GroupChannel Membership className=org.apache.catalina.tribes.membership.McastService address=228.0.0.4 port=45564 frequency=500 dropTime=3000/ Receiver className=org.apache.catalina.tribes.transport.nio.NioReceiver address=auto port=4000 autoBind=100 selectorTimeout=5000 maxThreads=6/ Sender className=org.apache.catalina.tribes.transport.ReplicationTransmitter Transport className=org.apache.catalina.tribes.transport.nio.PooledParallelSender/ /Sender Interceptor className=org.apache.catalina.tribes.group.interceptors.TcpFailureDetector/ Interceptor className=org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor/ /Channel Valve className=org.apache.catalina.ha.tcp.ReplicationValve filter=/ Valve className=org.apache.catalina.ha.session.JvmRouteBinderValve/ ClusterListener className=org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener/ ClusterListener className=org.apache.catalina.ha.session.ClusterSessionListener/ /Cluster -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Montag, 1. Februar 2010 11:51 An: Tomcat Users List Betreff: Re: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable On 01/02/2010 07:40, Alexander Diedler wrote: 44 requests currently being processed, 1656 idle workers I doubt you have that many AJP threads available on Tomcat. AJP connector config in server.xml? mod_jk config in httpd.conf? Looks like my guess was right. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexander, I took at look at your log file, and everything looks good to me, honestly. There are no errors that I can see, other than the setsockopt(SO_UPDATE_ACCEPT_CONTEXT) warning that apparently accompanies every child process shutdown.\ On 2/1/2010 6:17 AM, Alexander Diedler wrote: In httpd.conf: ThreadLimit 1920 ThreadsPerChild 1920 MaxRequestsPerChild 1 Okay, but this doesn't really say anything about your AJP connections, other than you might expect to see 1920 concurrent requests that might be sent to Tomcat. worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1 worker2 Great. Where are the settings for worker1 and worker2? These can be quite important. Essential, some might say. server.xml: Connectorport=8010 protocol=AJP/1.3 connectionTimeout=3 redirectPort=8444 request.useSecret=true request.secret=X maxThreads=1 tcpNoDelay=true / Okay, so you have 1 threads per server, and 2 app servers. That's max 2 threads configured for a maximum load of 3040 possible concurrent incoming connections. I would review all those settings at some point. It doesn't looks like mod_jk is allocating too many connections: 44 requests currently being processed, 1656 idle workers ...and Tomcat won't allocate a bunch of threads until it needs them, so it never will. So, this isn't really a problem... just an indication of sloppy resource management decisions. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktnHAEACgkQ9CaO5/Lv0PAWEwCdFYP+d6hbDEu0KlGc4wVDGUmJ 7aAAnjzkLPvLYt+uCkQ56tlh0uEvJfaa =u4Xi -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
Alexander Diedler wrote: Hello, Interesting, MaxRequestsPerChild limit is to low. How I should set the limit? Unless you have an application which leaks memory, you could try setting it to zero (= unlimited) Or, since you appear to be under Windows, just comment it out, since 0 is the default. http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1 worker2 Should this not be : worker.loadbalancer.balance_workers=worker1,worker2 http://tomcat.apache.org/connectors-doc/reference/workers.html balance_workers - A comma separated list of workers that the load balancer need to manage. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/1/2010 3:54 PM, André Warnier wrote: worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1 worker2 Should this not be : worker.loadbalancer.balance_workers=worker1,worker2 http://tomcat.apache.org/connectors-doc/reference/workers.html balance_workers-A comma separated list of workers that the load balancer need to manage. Good observation. I tried to look at the mod_jk code to figure out how the string parsing was done, and I have to admit that I can't understand any of that code. :( Certainly using a comma-separated list would help. I'd be interested in seeing the JkMount directives, too, and seeing if both nodes are getting any traffic. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktnRxoACgkQ9CaO5/Lv0PCs9ACgr8eqCglq7pp/miiT5FoIqevo vv8An38AXScIm52q8l0cgCjNzCCoOz4c =xUPB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable
Hello, Server Version: Apache/2.2.14 (Win32) mod_jk/1.2.28 Server Built: Sep 28 2009 22:41:08 Server loaded APR Version: 1.3.9 Compiled with APR Version: 1.3.9 Server loaded APU Version: 1.3.9 Compiled with APU Version: 1.3.9 Module Magic Number: 20051115:23 Hostname/port: XXX.XXX.XXX.XXX:80 Timeouts: connection: 300keep-alive: 5 MPM Name: WinNT MPM Information: Max Daemons: 64 Threaded: yes Forked: no Server Architecture: 32-bit Server Root: D:/tecracer/software/Apache Config File: D:/tecracer/software/Apache/conf/httpd.conf Server Built With: -D APACHE_MPM_DIR=server/mpm/winnt -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT=/apache -D SUEXEC_BIN=/apache/bin/suexec -D DEFAULT_ERRORLOG=logs/error.log -D AP_TYPES_CONFIG_FILE=conf/mime.types -D SERVER_CONFIG_FILE=conf/httpd.conf Server Version: Apache/2.2.14 (Win32) mod_jk/1.2.28 Server Built: Sep 28 2009 22:41:08 Current Time: Monday, 01-Feb-2010 08:38:14 W. Europe Standard Time Restart Time: Monday, 01-Feb-2010 04:58:09 W. Europe Standard Time Parent Server Generation: 0 Server uptime: 3 hours 40 minutes 5 seconds Total accesses: 28126 - Total Traffic: 340.1 MB 2.13 requests/sec - 26.4 kB/second - 12.4 kB/request 44 requests currently being processed, 1656 idle workers Listing Load Balancing Worker (1 Worker) [Hide] [S|E|R] Worker Status for loadbalancer Type Sticky Sessions Force Sticky Sessions Retries LB Method Locking Recover Wait Time Error Escalation Time Max Reply Timeouts [Hide] lb True False 3 Busyness Optimistic 60 30 0 Good Degraded Bad/Stopped Busy Max Busy Next Maintenance Last Reset [Hide] 2 0 0 0 70 74/196 13277 Balancer Members [Hide] Name Type Hostname Address:Port Connection Pool Timeout Connect Timeout Prepost Timeout Reply Timeout Retries Recovery Options Max Packet Size [Hide] worker1 ajp13 k0.domain.local xxx.xxx.x.78:8010 0 0 0 0 2 0 8192 worker2 ajp13 k1.domain.local xxx.xxx.x.77:8010 0 0 0 0 2 0 8192 Name Act State D F M V Acc Err CE RE Wr Rd Busy Max Route RR Cd Rs LR LE [S|E|R] worker1 ACT OK 0 100 1 0 425 (1/sec) 0 0 0 264K (908 /sec) 1.6M (5.3K/sec) 0 5 worker1 0/0 298 [S|E|R] worker2 ACT OK 0 100 1 0 2257 (0/sec) 112 24 0 1.3M (104 /sec) 17M (1.3K/sec) 0 11 worker2 0/0 13277 Greetings Alexander Diedler -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Sonntag, 31. Januar 2010 18:55 An: Tomcat Users List Betreff: Re: Tomcat Cluster 6.x and HTTP/1.1 503 Service Temporarily Unavailable On 31/01/2010 17:50, Alexander Diedler wrote: Hello, We spend a lot of time to investigate the mistake, but I would not be found by us. We have two Virtual Machines with Windows 2003 R2 Standard Edition and on every node there is a Apache 2.2.14 and Tomcat 6.0.20 (and tcnative installed) with mod_jk 1.2.28 (all 32 Bit) installed. After a period (hours up to a day) we receive on every application a 503 error through the Webserver. The Apache seems to work fine, jkmanager, htdocs, server-info and server-status can be opened and the apps seems can be accessed by HTTP Connector Port 8080. Is there are any idea, what is the problem in this case? Any suggestions, what we can improve to avoid this errors. Hard to tell with out any configuration info. At a guess, more httpd threads than AJP threads and unused connections not being closed quickly enough. Try fixing your timeouts or use JkOptions +DisableReuse Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
