According to the mongrel_cluster.yml file you provided, you're starting only 14 mongrels, but in your proxy balancer config you have 30 mongrels listed on ports 21000 - 21029. In this scenario, if apache tries to proxy to any ports higher than 21014 then you'll get a proxy error as a mongrel doesn't exist on that port.
Or am I missing something? -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: 18 October 2007 01:36 To: [email protected] Subject: Mongrel-users Digest, Vol 21, Issue 16 Send Mongrel-users mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://rubyforge.org/mailman/listinfo/mongrel-users or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than "Re: Contents of Mongrel-users digest..." Today's Topics: 1. proxy errors with apache2.2.3 + mongrels (Jacobo Garcia) 2. Re: proxy errors with apache2.2.3 + mongrels (Robert Mela) 3. Re: proxy errors with apache2.2.3 + mongrels (Jacobo Garcia) 4. Re: proxy errors with apache2.2.3 + mongrels (Robert Mela) 5. Re: proxy errors with apache2.2.3 + mongrels (Jacobo Garcia) 6. Re: proxy errors with apache2.2.3 + mongrels (Rafael G.) 7. Re: proxy errors with apache2.2.3 + mongrels (Jacobo Garcia) ---------------------------------------------------------------------- Message: 1 Date: Wed, 17 Oct 2007 10:45:08 +0200 From: Jacobo Garcia <[EMAIL PROTECTED]> Subject: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=utf-8 I've posted this to rails-deployment as well. I have to administer a medium size rails app (1'5 million requests each day), recently I've switched from lighttpd + fcgi to apache + mongrel. In the following lines I am going to describe the platform: All machines are running Debian Etch, with 4 gb ram and dual core intel32 processors. Web server runs debian's apache 2.2.3-4 mpm worker package and also NFS server, Two app servers runs 14 mongrels ach one. There is two db servers, master running mysql5 (debian package) with innodb and an slave server with myisam tables. This slave server has 8 gb ram and has a memcached with 4 gb reserved. We are experiencing problems in peak hours with timeout and slow navigation on pages, also there are problems with some file uploads. I'm investigating on improving quality of service on the site, and I've reached to some proxy errors in apache log, with more than 300 simultaneous stablished connections on the webserver[1]. There are 4 kinds of proxy errors: [Wed Oct 17 06:24:14 2007] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 192.168.10.154:21004 (bomer) failed [Tue Oct 16 23:28:01 2007] [error] [client 67.142.130.19] proxy: error reading status line from remote server bomber, referer: http://www.google.com/search?hl=es&q=tipos+de+herramintas++++&lr= [Wed Oct 17 07:14:11 2007] [error] (70007)The timeout specified has expired: proxy: prefetch request body failed to 192.168.10.154:21013 (kgb) from 200.66.13 3.30 () [Wed Oct 17 07:16:54 2007] [error] [client 201.141.199.86] proxy: Error reading from remote server returned by /myfiles/clubsx/Sin t?? tulo 007_0001(1).jpg, r eferer: http://www.espacioblog.com/clubsx/post/2006/06/04/los-archivos-chiwasnaked With slow or moderated traffic mongrels works well (they use cpu and appears in first lines of 'top' command) but with heavy traffic, mongrels go down. I am looking for help with solving this issues, I've been looking in google about proxy errors, but what I've found haven't helped, I can provide more details if they can help. Also I've been suggested to use nginx web server instead of apache, but I am reluctant to use it, unless proven better than apache, can anyone point me to technical argument for using nginx? [1] $ netstat -nat |grep ':80'| grep 'EST' | wc -l Thanks for your time. Jacobo Garc?a -- Posted via http://www.ruby-forum.com/. ------------------------------ Message: 2 Date: Wed, 17 Oct 2007 10:48:38 -0400 From: Robert Mela <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset="utf-8" "Error reading status line" means Mongrel closed the socket without sending any content back. IT WOULD BE REALLY HELPFUL IF MONGREL WOULD SEND BACK "503 Server Busy" WHEN IT'S BUSY. That way we would know immediately whether the num_processors limit had been reached or not. HUGE TIMESAVER FOR MONGREL USERS! ==== Next... Seeing your load balancer config, and mongrel_cluster.yml would help narrow things down & simplify the discussion. Also, add this to your log file config: %D ( request duration -) %{BALANCER_WORKER_NAME}e Request duration, in combination with start time and balancer_worker_name, is useful in finding cases where requests to a back end app server overlap. These numbers will also allow you to construct a histogram of concurrent requests each second. thanks. Jacobo Garcia wrote: > I've posted this to rails-deployment as well. > > I have to administer a medium size rails app (1'5 million requests > each day), recently I've switched from lighttpd + fcgi to apache + > mongrel. In the following lines I am going to describe the platform: > > All machines are running Debian Etch, with 4 gb ram and dual core > intel32 processors. Web server runs debian's apache 2.2.3-4 mpm worker > package and also NFS server, Two app servers runs 14 mongrels ach > one. There is two db servers, master running mysql5 (debian package) > with innodb and an slave server with myisam tables. This slave server > has 8 gb ram and has a memcached with 4 gb reserved. > > We are experiencing problems in peak hours with timeout and slow > navigation on pages, also there are problems with some file uploads. > I'm investigating on improving quality of service on the site, and > I've reached to some proxy errors in apache log, with more than 300 > simultaneous stablished connections on the webserver[1]. > > There are 4 kinds of proxy errors: > > [Wed Oct 17 06:24:14 2007] [error] (111)Connection refused: proxy: > HTTP: attempt to connect to 192.168.10.154:21004 (bomer) failed > > [Tue Oct 16 23:28:01 2007] [error] [client 67.142.130.19] proxy: error > reading status line from remote server bomber, referer: > http://www.google.com/search?hl=es&q=tipos+de+herramintas++++&lr= > > [Wed Oct 17 07:14:11 2007] [error] (70007)The timeout specified has > expired: proxy: prefetch request body failed to 192.168.10.154:21013 > (kgb) from 200.66.13 > 3.30 () > > [Wed Oct 17 07:16:54 2007] [error] [client 201.141.199.86] proxy: > Error reading from remote server returned by /myfiles/clubsx/Sin t?? > tulo 007_0001(1).jpg, r > eferer: > http://www.espacioblog.com/clubsx/post/2006/06/04/los-archivos-chiwasnaked > > With slow or moderated traffic mongrels works well (they use cpu and > appears in first lines of 'top' command) but with heavy traffic, > mongrels go down. > > I am looking for help with solving this issues, I've been looking in > google about proxy errors, but what I've found haven't helped, I can > provide more details if they can help. > > Also I've been suggested to use nginx web server instead of apache, > but I am reluctant to use it, unless proven better than apache, can > anyone point me to technical argument for using nginx? > > [1] $ netstat -nat |grep ':80'| grep 'EST' | wc -l > > Thanks for your time. > > Jacobo Garc?a > -------------- next part -------------- A non-text attachment was scrubbed... Name: rob.vcf Type: text/x-vcard Size: 123 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20071017/6aaf9e30/a ttachment-0001.vcf ------------------------------ Message: 3 Date: Wed, 17 Oct 2007 17:36:34 +0200 From: Jacobo Garcia <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=utf-8 mongrel_cluster.yml: --- port: "21000" docroot: /var/www/mysite/current/public/ cwd: /var/www/mysite/current pid_file: /var/www/mysite/tmp/pids/mongrel.pid log_file: /var/www/mysite/current/log/mongrel.log environment: production servers: 14 user: deploys group: www-data proxy balancer: ServerName my.ip.add.ress NameVirtualHost *:80 <Proxy balancer://mongrel_cluster> BalancerMember http://bomber:21000 keepalive=on max=1 retry=30 BalancerMember http://bomber:21001 keepalive=on max=1 retry=30 BalancerMember http://bomber:21002 keepalive=on max=1 retry=30 BalancerMember http://bomber:21003 keepalive=on max=1 retry=30 BalancerMember http://bomber:21004 keepalive=on max=1 retry=30 BalancerMember http://bomber:21005 keepalive=on max=1 retry=30 BalancerMember http://bomber:21006 keepalive=on max=1 retry=30 BalancerMember http://bomber:21007 keepalive=on max=1 retry=30 BalancerMember http://bomber:21008 keepalive=on max=1 retry=30 BalancerMember http://bomber:21009 keepalive=on max=1 retry=30 BalancerMember http://bomber:21010 keepalive=on max=1 retry=30 BalancerMember http://bomber:21011 keepalive=on max=1 retry=30 BalancerMember http://bomber:21012 keepalive=on max=1 retry=30 BalancerMember http://bomber:21013 keepalive=on max=1 retry=30 BalancerMember http://bomber:21014 keepalive=on max=1 retry=30 BalancerMember http://bomber:21015 keepalive=on max=1 retry=30 BalancerMember http://bomber:21016 keepalive=on max=1 retry=30 BalancerMember http://bomber:21017 keepalive=on max=1 retry=30 BalancerMember http://bomber:21018 keepalive=on max=1 retry=30 BalancerMember http://bomber:21019 keepalive=on max=1 retry=30 BalancerMember http://bomber:21020 keepalive=on max=1 retry=30 BalancerMember http://bomber:21021 keepalive=on max=1 retry=30 BalancerMember http://bomber:21022 keepalive=on max=1 retry=30 BalancerMember http://bomber:21023 keepalive=on max=1 retry=30 BalancerMember http://bomber:21024 keepalive=on max=1 retry=30 BalancerMember http://bomber:21025 keepalive=on max=1 retry=30 BalancerMember http://bomber:21026 keepalive=on max=1 retry=30 BalancerMember http://bomber:21027 keepalive=on max=1 retry=30 BalancerMember http://bomber:21028 keepalive=on max=1 retry=30 BalancerMember http://bomber:21029 keepalive=on max=1 retry=30 BalancerMember http://kgb:21000 keepalive=on max=1 retry=30 BalancerMember http://kgb:21001 keepalive=on max=1 retry=30 BalancerMember http://kgb:21002 keepalive=on max=1 retry=30 BalancerMember http://kgb:21003 keepalive=on max=1 retry=30 BalancerMember http://kgb:21004 keepalive=on max=1 retry=30 BalancerMember http://kgb:21005 keepalive=on max=1 retry=30 BalancerMember http://kgb:21006 keepalive=on max=1 retry=30 BalancerMember http://kgb:21007 keepalive=on max=1 retry=30 BalancerMember http://kgb:21008 keepalive=on max=1 retry=30 BalancerMember http://kgb:21009 keepalive=on max=1 retry=30 BalancerMember http://kgb:21010 keepalive=on max=1 retry=30 BalancerMember http://kgb:21011 keepalive=on max=1 retry=30 BalancerMember http://kgb:21012 keepalive=on max=1 retry=30 BalancerMember http://kgb:21013 keepalive=on max=1 retry=30 BalancerMember http://kgb:21014 keepalive=on max=1 retry=30 BalancerMember http://kgb:21015 keepalive=on max=1 retry=30 BalancerMember http://kgb:21016 keepalive=on max=1 retry=30 BalancerMember http://kgb:21017 keepalive=on max=1 retry=30 BalancerMember http://kgb:21018 keepalive=on max=1 retry=30 BalancerMember http://kgb:21019 keepalive=on max=1 retry=30 BalancerMember http://kgb:21020 keepalive=on max=1 retry=30 BalancerMember http://kgb:21021 keepalive=on max=1 retry=30 BalancerMember http://kgb:21022 keepalive=on max=1 retry=30 BalancerMember http://kgb:21023 keepalive=on max=1 retry=30 BalancerMember http://kgb:21024 keepalive=on max=1 retry=30 BalancerMember http://kgb:21025 keepalive=on max=1 retry=30 BalancerMember http://kgb:21026 keepalive=on max=1 retry=30 BalancerMember http://kgb:21027 keepalive=on max=1 retry=30 BalancerMember http://kgb:21028 keepalive=on max=1 retry=30 BalancerMember http://kgb:21029 keepalive=on max=1 retry=30 </Proxy> I've been added the log parameters you've suggested to apache, i'll put em in a future messages. thanks Robert Mela wrote: > "Error reading status line" means Mongrel closed the socket without > sending any content back. > > IT WOULD BE REALLY HELPFUL IF MONGREL WOULD SEND BACK "503 Server Busy" > WHEN IT'S BUSY. That way we would know immediately whether the > num_processors limit had been reached or not. HUGE TIMESAVER FOR > MONGREL USERS! > > ==== > > Next... > > Seeing your load balancer config, and mongrel_cluster.yml would help > narrow things down & simplify the discussion. > > Also, add this to your log file config: > > %D ( request duration -) > %{BALANCER_WORKER_NAME}e > > > Request duration, in combination with start time and > balancer_worker_name, is useful in finding cases where requests to a > back end app server overlap. These numbers will also allow you to > construct a histogram of concurrent requests each second. > > thanks. -- Posted via http://www.ruby-forum.com/. ------------------------------ Message: 4 Date: Wed, 17 Oct 2007 12:27:00 -0400 From: Robert Mela <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset="iso-8859-1" Another avenue to consider is everything in order on the systems involved? If Rails is chewing up lots of RAM you could be swapping... here's some diagnostics I usually run ( caveat: I'm not a sysadmin, nor do I play one on TV ): sar -rB 1 0 sar 1 0 sar -n EDEV 1 0 (network IO incl. errors) sar -d 1 0 ( look for excessive block device IO )? What sorts of things do other folks on this list look at? Jacobo Garcia wrote: > mongrel_cluster.yml: > > --- > port: "21000" > docroot: /var/www/mysite/current/public/ > cwd: /var/www/mysite/current > pid_file: /var/www/mysite/tmp/pids/mongrel.pid > log_file: /var/www/mysite/current/log/mongrel.log > environment: production > servers: 14 > user: deploys > group: www-data > > > proxy balancer: > ServerName my.ip.add.ress > NameVirtualHost *:80 > <Proxy balancer://mongrel_cluster> > BalancerMember http://bomber:21000 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21001 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21002 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21003 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21004 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21005 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21006 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21007 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21008 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21009 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21010 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21011 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21012 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21013 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21014 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21015 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21016 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21017 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21018 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21019 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21020 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21021 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21022 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21023 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21024 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21025 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21026 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21027 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21028 keepalive=on max=1 retry=30 > BalancerMember http://bomber:21029 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21000 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21001 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21002 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21003 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21004 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21005 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21006 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21007 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21008 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21009 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21010 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21011 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21012 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21013 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21014 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21015 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21016 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21017 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21018 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21019 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21020 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21021 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21022 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21023 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21024 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21025 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21026 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21027 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21028 keepalive=on max=1 retry=30 > BalancerMember http://kgb:21029 keepalive=on max=1 retry=30 > </Proxy> > > I've been added the log parameters you've suggested to apache, i'll put > em in a future messages. > > thanks > > Robert Mela wrote: > >> "Error reading status line" means Mongrel closed the socket without >> sending any content back. >> >> IT WOULD BE REALLY HELPFUL IF MONGREL WOULD SEND BACK "503 Server Busy" >> WHEN IT'S BUSY. That way we would know immediately whether the >> num_processors limit had been reached or not. HUGE TIMESAVER FOR >> MONGREL USERS! >> >> ==== >> >> Next... >> >> Seeing your load balancer config, and mongrel_cluster.yml would help >> narrow things down & simplify the discussion. >> >> Also, add this to your log file config: >> >> %D ( request duration -) >> %{BALANCER_WORKER_NAME}e >> >> >> Request duration, in combination with start time and >> balancer_worker_name, is useful in finding cases where requests to a >> back end app server overlap. These numbers will also allow you to >> construct a histogram of concurrent requests each second. >> >> thanks. >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: rob.vcf Type: text/x-vcard Size: 116 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20071017/f4c37bf9/a ttachment-0001.vcf ------------------------------ Message: 5 Date: Wed, 17 Oct 2007 18:50:20 +0200 From: Jacobo Garcia <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=utf-8 I use munin to monitor machines and all of them are in excelent shape :) Robert Mela wrote: > Another avenue to consider is everything in order on the systems > involved? If Rails is chewing up lots of RAM you could be swapping... > here's some diagnostics I usually run ( caveat: I'm not a sysadmin, nor > do I play one on TV ): > > sar -rB 1 0 > sar 1 0 > sar -n EDEV 1 0 (network IO incl. errors) > sar -d 1 0 ( look for excessive block device IO )? > > What sorts of things do other folks on this list look at? -- Posted via http://www.ruby-forum.com/. ------------------------------ Message: 6 Date: Thu, 18 Oct 2007 01:19:39 +0200 From: "Rafael G." <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset="utf-8" Jacobo Garcia escribi?: > [Tue Oct 16 23:28:01 2007] [error] [client 67.142.130.19] proxy: error > reading status line from remote server bomber, referer: > http://www.google.com/search?hl=es&q=tipos+de+herramintas++++&lr= > > We had this problem and solved it adding the following lines in our apache conf: #Fix for Apache bug 39499 SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 Regards! -------------- next part -------------- A non-text attachment was scrubbed... Name: rgo.vcf Type: text/x-vcard Size: 251 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20071018/159b6c99/a ttachment-0001.vcf ------------------------------ Message: 7 Date: Thu, 18 Oct 2007 01:36:06 +0200 From: Jacobo Garcia <[EMAIL PROTECTED]> Subject: Re: [Mongrel] proxy errors with apache2.2.3 + mongrels To: [email protected] Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=utf-8 Rafael Garc?a wrote: > Jacobo Garcia escribi?: >> [Tue Oct 16 23:28:01 2007] [error] [client 67.142.130.19] proxy: error >> reading status line from remote server bomber, referer: >> http://www.google.com/search?hl=es&q=tipos+de+herramintas++++&lr= >> >> > We had this problem and solved it adding the following lines in our > apache conf: > > #Fix for Apache bug 39499 > SetEnv force-proxy-request-1.0 1 > SetEnv proxy-nokeepalive 1 > > > Regards! I've already has this lines (in virtual host conf) that is different than proxy conf, should I put them in proxy conf? -- Posted via http://www.ruby-forum.com/. ------------------------------ _______________________________________________ Mongrel-users mailing list [email protected] http://rubyforge.org/mailman/listinfo/mongrel-users End of Mongrel-users Digest, Vol 21, Issue 16 ********************************************* _______________________________________________ Mongrel-users mailing list [email protected] http://rubyforge.org/mailman/listinfo/mongrel-users
