Bug in 1.5-dev15, dev-14 and maybe lower?
Hi All, I recently upgraded to HAProxy dev-14 (and since this morning dev-15) from dev11-ss-20120604. But, now we are experiencing uploads that are 'hanging'. When uploading a file over HTTP the upload suddenly stalls. I cant get my finger on it why. Sometimes it is right after the upload starts, sometimes somewhere in the middle and (surprise, surprise!) sometimes almost at the end. After a while the upload continues again and finishes or stalls again. Uploads usually go with higher speeds 200-300mbit/s so HAProxy CPU usage goes up a bit (10-15% cpu usage). Is this a bug in HAProxy or is it my config? Downgrading to dev11-ss-20120604 fixes the issue. Greets, Sander Klein My config: ### # Global Settings ### global log 127.0.0.1 local0 # log 127.0.0.1 local0 notice # log 127.0.0.1 local0 err # log 127.0.0.1 local1 debug daemon userhaproxy group haproxy maxconn 32768 spread-checks 3 stats socket/var/run/haproxy.stat mode 666 level admin #debug #quiet ### # Defaults ### defaults log global timeout check 2s timeout client 60s timeout connect 10s timeout http-keep-alive 30s timeout http-request30s timeout queue 30s timeout server 60s timeout tarpit 120s errorfile 400 /etc/haproxy/errors.loc/400.http errorfile 403 /etc/haproxy/errors.loc/403.http errorfile 500 /etc/haproxy/errors.loc/500.http errorfile 502 /etc/haproxy/errors.loc/502.http errorfile 503 /etc/haproxy/errors.loc/503.http errorfile 504 /etc/haproxy/errors.loc/504.http ### # Define the admin section ### listen admin bindxxx.xxx.xxx.xxx:8080 # bind:::::xx:8080 modehttp stats enable stats uri /haproxy?stats stats auth admin:passwordhere! stats admin if TRUE stats refresh 5s ### # Mass hosting frontend ### frontend cluster1-in # Mass hosting VIP bind x.x.x.x:80 bind :::::xx:80 ... more bind stuff... mode http maxconn 4096 option httplog option dontlog-normal option dontlognull option forwardfor option http-server-close option splice-auto option tcp-smart-accept capture request header Host len 64 capture request header User-Agent len 16 capture request header Content-Length len 10 capture request header Referer len 256 capture response header Content-Length len 10 # # Some security stuff starts here # # block annoying worms that fill the logs... # deny NULL character, script tag and #removed xmlrpc.php#removed in URL's acl forbidden_uris url_sub -i %00 script # /../../ attacks acl forbidden_uris url_reg -i (%2f|%5c|/|)(\.|%2e)(\.|%2e)(%2f|%5c|/|) # Deny requests for following files: acl forbidden_uris path_end -i /root.exe /cmd.exe /default.ida /awstats.pl .dll # Deny script kiddy stuff eating our connections acl forbidden_uris url_sub -f /etc/haproxy/filters/phpmyadmin.txt block if forbidden_uris # HTTP content smugling acl forbidden_hdrs hdr_cnt(host) gt 1 acl forbidden_hdrs hdr_cnt(content-length) gt 1 acl forbidden_hdrs hdr_cnt(proxy-authorization) gt 0 block if forbidden_hdrs # Block offensive User-Agents acl offender hdr_sub(User-Agent) -i msnbot acl offender hdr_sub(User-Agent) -i baiduspider block if offender # Remove bogus X-Forwarded-For headers # We don't care about RFC1918 reqidel ^X-Forwarded-For:\ xxx\.xxx\.xxx ... more reqidel's like the above... # Add X-Forwarded-Proto headers acl no-ssl dst_port 80 reqadd X-Forwarded-Proto:\ http if no-ssl # Web cluster acl iscluster1-1 hdr(host) -f /etc/haproxy/cluster1-1.txt acl iscluster1-2 hdr(host) -f /etc/haproxy/cluster1-2.txt acl iscluster1-2 hdr_sub(host) -i some.domain acl iscluster1-2 hdr_sub(host) -i other.domain acl iscluster1-2 hdr_sub(host) -i another.domain use_backend cluster1-1if iscluster1-1 use_backend cluster1-2if iscluster1-2 default_backend cluster1-1 ### # 1 backend ### backend cluster1-1 fullconn4096 modehttp balance roundrobin option abortonclose option tcp-smart-connect option redispatch option httpchk GET /db.php HTTP/1.0 server
Re: Need your help for error message too many redirects
Enable logs in your HAProxy configuration, configure your syslog to collect them and in the end do a tail on your log file :) cheers On Wed, Dec 12, 2012 at 10:15 AM, Christophe Rahier christo...@qualifio.com wrote: Hi Baptiste, Just a stupid question but how can I access this log? Thanks, Christophe Le 12/12/12 07:40, « Baptiste » bed...@gmail.com a écrit : Hi, You seem to have an issue on your server side. Try logging the response Location header and compare it with the requested URL. Your server may be redirecting you to a URL that is still redirected to it. add capture response header Location len 64 in your manager frontend section and share your logs here. cheers On Wed, Dec 12, 2012 at 7:03 AM, Christophe Rahier christo...@qualifio.com wrote: Hi, I'd like to use haproxy with a certificate directly installed on this server. For this, I copied my key_andcert.pem. I use the same haproxy for our application (managers / players) with 2 different official IP. When I try to access to my Players, no problem. When I try to access to my Managers, Chrome returns the error: too many redirects. Please find below my config file. Thanks for your help and sorry for my poor English. Regards, Christophe global daemon maxconn 1 stats socket /var/run/haproxy.sock level admin mode 600 stats timeout 1m log 192.168.0.113 local0 log 127.0.0.1 local1 notice defaults logglobal option dontlognull retries2 timeout client 60s timeout server 150s timeout connect 5s timeout http-keep-alive 10s frontend Players-Farm mode http option forwardfor option httplog option http-server-close bind 192.168.0.113:443 ssl crt /etc/haproxy/key_andcert.pem bind 192.168.0.113:80 default_backend Players backend Players mode http server Player1 192.168.0.10:80 check on-marked-down shutdown-sessions server Player2 192.168.0.11:80 check on-marked-down shutdown-sessions server Player3 192.168.0.12:80 check on-marked-down shutdown-sessions option httpchk HEAD /cfadmin/ping.cfm HTTP/1.0 stats enable balance source frontend Managers-Farm mode http option forwardfor option httplog option http-server-close bind 192.168.0.113:444 ssl crt /etc/haproxy/key_andcert.pem bind 192.168.0.113:81 default_backend Managers backend Managers mode http server Manager01 192.168.0.60:80 check on-marked-down shutdown-sessions server Manager02 192.168.0.61:80 check on-marked-down shutdown-sessions option httpchk HEAD /cfadmin/ping.cfm HTTP/1.0 stats enable balance source listen info 192.168.0.113:90 mode http balance source stats uri /
Re: Bug in 1.5-dev15, dev-14 and maybe lower?
Hi Sander, On Wed, Dec 12, 2012 at 10:34:59AM +0100, Sander Klein wrote: Hi All, I recently upgraded to HAProxy dev-14 (and since this morning dev-15) from dev11-ss-20120604. But, now we are experiencing uploads that are 'hanging'. When uploading a file over HTTP the upload suddenly stalls. I cant get my finger on it why. Sometimes it is right after the upload starts, sometimes somewhere in the middle and (surprise, surprise!) sometimes almost at the end. After a while the upload continues again and finishes or stalls again. Uploads usually go with higher speeds 200-300mbit/s so HAProxy CPU usage goes up a bit (10-15% cpu usage). Is this a bug in HAProxy or is it my config? Downgrading to dev11-ss-20120604 fixes the issue. Sander, such a random behaviour cannot be caused by a config issue. So there is a bug, definitely. Maybe it's always been there and your config enlightens it, but there is no reason for a config to cause traffic pauses. Could you try to disable the splice options just to see ? And if that does not change anything, please also try to disable option abortonclose. That will help us narrow the issue down. Anyway, I don't see anything wrong with your config. If you can easily reproduce this, I'd be interested in getting a network traffic capture on the machine running haproxy, I don't know if you can get send this. Thanks, Willy
Re: HAProxy, high mysql request rate and TCP source port exhaustion
❦ 12 décembre 2012 14:45 CET, Baptiste bed...@gmail.com : Maybe some of you already experimented source port exhaustion. Here is a blog post giving some information about it: http://blog.exceliance.fr/2012/12/12/haproxy-high-mysql-request-rate-and-tcp-source-port-exhaustion/ Great post! But, you should stress more that that tcp_tw_recycle _must_ never be used on public-facing servers. The side effects are that some people behing the same NAT address won't be able to connect to the service. -- Make your program read from top to bottom. - The Elements of Programming Style (Kernighan Plauger)
