On Tue, Mar 10, 2015 at 11:48 AM, Roland RoLaNd wrote:
> Hello,
>
> i am running haproxy version: 1.5.11 on EC2 instances behind an AWS load
> balancer
>
> lately i am noticing a lot of 503 forbidden logs with "SC" as termination
> state due to "nosrv" error
>
> my backend servers(which are behind an ELB of their own) are all healthy and
> responsive
>
> moreover i set a loop that checks port 80 between haproxy and backend
> servers; and it never failed; it was checking the connection every 10 ms
>
> this is a log sample:
>
> Mar 10 10:33:50 api haproxy[1056]: 172.16.100.169:15235
> [10/Mar/2015:10:33:50.905] API API/ 8/-1/-1/-1/8 503 213 - - SC--
> 79/79/0/0/0 0/0 {177.103.215.19|Dalvik/1.6.0 (Linux; U; Android 4.4.4;
> XT1032 Build/KXB21.14-L1.} "POST /api/v2.3/androidevent?buildnumber=1.10
> HTTP/1.1"
>
>
> and this is my current config:
>
> global
> log /dev/loglocal0
> log /dev/loglocal1 notice
> chroot /var/lib/haproxy
> stats socket /run/haproxy/admin.sock mode 660 level admin
> stats timeout 30s
> user haproxy
> group haproxy
> maxconn 65000
> daemon
>
> # Default SSL material locations
> ca-base /etc/ssl/certs
> crt-base /etc/ssl/private
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL).
> ssl-default-bind-ciphers
> kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
> ssl-default-bind-options no-sslv3
>
> defaults
> log global
> modehttp
> option httplog
> option dontlognull
> timeout connect 1
> timeout client 5
> timeout server 5
> # users which we are redrecting no where, example rejected will die
> in 50 ms
> timeout tarpit 50
> errorfile 400 /etc/haproxy/errors/400.http
> errorfile 403 /etc/haproxy/errors/403.http
> errorfile 408 /etc/haproxy/errors/408.http
> errorfile 500 /etc/haproxy/errors/500.http
> errorfile 502 /etc/haproxy/errors/502.http
> errorfile 503 /etc/haproxy/errors/503.http
> errorfile 504 /etc/haproxy/errors/504.http
> balance roundrobin
> # keeps keep alive between client and proxy but disable it between
> proxy and backedn
> option http-server-close
> option forwardfor
> option redispatch
>retries 99
>
> frontend API
> bind *:80
>
>
> maxconn 6
> # Blacklist: Deny access to some IPs before anything else is
> checked
> tcp-request content reject if { src -f /etc/haproxy/blacklist.lst }
> http-request set-header X-custom-http-scheme
> %[hdr(X-Forwarded-Proto)]
>
>
> stick-table type ip size 500k expire 30s store
> conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)
>
>
> option http-server-close
> # elb logs pubc ips
> capture request header X-Forwarded-For len 50
> capture request header User-Agent len 64
> acl network_allowed src x.x.x.x
> acl restricted_page path_beg /restricted
> http-request deny if restricted_page !network_allowed
> # direct uris to propper elb
> acl uri_api path_beg /api
> acl uri_wdev path_beg /wdev
> acl uri_staging path_beg /staging
>
> use_backend api if uri_api
> use_backend wdev if uri_wdev
> use_backend staging if uri_staging
>
>
>
> default_backend API
>
> backend API
> server API ELB_CNAME:80 check
> backend wdev
> server wdev ELB_CNAME:80 check
> backend staging
> server staging ELB_CNAME:80 check
>
>
>
>
>
Hi Roland,
This is by ELB design... It can change its IP address based on the load...
When this arrives, the only workaround is to reload HAProxy.
Soon, HAProxy will perform DNS resolution to kept updated on the fly
of server IP address changes.
Baptiste
