Cannot bind to source address before connect() for server

[Simon Yeo]

Has anyone see this error in haproxy logs?

Somewhere between 65k and 70k sessions, haproxy starts spewing the
error and things start dying from that point.  CPU(s) get pegged at
100%, healthchecks start failing, etc.

Some relevant information:
-- incoming bandwith is no more than 50Mbps at 65k sessions
-- Centos 5.2 (kernel 2.6.18-92.1.22.el5)
-- haproxy-1.3.14.11

Config details:
        maxconn 400000
        mode http
        balance roundrobin
        option httpchk HEAD / HTTP/1.0
        option forwardfor except 127.0.0.1
        option httpclose
each server gets "check inter 3s source [ip-address]", ip-address varies

sysctl stuff:
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 400000
net.ipv4.tcp_synack_retries = 3
fs.file-max = 1048576
net.core.somaxconn = 400000
net.core.netdev_max_backlog = 400000
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_max_tw_buckets = 1440000

thanks in advance,
--simon