Hello, I'm seeing some inconsistent/strange behavior with HAProxy (1.5.14 and 1.6.1) not being able to bind to a socket despite 'net.ipv4.ip_nonlocal_bind = 1' being set. HAProxy starts up without issue initially but after several reloads, the command 'service haproxy reload' starts failing and reports that HAProxy "cannot bind socket" for each of the listens/frontends, even for IPs that ARE on that server. The existing HAProxy process contiues to run without picking up the new changes.
After the reload initially fails all subsequent 'service haproxy reload' commands also fail. Running 'service haproxy restart' restarts and immediately binds to the IPs:ports specified in each listen/frontend that it just complained that it could not bind to. Here's some background info. There are two servers (lb-01 and lb-02). Virtual IPs are managed by keepalived (v1.2.19) in two vrrp_instances. Each vrrp_instance contains half of the total virtual IPs. The first vrrp_instance has lb-01 defined as MASTER and lb-02 as BACKUP and the second vrrp_instance has lb-02 defined as MASTER and lb-01 as BACKUP. This allows each server to act as failover for the other server. This was tested extensively while I was adding support for ip rules to keepalived and works without issue. All of HAProxy's configuration is stored in consul (v0.5.2). consul-template (v0.11.1) writes out /etc/haproxy/haproxy.cfg using the data in consul and then consul-template calls 'service haproxy reload'. The OS is CentOS 6.4 and the kernel version is 2.6.32-358.23.2.el6.x86_64. Here is an example of what I'm seeing (actual IPs have been substituted). 192.168.10.0/24 IPs are assigned to eth0 and 192.168.200.0/24 IPs are assigned to eth1. (output is from lb-02) Reloading haproxy: [ALERT] 301/141300 (25939) : Starting proxy haproxy-stats: cannot bind socket [192.168.10.27:80] [ALERT] 301/141300 (25939) : Starting proxy haproxy-fe1: cannot bind socket [192.168.200.100:80] [ALERT] 301/141300 (25939) : Starting proxy haproxy-fe2: cannot bind socket [192.168.200.120:80] [ALERT] 301/141300 (25939) : Starting proxy haproxy-fe3: cannot bind socket [192.168.200.110:80] What's strange is that HAProxy is already listening to these IPs:port so it seems to be some kind of race condition. Of these IPs, 192.168.10.27 is statically assigned to eth0 and is the only IP assigned to that interface. 192.168.200.110 and 192.168.200.120 are assigned to eth1 on lb-02. 192.168.200.100 is assigned to eth1 on lb-01. Without setting 'net.ipv4.ip_nonlocal_bind = 1' I would expect to see "cannot bind socket" for 192.168.200.100 but it doesn't make any sense that HAProxy also reports that it cannot bind on IPs:ports that are assigned to that server. Does anyone have ideas as to why this might occur? Best Regards, Chris Riley
