Hi Andrew: Thanks for you feedback, but I'm describing a very specific bug wherein the old haproxy will unlink the new haproxy's bound unix domain socket upon reload due to a race condition in the domain socket cleanup code if a listen overflow occurs while the graceful is in process.
On Wed, Apr 12, 2017 at 11:39 AM, Andrew Smalley <[email protected]> wrote: > HI James > > When you do a graceful reload of haproxy this is what happens. > > 1. the old process will accept no more connections and the stats page is > stopped and so is the socket > 2. a new haproxy instance is started where new clients get connected to, > and this has the live socket > 3. when the old haproxy instance has no more clients left it dies silently > leaving all the clients on the new haproxy instance. > > This is expected behavior as you want the first haproxy to die when the > last client leaves. > > > Regards > > Andrew Smalley > > Loadbalancer.org Ltd. > > > > On 12 April 2017 at 19:32, James Brown <[email protected]> wrote: > >> This just hit us again on a different set of load balancers... if there's >> a listen socket overflow on a domain socket during graceful, haproxy >> completely deletes the domain socket and becomes inaccessible. >> >> On Tue, Feb 21, 2017 at 6:47 PM, James Brown <[email protected]> wrote: >> >>> Under load, we're sometimes seeing a situation where HAProxy will >>> completely delete a bound unix domain socket after a reload. >>> >>> The "bad flow" looks something like the following: >>> >>> >>> - haproxy is running on pid A, bound to /var/run/domain.sock (via a >>> bind line in a frontend) >>> - we run `haproxy -sf A`, which starts a new haproxy on pid B >>> - pid B binds to /var/run/domain.sock.B >>> - pid B moves /var/run/domain.sock.B to /var/run/domain.sock (in >>> uxst_bind_listener) >>> - in the mean time, there are a zillion connections to >>> /var/run/domain.sock and pid B isn't started up yet; backlog is exhausted >>> - pid B signals pid A to shut down >>> - pid A runs the destroy_uxst_socket function and tries to connect >>> to /var/run/domain.sock to see if it's still in use. The connection fails >>> (because the backlog is full). Pid A unlinks /var/run/domain.sock. >>> Everything is sad forever now. >>> >>> I'm thinking about just commenting out the call to destroy_uxst_socket >>> since this is all on a tmpfs and we don't really care if spare sockets are >>> leaked when/if we change configuration in the future. Arguably, the >>> solution should be something where we don't overflow the listen socket at >>> all; I'm thinking about also binding to a TCP port on localhost and just >>> using that for the few seconds it takes to reload (since otherwise we run >>> out of ephemeral sockets to 127.0.0.1); it still seems wrong for haproxy to >>> unlink the socket, though. >>> >>> This has proven extremely irritating to reproduce (since it only occurs >>> if there's enough load to fill up the backlog on the socket between when >>> pid B starts up and when pid A shuts down), but I'm pretty confident that >>> what I described above is happening, since periodically on reloads the >>> domain socket isn't there and this code fits. >>> >>> Our configs are quite large, so I'm not reproducing them here. The >>> reason we bind on a domain socket at all is because we're running two sets >>> of haproxies — one in multi-process mode doing TCP-mode SSL termination >>> pointing back over a domain socket to a single-process haproxy applying all >>> of our actual config. >>> >>> -- >>> James Brown >>> Systems >>> Engineer >>> >> >> >> >> -- >> James Brown >> Engineer >> > > -- James Brown Engineer
