I think the gold standard for graceful restarts is nginx - it will start a new instance (could be a new binary), send the accept fd's to the new instance, then the original instance will stop accepting new requests and allow the existing connections to drain off. The whole process is controlled by signals and you can even decide there is a problem with the new instance and have the old one resume taking traffic. I love it because I can bounce nginx all day long and noone notices. I could see haproxy having the same ability when nbproc = 1, but not exactly a two weekend project.
On Mon, Apr 13, 2015 at 1:24 PM, Joseph Lynch <[email protected]> wrote: > Hello, > > I published an article today on Yelp's engineering blog ( > http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html) > that shows a technique we use for low latency, zero downtime restarts of > HAProxy. This solves the "when I restart HAProxy some of my clients get > RSTs" problems that can occur. We built it to solve the RSTs in our > internal load balancing, so there is a little more work to be done to > modify the method to work with external traffic, which I talk about in the > post. > > The solution basically consists of using Linux queuing disciplines to > delay SYN packets for the duration of the restart. It can definitely be > improved by further tuning the qdiscs or replacing the iptables mangle with > a u8/u32 tc filter, but I decided it was better to talk about the idea and > if the community likes it, then we can optimize it further. > > -Joey >
