Has nobody else run into this w/ consul? Given the plethora of tools around consul and haproxy and templating, I know others are using reloads to keep backend current, but the old haproxy PIDs stick around listening w/ incorrect backends.
On Thu, Jan 28, 2016 at 8:52 PM David Birdsong <[email protected]> wrote: > On Thu, Jan 28, 2016 at 6:35 PM, Pavlos Parissis < > [email protected]> wrote: > >> On 28/01/2016 10:35 μμ, David Birdsong wrote: >> > I've been running into a problem for a few weeks that I was hoping to >> > see disappear w/ a simple upgrade to 1.6.3. >> > >> > I'm using consul and it's templating to dynamically expand a backend >> > list which then runs an haproxy reload using the init scripts in the >> > contrib dir. >> > >> > I haven't been able to trace how the situation is triggered,but >> > basically I find haproxy processes that are still listening on their >> > bound sockets long after having received a reload signal via the '-sf' >> > parameter. >> > >> > My first fix was to ensure that reloads weren't happening too fast and >> > potentially stomping on the pid file which would explain a process never >> > getting a signal--so I now have a lock + sleeps for the automatic reload >> > script. No change. >> > >> > What's probably most interesting is that a SIGKILL is necessary to >> > remove the 'stale' processes that are still listening on their sockets >> > (as determined by lsof.) >> > >> > I can supply the config if needed, there is a single listener w/ a pem >> > directory specified if that's helpful. >> > >> >> The previous processes don't die that fast because there are connections >> still alive on them. Which is caused by very long timeout settings. >> > > The haproxy instance has many tcp-mode connections, and so, yes, the > processes often hang around for many days. I'm familiar with haproxy's > exiting when all connections have finished. > > What I'm looking for help about is the listen socket remaining. lsof > indicates a listener on the port and so accepts new connections on the > 'stale' process and configuration file. > > >> >> The behavior you see is normal. >> >> Cheers, >> Pavlos >> >> >> >>
