On Tue, Feb 2, 2016 at 7:09 PM Bryan Talbot <[email protected]> wrote:
> On Tue, Feb 2, 2016 at 3:56 PM, David Birdsong <[email protected]> > wrote: > >> Has nobody else run into this w/ consul? Given the plethora of tools >> around consul and haproxy and templating, I know others are using reloads >> to keep backend current, but the old haproxy PIDs stick around listening w/ >> incorrect backends. >> > > I'm not using consul but am using haproxy in a docker container and > reloading when backend hosts change registrations. I haven't seen this > issue. I run using haproxy-systemd-wrapper and HUP that process to reload. > does that mean the wrapper ensures that an old process exits and forces if it doesn't eventually? > > -Bryan > > > > >> >> On Thu, Jan 28, 2016 at 8:52 PM David Birdsong <[email protected]> >> wrote: >> >>> On Thu, Jan 28, 2016 at 6:35 PM, Pavlos Parissis < >>> [email protected]> wrote: >>> >>>> On 28/01/2016 10:35 μμ, David Birdsong wrote: >>>> > I've been running into a problem for a few weeks that I was hoping to >>>> > see disappear w/ a simple upgrade to 1.6.3. >>>> > >>>> > I'm using consul and it's templating to dynamically expand a backend >>>> > list which then runs an haproxy reload using the init scripts in the >>>> > contrib dir. >>>> > >>>> > I haven't been able to trace how the situation is triggered,but >>>> > basically I find haproxy processes that are still listening on their >>>> > bound sockets long after having received a reload signal via the >>>> '-sf' >>>> > parameter. >>>> > >>>> > My first fix was to ensure that reloads weren't happening too fast and >>>> > potentially stomping on the pid file which would explain a process >>>> never >>>> > getting a signal--so I now have a lock + sleeps for the automatic >>>> reload >>>> > script. No change. >>>> > >>>> > What's probably most interesting is that a SIGKILL is necessary to >>>> > remove the 'stale' processes that are still listening on their sockets >>>> > (as determined by lsof.) >>>> > >>>> > I can supply the config if needed, there is a single listener w/ a pem >>>> > directory specified if that's helpful. >>>> > >>>> >>>> The previous processes don't die that fast because there are connections >>>> still alive on them. Which is caused by very long timeout settings. >>>> >>> >>> The haproxy instance has many tcp-mode connections, and so, yes, the >>> processes often hang around for many days. I'm familiar with haproxy's >>> exiting when all connections have finished. >>> >>> What I'm looking for help about is the listen socket remaining. lsof >>> indicates a listener on the port and so accepts new connections on the >>> 'stale' process and configuration file. >>> >>> >>>> >>>> The behavior you see is normal. >>>> >>>> Cheers, >>>> Pavlos >>>> >>>> >>>> >>>>
