Re: HAProxy reloads lets old and outdated processes

Hi Pierre, > Apart from that, we exchanged off-list with Willy about the submitted patch. > It seems that if fixes the issue. I now have only one instance bound to the > TCP sockets after the reloads, the others are there just to terminate the > existing connections. And thank you for the quick

RE: HAProxy reloads lets old and outdated processes

Hi, I didn't subscribed to the list and noticed that there was several exchanges on this thread that I didn't read so far. To share a bit more of our context: * we do not reload every 2ms, this was the setting used to be able to reproduce easily and in a short period of time. Our reload

Re: HAProxy reloads lets old and outdated processes

Hi Holger, On Tue, Oct 25, 2016 at 12:38:26PM +0200, Holger Just wrote: > Hey Willy, > > Willy Tarreau wrote: > > I absolutely despise systemd and each time I have to work on the > > wrapper I feel like I'm going to throw up. So for me working on this > > crap is a huge pain each time. But I'm

Re: HAProxy reloads lets old and outdated processes

Hey Willy, Willy Tarreau wrote: > I absolutely despise systemd and each time I have to work on the > wrapper I feel like I'm going to throw up. So for me working on this > crap is a huge pain each time. But I'm really fed up with seeing > people having problems in this crazy environment because

Re: HAProxy reloads lets old and outdated processes

Hi Jarno, On Tue, Oct 25, 2016 at 11:43:44AM +0300, Jarno Huuskonen wrote: > This is probably a bit off topic, but there's sd_notify call > (and [Service] Type=notify) > where service can notify systemd that it's done starting/reloading > configuration: >

Re: HAProxy reloads lets old and outdated processes

Hi, On Sat, Oct 22, Willy Tarreau wrote: > Another important point, when you say you restart every 2ms, are you > certain you have a way to ensure that everything is completely started > before you issue your signal to kill the old process ? I'm asking because > thanks to the principle that the

Re: HAProxy reloads lets old and outdated processes

On 25/10/2016 01:21 πμ, Willy Tarreau wrote: > Hi guys, > > On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote: >> Not fixing *real world issues* because we don't agree with the use-case or >> there is a design misconception somewhere else is dangerous. We don't have >> to support every

Re: HAProxy reloads lets old and outdated processes

Good morning, Got my coffee ready before I read and reply:-) On 25/10/2016 12:42 πμ, Lukas Tribus wrote: > Hello, > > > Am 24.10.2016 um 22:32 schrieb Pavlos Parissis: >> >> IMHO: Ask the users to not perform reloads every 2miliseconds. It is >> insane. You may spend X hours on this which

Re: HAProxy reloads lets old and outdated processes

Hi guys, On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote: > Not fixing *real world issues* because we don't agree with the use-case or > there is a design misconception somewhere else is dangerous. We don't have > to support every single obscure use-case out there, that's not what I

Re: HAProxy reloads lets old and outdated processes

Hello, Am 24.10.2016 um 22:32 schrieb Pavlos Parissis: IMHO: Ask the users to not perform reloads every 2miliseconds. It is insane. You may spend X hours on this which will make the code a bot more complex and cause possible breakages somewhere else. Not fixing *real world issues* because

Re: HAProxy reloads lets old and outdated processes

On 24/10/2016 09:13 μμ, Willy Tarreau wrote: > Hi again, > > On Mon, Oct 24, 2016 at 07:41:06PM +0200, Willy Tarreau wrote: >> I don't know if this is something you're interested in experimenting >> with. This is achieved using fcntl(F_SETLKW). It should be done in the >> wrapper as well. > >

Testers needed [Re: HAProxy reloads lets old and outdated processes]

On Mon, Oct 24, 2016 at 09:13:13PM +0200, Willy Tarreau wrote: > BTW, that just makes me realize that we also have another possibility for this > precisely using a pipe (which are more portable than mandatory locks). Let's > see if that would work. The wrapper creates a pipe then forks. The child

Re: HAProxy reloads lets old and outdated processes

Hi again, On Mon, Oct 24, 2016 at 07:41:06PM +0200, Willy Tarreau wrote: > I don't know if this is something you're interested in experimenting > with. This is achieved using fcntl(F_SETLKW). It should be done in the > wrapper as well. Finally I did it and it doesn't help at all. The

Re: HAProxy reloads lets old and outdated processes

Hi Pierre, On Mon, Oct 24, 2016 at 12:46:34PM +, Pierre Cheynier wrote: > Unfortunately, I remember we had the same issue (but less frequently) on > CentOS6 which is init-based. OK but be careful, we used to have other issues with signals in the past, it's not necessarily exactly the same

Re: HAProxy reloads lets old and outdated processes

Hi, here is the idea : [...] ExecReload= ExecReload=/bin/kill -USR2 $MAINPID [...] not the sexiest solution but it do the job and I never got the problem anymore ;) 2016-10-24 17:43 GMT+02:00 Pierre Cheynier : >> A solution I use is to delay next reload in systemd unit

RE: HAProxy reloads lets old and outdated processes

> A solution I use is to delay next reload in systemd unit until a > reload is in progress. Unfortunately, even when doing this you can end up in the situation described before, because for systemd a reload is basically a SIGUSR2 to send. You do not wait for some callback saying "I'm now OK and

Re: HAProxy reloads lets old and outdated processes

Hi, I had similar issues when reloading haproxy with lot of ssl (long to fork). A solution I use is to delay next reload in systemd unit until a reload is in progress. 2016-10-24 17:06 GMT+02:00 Willy Tarreau : > On Mon, Oct 24, 2016 at 01:09:59PM +, Pierre Cheynier wrote: >> >

Re: HAProxy reloads lets old and outdated processes

On Mon, Oct 24, 2016 at 01:09:59PM +, Pierre Cheynier wrote: > > Same for all of them. Very interesting, SIGUSR2 (12) is set > > in SigIgn :-)  One question is "why", but at least we know we > > have a workaround consisiting in unblocking these signals in > > haproxy-systemd-wrapper, as we did

Re: HAProxy reloads lets old and outdated processes

On 24 October 2016 at 13:46, Pierre Cheynier wrote: > Hi, > > Sorry, wrong order in the answers. > > > Yes it has something to do with it because it's the systemd-wrapper which > > delivers the signal to the old processes in this mode, while in the > normal > > mode the

RE: HAProxy reloads lets old and outdated processes

> Same for all of them. Very interesting, SIGUSR2 (12) is set > in SigIgn :-)  One question is "why", but at least we know we > have a workaround consisiting in unblocking these signals in > haproxy-systemd-wrapper, as we did in haproxy. > Care to retry with the attached patch ? Same behaviour.

RE: HAProxy reloads lets old and outdated processes

Hi, Sorry, wrong order in the answers. > Yes it has something to do with it because it's the systemd-wrapper which > delivers the signal to the old processes in this mode, while in the normal > mode the processes get the signal directly from the new process. Another > important point is that

Re: HAProxy reloads lets old and outdated processes

Hi Pierre, On Mon, Oct 24, 2016 at 12:16:32PM +, Pierre Cheynier wrote: > $ grep ^Sig /proc/43135/status > SigQ:    0/192473 > SigPnd:     > SigBlk:     > SigIgn:    1000 > SigCgt:    000180004803 (...) Same for all of them. Very interesting,

RE: HAProxy reloads lets old and outdated processes

Hi, > Pierre, could you please issue "grep ^Sig /proc/pid/status" for each > wrapper and haproxy process ? I'm interested in seeing SigIgn and > SigBlk particularly. > Sure, here is the output for the following pstree: $ ps fauxww | grep haproxy | grep -v grep root 43135  0.0  0.0  46340 

Re: HAProxy reloads lets old and outdated processes

Hi Maciej, On Fri, Oct 21, 2016 at 03:44:22PM -0700, Maciej Katafiasz wrote: > There was a similar issue with reloads in Docker that I reported a while > back: https://www.mail-archive.com/haproxy@formilux.org/msg21485.html . It > was ultimately tracked down to a faulty Golang compiler version,

Re: HAProxy reloads lets old and outdated processes

On 21 October 2016 at 15:33, Willy Tarreau wrote: > > On Fri, Oct 21, 2016 at 03:05:55PM +, Pierre Cheynier wrote: > > First let's clarify again: we are on systemd-based OS (centOS7), so > reload is > > done by sending SIGUSR2 to haproxy-systemd-wrapper. > > Theoretically, this

Re: HAProxy reloads lets old and outdated processes

Hi Pierre, On Fri, Oct 21, 2016 at 03:05:55PM +, Pierre Cheynier wrote: > First let's clarify again: we are on systemd-based OS (centOS7), so reload is > done by sending SIGUSR2 to haproxy-systemd-wrapper. > Theoretically, this has absolutely no relation with our current issue (if I >

RE: HAProxy reloads lets old and outdated processes

Hi Willy, Thanks for your answer and sorry for my delay. First let's clarify again: we are on systemd-based OS (centOS7), so reload is done by sending SIGUSR2 to haproxy-systemd-wrapper. Theoretically, this has absolutely no relation with our current issue (if I understand well the way the old

Re: HAProxy reloads lets old and outdated processes

Hi Pierre, On Fri, Oct 14, 2016 at 10:54:43AM +, Pierre Cheynier wrote: > Hi Lukas, > > > I did not meant no-reuseport to workaround or "solve" the problem > definitely, but rather to see if the problems can still be triggered, > since you can reproduce the problem easily. > > This still

RE: HAProxy reloads lets old and outdated processes

Hi, Any updates/findings on that issue ? Many thanks, Pierre > From : Pierre Cheynier > To: Lukas Tribus; haproxy@formilux.org > Sent: Friday, October 14, 2016 12:54 PM > Subject: RE: HAProxy reloads lets old and outdated processes >   > Hi Lukas, > > > I

RE: HAProxy reloads lets old and outdated processes

Hi Lukas, > I did not meant no-reuseport to workaround or "solve" the problem definitely, but rather to see if the problems can still be triggered, since you can reproduce the problem easily. This still happens using snapshot 20161005 with no-reuseport set, a bit less probably because reload

Re: HAProxy reloads lets old and outdated processes

Hi Pierre, Am 13.10.2016 um 18:56 schrieb Pierre Cheynier: This becomes impossible in PaaS-like approach where many events occurs and may trigger reloads every seconds. BTW, the new "no-reuseport" feature does not help in my case (as well as ip/nftables or tc workarounds) because it

HAProxy reloads lets old and outdated processes

Hi list, I experiment the following behaviour : I'm on 1.6.8 (same behaviour in 1.4/1.5), use systemd and noticed that when reloads are relatively frequent, old processes sometimes never dies and stays bound to the TCP socket(s), thanks to SO_REUSEPORT. Here is an example of process tree: