Hi Pierre,
> Apart from that, we exchanged off-list with Willy about the submitted patch.
> It seems that if fixes the issue. I now have only one instance bound to the
> TCP sockets after the reloads, the others are there just to terminate the
> existing connections.
And thank you for the quick t
Hi,
I didn't subscribed to the list and noticed that there was several exchanges on
this thread that I didn't read so far.
To share a bit more of our context:
* we do not reload every 2ms, this was the setting used to be able to reproduce
easily and in a short period of time. Our reload ave
Hi Holger,
On Tue, Oct 25, 2016 at 12:38:26PM +0200, Holger Just wrote:
> Hey Willy,
>
> Willy Tarreau wrote:
> > I absolutely despise systemd and each time I have to work on the
> > wrapper I feel like I'm going to throw up. So for me working on this
> > crap is a huge pain each time. But I'm re
Hey Willy,
Willy Tarreau wrote:
> I absolutely despise systemd and each time I have to work on the
> wrapper I feel like I'm going to throw up. So for me working on this
> crap is a huge pain each time. But I'm really fed up with seeing
> people having problems in this crazy environment because on
Hi Jarno,
On Tue, Oct 25, 2016 at 11:43:44AM +0300, Jarno Huuskonen wrote:
> This is probably a bit off topic, but there's sd_notify call
> (and [Service] Type=notify)
> where service can notify systemd that it's done starting/reloading
> configuration:
> https://www.freedesktop.org/software/syste
Hi Pavlos,
On Tue, Oct 25, 2016 at 10:34:14AM +0200, Pavlos Parissis wrote:
> Well, I have full confidence on the quality of your code
You should not, keep in mind that I still hold and by far the record
on the number of bugs in this project :-)
> (assuming you will
> polish the patch to handle
Hi,
On Sat, Oct 22, Willy Tarreau wrote:
> Another important point, when you say you restart every 2ms, are you
> certain you have a way to ensure that everything is completely started
> before you issue your signal to kill the old process ? I'm asking because
> thanks to the principle that the wr
On 25/10/2016 01:21 πμ, Willy Tarreau wrote:
> Hi guys,
>
> On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote:
>> Not fixing *real world issues* because we don't agree with the use-case or
>> there is a design misconception somewhere else is dangerous. We don't have
>> to support every
Good morning,
Got my coffee ready before I read and reply:-)
On 25/10/2016 12:42 πμ, Lukas Tribus wrote:
> Hello,
>
>
> Am 24.10.2016 um 22:32 schrieb Pavlos Parissis:
>>
>> IMHO: Ask the users to not perform reloads every 2miliseconds. It is
>> insane. You may spend X hours on this which will
Hi guys,
On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote:
> Not fixing *real world issues* because we don't agree with the use-case or
> there is a design misconception somewhere else is dangerous. We don't have
> to support every single obscure use-case out there, that's not what I a
Hello,
Am 24.10.2016 um 22:32 schrieb Pavlos Parissis:
IMHO: Ask the users to not perform reloads every 2miliseconds. It is insane.
You may spend X hours on this which will make the code a bot more complex and
cause possible breakages somewhere else.
Not fixing *real world issues* because we
On 24/10/2016 09:13 μμ, Willy Tarreau wrote:
> Hi again,
>
> On Mon, Oct 24, 2016 at 07:41:06PM +0200, Willy Tarreau wrote:
>> I don't know if this is something you're interested in experimenting
>> with. This is achieved using fcntl(F_SETLKW). It should be done in the
>> wrapper as well.
>
> Fin
On Mon, Oct 24, 2016 at 09:13:13PM +0200, Willy Tarreau wrote:
> BTW, that just makes me realize that we also have another possibility for this
> precisely using a pipe (which are more portable than mandatory locks). Let's
> see if that would work. The wrapper creates a pipe then forks. The child
>
Hi again,
On Mon, Oct 24, 2016 at 07:41:06PM +0200, Willy Tarreau wrote:
> I don't know if this is something you're interested in experimenting
> with. This is achieved using fcntl(F_SETLKW). It should be done in the
> wrapper as well.
Finally I did it and it doesn't help at all. The signal-based
Hi Pierre,
On Mon, Oct 24, 2016 at 12:46:34PM +, Pierre Cheynier wrote:
> Unfortunately, I remember we had the same issue (but less frequently) on
> CentOS6 which is init-based.
OK but be careful, we used to have other issues with signals in the
past, it's not necessarily exactly the same thi
Hi,
here is the idea :
[...]
ExecReload=
ExecReload=/bin/kill -USR2 $MAINPID
[...]
not the sexiest solution but it do the job and I never got the problem
anymore ;)
2016-10-24 17:43 GMT+02:00 Pierre Cheynier :
>> A solution I use is to delay next reload in systemd unit until a
>> reload is in p
> A solution I use is to delay next reload in systemd unit until a
> reload is in progress.
Unfortunately, even when doing this you can end up in the situation described
before, because for systemd a reload is basically a SIGUSR2 to send. You do not
wait for some callback saying "I'm now OK and
Hi,
I had similar issues when reloading haproxy with lot of ssl (long to fork).
A solution I use is to delay next reload in systemd unit until a
reload is in progress.
2016-10-24 17:06 GMT+02:00 Willy Tarreau :
> On Mon, Oct 24, 2016 at 01:09:59PM +, Pierre Cheynier wrote:
>> > Same for all of
On Mon, Oct 24, 2016 at 01:09:59PM +, Pierre Cheynier wrote:
> > Same for all of them. Very interesting, SIGUSR2 (12) is set
> > in SigIgn :-) One question is "why", but at least we know we
> > have a workaround consisiting in unblocking these signals in
> > haproxy-systemd-wrapper, as we did
On 24 October 2016 at 13:46, Pierre Cheynier wrote:
> Hi,
>
> Sorry, wrong order in the answers.
>
> > Yes it has something to do with it because it's the systemd-wrapper which
> > delivers the signal to the old processes in this mode, while in the
> normal
> > mode the processes get the signal d
> Same for all of them. Very interesting, SIGUSR2 (12) is set
> in SigIgn :-) One question is "why", but at least we know we
> have a workaround consisiting in unblocking these signals in
> haproxy-systemd-wrapper, as we did in haproxy.
> Care to retry with the attached patch ?
Same behaviour.
Hi,
Sorry, wrong order in the answers.
> Yes it has something to do with it because it's the systemd-wrapper which
> delivers the signal to the old processes in this mode, while in the normal
> mode the processes get the signal directly from the new process. Another
> important point is that exa
Hi Pierre,
On Mon, Oct 24, 2016 at 12:16:32PM +, Pierre Cheynier wrote:
> $ grep ^Sig /proc/43135/status
> SigQ: 0/192473
> SigPnd:
> SigBlk:
> SigIgn: 1000
> SigCgt: 000180004803
(...)
Same for all of them. Very interesting, SI
Hi,
> Pierre, could you please issue "grep ^Sig /proc/pid/status" for each
> wrapper and haproxy process ? I'm interested in seeing SigIgn and
> SigBlk particularly.
>
Sure, here is the output for the following pstree:
$ ps fauxww | grep haproxy | grep -v grep
root 43135 0.0 0.0 46340
Hi Maciej,
On Fri, Oct 21, 2016 at 03:44:22PM -0700, Maciej Katafiasz wrote:
> There was a similar issue with reloads in Docker that I reported a while
> back: https://www.mail-archive.com/haproxy@formilux.org/msg21485.html . It
> was ultimately tracked down to a faulty Golang compiler version, wh
On 21 October 2016 at 15:33, Willy Tarreau wrote:
>
> On Fri, Oct 21, 2016 at 03:05:55PM +, Pierre Cheynier wrote:
> > First let's clarify again: we are on systemd-based OS (centOS7), so
> reload is
> > done by sending SIGUSR2 to haproxy-systemd-wrapper.
> > Theoretically, this has absolutely
Hi Pierre,
On Fri, Oct 21, 2016 at 03:05:55PM +, Pierre Cheynier wrote:
> First let's clarify again: we are on systemd-based OS (centOS7), so reload is
> done by sending SIGUSR2 to haproxy-systemd-wrapper.
> Theoretically, this has absolutely no relation with our current issue (if I
> understa
Hi Willy,
Thanks for your answer and sorry for my delay.
First let's clarify again: we are on systemd-based OS (centOS7), so reload is
done by sending SIGUSR2 to haproxy-systemd-wrapper.
Theoretically, this has absolutely no relation with our current issue (if I
understand well the way the old
Hi Pierre,
On Fri, Oct 14, 2016 at 10:54:43AM +, Pierre Cheynier wrote:
> Hi Lukas,
>
> > I did not meant no-reuseport to workaround or "solve" the problem
> definitely, but rather to see if the problems can still be triggered,
> since you can reproduce the problem easily.
>
> This still h
Hi,
Any updates/findings on that issue ?
Many thanks,
Pierre
> From : Pierre Cheynier
> To: Lukas Tribus; haproxy@formilux.org
> Sent: Friday, October 14, 2016 12:54 PM
> Subject: RE: HAProxy reloads lets old and outdated processes
>
> Hi Lukas,
>
> > I di
Hi Lukas,
> I did not meant no-reuseport to workaround or "solve" the problem
definitely, but rather to see if the problems can still be triggered,
since you can reproduce the problem easily.
This still happens using snapshot 20161005 with no-reuseport set, a bit less
probably because reload i
Hi Pierre,
Am 13.10.2016 um 18:56 schrieb Pierre Cheynier:
This becomes impossible in PaaS-like approach where many events occurs and may trigger
reloads every seconds. BTW, the new "no-reuseport" feature does not help in my
case (as well as ip/nftables or tc workarounds) because it introduce
Hi list,
I experiment the following behaviour : I'm on 1.6.8 (same behaviour in
1.4/1.5), use systemd and noticed that when reloads are relatively frequent,
old processes sometimes never dies and stays bound to the TCP socket(s), thanks
to SO_REUSEPORT.
Here is an example of process tree:
roo
33 matches
Mail list logo
