Hi Willy,

Thanks for your answer and sorry for my delay.

First let's clarify again: we are on systemd-based OS (centOS7), so reload is 
done by sending SIGUSR2 to haproxy-systemd-wrapper.
Theoretically, this has absolutely no relation with our current issue (if I 
understand well the way the old process are managed)

This happens on servers with live traffic, but with a reasonable amount of 
connections. I'm also able to reproduce with no connections, but I've to be a 
bit more aggressive with the reloads frequency (probably because children are 
faster to die).

For me the problem is not that we still have connections or not, it is that in 
this case some old processes are never "aware" that they should die, so they 
continues to listen for incoming requests, thanks to SO_REUSEPORT.

Consequently, you end up with N process listening with different configs.

In the pstree I pasted in the previous message, there are 3 minutes between the 
first living instance and the last (and as you can see, we are quite aggressive 
with long connections) :

     timeout client 2s
     timeout server 5s
     timeout connect 200ms
     timeout http-keep-alive 200ms

Here is a Dockerfile that can be used to reproduce (where I use 
haproxy-systemd-wrapper, just run with default settings - ie nb of reloads=300 
and interval between each=2ms -) :

https://github.com/pierrecdn/haproxy-reload-issue

docker build -t haproxy-reload-issue . && docker run --rm -ti 
haproxy-reload-issue

Thanks,

Pierre
    
> Hi Pierre,
>
> (...)
>
> Is this with live traffic or on a test machine ? Could you please check
> whether these instances have one connection attached ? I don't see any
> valid reason for a dying process not to leave once it doesn't have any
> more connection. And during my last attempts at fixing such issues by
> carefully reviewing the code and hammering the systemd-wrapper like mad,
> I couldn't get this behaviour to happen a single time. Thus it would be
> nice to know what these processes are doing there and why they don't
> stop.
> 
> Regards,
> Willy
     

Reply via email to