Re: Bind socket not exists from time to time?
On Thu, Apr 11, 2019 at 02:32:28PM -0400, francis Lavalliere wrote:
> So just to be clear, there is only 1 service, using a Master -> Worker
> config.
>
> The service is working since many hours (no doing any restart, simply
> modification of the config and executing the hot-reload command)...
Yes but that's also what I was suggesting, that maybe it's reloaded
twice very fast in parallel and that one of them fails and removes
the socket.
> I'll try to dig deeper on myside also when I got a chance, I was mostly
> wondering if anyone had that kind of issue (i know using sockets file isnt
> much people are using).
In fact yes, most people have this for the CLI ("stats socket") so it's
not at all something uncommon.
Willy
Re: Bind socket not exists from time to time?
So just to be clear, there is only 1 service, using a Master -> Worker
config.
The service is working since many hours (no doing any restart, simply
modification of the config and executing the hot-reload command)...
I'll try to dig deeper on myside also when I got a chance, I was mostly
wondering if anyone had that kind of issue (i know using sockets file isnt
much people are using).
- Francis L.
On Thu, Apr 11, 2019 at 12:19 PM Willy Tarreau wrote:
> Hello Francis,
>
> On Thu, Apr 11, 2019 at 11:20:57AM -0400, francis Lavalliere wrote:
> > Hello,
> >
> > Anyone else would have seen this behavior before?
> > I am currently using HAProxy 1.9.3
> >
> > I have an "haproxy / consul template" that generate an haproxy-x.sock
> > file, from time to time when my config reloads, I find out after a few
> > minutes that the service doesn't work and actually the socket file doesnt
> > exists on the disk...
> >
> > Going to the HA Proxy server i can see the haproxy has a valid
> > configuration, I even reload the worker threads, and the socket file
> still
> > does not exists on disk.
> >
> > ie:
> >
> > listen x
> >
> > bind /var/run/haproxy-x.sock accept-proxy
> >
> > bind-process 2
> >
> > I am wondering what could be the cause of this. I am assuming that the
> > haproxy template that I generate at some point create that listen, reload
> > the worker threads) then regenerate another configuration without that
> > socket, it then get removed... and then re-regenerate the config with the
> > listen and socket directive... but somehow the socket file doesnt get
> > reecreated.
> >
> > I might be wrong, as for now I have not much other possible ways to
> > reproduce easily as its happening randomly.
>
> I've not seen this but I can more or less imagine something that could
> cause this. Let's imagine that for whatever reason your service scripts
> start two haproxy processes in parallel, which end up like this :
>
>process1 process2
> parse conf parse conf
> unlink("/var/run/..")
> bind("/var/run/..")
> unlink("/var/run/..")
> bind("/var/run/..")
> start other stuff start other stuff
> fail on something
> unlink("/var/run/...")
> exitcontinue to run
>
> In this case you can end up with process 2 running with process 1 having
> removed its socket while cleaning up on exit. It would randomly happen
> depending on the start order but that might be a possible explanation.
> Now maybe it's totally unrelated but I don't have any other idea for
> now.
>
> Regards,
> Willy
>
Re: Bind socket not exists from time to time?
Hello Francis,
On Thu, Apr 11, 2019 at 11:20:57AM -0400, francis Lavalliere wrote:
> Hello,
>
> Anyone else would have seen this behavior before?
> I am currently using HAProxy 1.9.3
>
> I have an "haproxy / consul template" that generate an haproxy-x.sock
> file, from time to time when my config reloads, I find out after a few
> minutes that the service doesn't work and actually the socket file doesnt
> exists on the disk...
>
> Going to the HA Proxy server i can see the haproxy has a valid
> configuration, I even reload the worker threads, and the socket file still
> does not exists on disk.
>
> ie:
>
> listen x
>
> bind /var/run/haproxy-x.sock accept-proxy
>
> bind-process 2
>
> I am wondering what could be the cause of this. I am assuming that the
> haproxy template that I generate at some point create that listen, reload
> the worker threads) then regenerate another configuration without that
> socket, it then get removed... and then re-regenerate the config with the
> listen and socket directive... but somehow the socket file doesnt get
> reecreated.
>
> I might be wrong, as for now I have not much other possible ways to
> reproduce easily as its happening randomly.
I've not seen this but I can more or less imagine something that could
cause this. Let's imagine that for whatever reason your service scripts
start two haproxy processes in parallel, which end up like this :
process1 process2
parse conf parse conf
unlink("/var/run/..")
bind("/var/run/..")
unlink("/var/run/..")
bind("/var/run/..")
start other stuff start other stuff
fail on something
unlink("/var/run/...")
exitcontinue to run
In this case you can end up with process 2 running with process 1 having
removed its socket while cleaning up on exit. It would randomly happen
depending on the start order but that might be a possible explanation.
Now maybe it's totally unrelated but I don't have any other idea for
now.
Regards,
Willy
Bind socket not exists from time to time?
Hello, Anyone else would have seen this behavior before? I am currently using HAProxy 1.9.3 I have an "haproxy / consul template" that generate an haproxy-x.sock file, from time to time when my config reloads, I find out after a few minutes that the service doesn't work and actually the socket file doesnt exists on the disk... Going to the HA Proxy server i can see the haproxy has a valid configuration, I even reload the worker threads, and the socket file still does not exists on disk. ie: listen x bind /var/run/haproxy-x.sock accept-proxy bind-process 2 I am wondering what could be the cause of this. I am assuming that the haproxy template that I generate at some point create that listen, reload the worker threads) then regenerate another configuration without that socket, it then get removed... and then re-regenerate the config with the listen and socket directive... but somehow the socket file doesnt get reecreated. I might be wrong, as for now I have not much other possible ways to reproduce easily as its happening randomly.
