On Wed, Oct 26, 2016 at 11:53:08AM +0200, Gabriele Cerami wrote:
> On 22 Oct, Willy Tarreau wrote:
> > Hi Gabriele,
> > 
> > On Tue, Oct 18, 2016 at 09:40:14PM +0200, Gabriele Cerami wrote:
> > > Hi,
> > > 
> > > We're having a problem with version 1.5.14 of haproxy, packaged for
> > > CentOS 7, but it seems even the code in master is affected.
> > > 
> > > In situations where bind is not possible (in our case, the address was
> > > already in use) tcp_connect_server returns with a status of 256
> > > (ERR_ALERT). This value is then passed down as exit code for
> > > haproxy-systemd-wrapper.
> > 
> > Huh ? First, tcp_connect_server() is not involved here, it's used to
> > connect to a server, so it never provides an exit code. And even if
> > instead you meant a bind issue instead, all error codes are limited
> > to 5 bits so the higher value you can have is 0x1F = 31 if all flags
> 
> Yeah, sorry about that hastily made root cause analysis.
> I'll stick to the facts.
> This is what we're seeing
> 
> http://paste.openstack.org/show/586174/

OK so it's the bind for the listener which fails, that makes more sense.
I'm amazed to see that some warnings and errors are reported as coming
from the wrapper while they're emitted by haproxy. Is it another funny
trick of systemd to make your troubleshooting more complicated ?

I don't understand why yet, but I'm not seeing the haproxy message
which should say "[haproxy.main()] cannot bind socket.", instead we're
seeing a different syntax and it is reported from the wrapper. Also
the exit path here clearly does "exit(1)". So I don't know much what
to think of it. Are you sure your version doesn't contain certain patches
affecting the error return path, maybe just some local debugging patches
that were forgotten ?

> The complete output is somewhat difficult to get, it's available here
> http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/9697c38/logs/undercloud/var/log/undercloud_install.txt.gz
> 
> but you'd have to download it, ungzip it, then open the journal log with
> journalctl --file
> var/log/journal/d327fbd51a9f4c4997bdeba685dc22d2/system.journal  | less
> -AXFR

Unfortunately you just made me realize another good reason for staying
away from systemd-based distros, which is that your logs can only be
processed by people using the same stuff, but I personally can't do
anything with these files :-/

> As you can see, the thread spawned by the wrapper is returning 256 as
> status code, and this is interpreded as 0/success by systemd, even if
> it's actually failing to bind to a port, and thus exiting.

That's normal because exit codes are supposed to be in the lower 8 (well
7) bits. So in practice systemd really receives zero here. However I'm
seeing something now. We're directly returning status here which is wrong
as it contains some higher bits, we should use WEXITSTATUS() there if
WIFEXITED() is true. In fact, now I'm pretty sure that this 256 simply
is exit code 1 shifted 8 bits left. Please note that the wrapper in 1.5.14
still has lots of defects that were fixed later, specifically in the
return path. But this one was not fixed yet and definitely needs to be
addressed.

I'll try to propose a patch (but you'll have to adjust it by hand if you
want to put it into 1.5.14).

Willy

Reply via email to