Re: raise() on HAProxy 2.0

2020-05-19 Thread Olivier D
Hello Willy,

Le ven. 15 mai 2020 à 17:33, Willy Tarreau  a écrit :

>
> Is it 100% reproducible and if so can you please share a minimal config
> and reproducer so that we can quickly focus on it ?
>
Unfortunately I was unable to reproduce it. It only happens for several
hours in a row, then stop (for no reason), then triggers again.
And the config file contains ~ 1000 certificates...

It did not trigger before upgrade to 2.0.14 (I was using 2.0.13 before).

let me know if I can dig into some of the coredump to provide any useful
information.

 Olivier


Re: raise() on HAProxy 2.0

2020-05-15 Thread Willy Tarreau
Hi Olivier,

On Thu, May 14, 2020 at 04:24:53PM +0200, Olivier D wrote:
> Hello,
> 
> I'm spamming a lot these days :)
> 
> I found a strange coredump on HAProxy 2.0.14 that started a few days ago
> for no reason. It's not a coredump but a raise().

Note that if your cores are enabled you ought to get one on raise().

> Stacktrace :
> 
> #0  0x7fde8c9f8495 in raise () from /lib64/libc.so.6
> #1  0x7fde8c9f9c75 in abort () from /lib64/libc.so.6
> #2  0x7fde8ca363a7 in __libc_message () from /lib64/libc.so.6
> #3  0x7fde8ca3bdee in malloc_printerr () from /lib64/libc.so.6
> #4  0x7fde8ca3ec3d in _int_free () from /lib64/libc.so.6
> #5  0x0047a885 in ssl_sock_free_ssl_conf () at src/ssl_sock.c:3740
> #6  0x0047bdd2 in ssl_sock_free_all_ctx () at src/ssl_sock.c:5063
> #7  0x0047c301 in ssl_sock_destroy_bind_conf () at
> src/ssl_sock.c:5095
> #8  0x0050c8fb in deinit () at src/haproxy.c:2533
> #9  0x0050dc3f in main () at src/haproxy.c:3449

This seriously looks like a double free or a free of an invalid pointer.

> This seems to happen when issuing the following command :
> 'set ssl ocsp-response xxx' |socat stdio /var/run/haproxy.sock

Is it 100% reproducible and if so can you please share a minimal config
and reproducer so that we can quickly focus on it ?

> This is the first time I see such a behaviour :/
> 
> I can provide a "bt full" output privately if needed.

It could help if not reproduced, but let's first see if there's a way
to reproduce it. The SSL setup/release code is extremely complex due
to non-atomic setup that requires to be released on error, and it's
very possible that we have a double free there, possibly caused by
a missing assignment of a pointer to NULL after freeing it.

Thanks,
Willy



raise() on HAProxy 2.0

2020-05-14 Thread Olivier D
Hello,

I'm spamming a lot these days :)

I found a strange coredump on HAProxy 2.0.14 that started a few days ago
for no reason. It's not a coredump but a raise().

Stacktrace :

#0  0x7fde8c9f8495 in raise () from /lib64/libc.so.6
#1  0x7fde8c9f9c75 in abort () from /lib64/libc.so.6
#2  0x7fde8ca363a7 in __libc_message () from /lib64/libc.so.6
#3  0x7fde8ca3bdee in malloc_printerr () from /lib64/libc.so.6
#4  0x7fde8ca3ec3d in _int_free () from /lib64/libc.so.6
#5  0x0047a885 in ssl_sock_free_ssl_conf () at src/ssl_sock.c:3740
#6  0x0047bdd2 in ssl_sock_free_all_ctx () at src/ssl_sock.c:5063
#7  0x0047c301 in ssl_sock_destroy_bind_conf () at
src/ssl_sock.c:5095
#8  0x0050c8fb in deinit () at src/haproxy.c:2533
#9  0x0050dc3f in main () at src/haproxy.c:3449


This seems to happen when issuing the following command :
'set ssl ocsp-response xxx' |socat stdio /var/run/haproxy.sock

This is the first time I see such a behaviour :/

I can provide a "bt full" output privately if needed.

HAProxy build options:
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=

Built with OpenSSL version : OpenSSL 1.1.1g  21 Apr 2020



Olivier