On Tue, Jul 04, 2017 at 10:29:27PM +0200, Lukas Tribus wrote:
> > On Tue, Jul 04, 2017 at 09:56:09PM +0200, Lukas Tribus wrote:
> >> Hi Emeric,
> >>
> >>
> >> since 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show
> >> map/acl' on cli") my setup crashes when a request comes in
> >> going through SSL termination.
> >>
> >> memory corruption, invalid pointers, double free is what haproxy
> >> randomly crashes with.
> > Hmmm bad! Do you want me to revert it now or can this wait for
> > Emeric to try to spot the issue ? William, I've seen you've already
> > backported it into 1.7, be careful not to release the next version
> > before this one is fixed!
>
> No, I got it wrong, its a different commit that is 1.8-only, sorry.
Ah cool! Emeric already spent a few days trying to sort out this
really painful one, I feared to find him hung in the office after
discovering he has to do it again :-) So William that's OK for 1.7.
> The commit is 019489 (MAJOR: task: task scheduler rework).
It could very well make sense especially on certain corner cases where
tasks don't quit cleanly. I insisted that this one was merged early so
that potential issues emerge. The other sensitive one is the scheduler
applet rework (no crash expected but possibly unexpected freezes in
bogus applets).
> I hope I got the right one this time (I don't know why, but I bisected to the
> wrong commit twice now).
It happens to me all the time. You press up arrow enter and realize too
late you got "good" or "bad" wrong.
> root@www:/usr/sbin# haproxy -vv
> HA-Proxy version 1.8-dev2-019489-36 2017/06/27
> Copyright 2000-2017 Willy Tarreau <[email protected]>
>
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
> OPTIONS = USE_GETADDRINFO=1 USE_SLZ=1 USE_OPENSSL=1 USE_PCRE=1
> USE_PCRE_JIT=1
>
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with OpenSSL version : OpenSSL 1.1.0g-dev xx XXX xxxx
> Running on OpenSSL version : OpenSSL 1.1.0g-dev xx XXX xxxx
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with network namespace support.
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Encrypted password support via crypt(3): yes
> Built with PCRE version : 8.38 2015-11-23
> Running on PCRE version : 8.38 2015-11-23
> PCRE library supports JIT : yes
>
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
>
> root@www:/usr/sbin# gdb -ex=r --args haproxy -f /etc/haproxy/haproxy-slim.cfg
> -d
> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from haproxy...done.
> Starting program: /usr/sbin/haproxy -f /etc/haproxy/haproxy-slim.cfg -d
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> [WARNING] 184/222222 (30237) : config : log format ignored for frontend
> 'tls-termination' since it has no log address.
> [WARNING] 184/222222 (30237) : Proxy 'tls-termination': no-sslv3/no-tlsv1x
> are ignored for bind ':443' at [/etc/haproxy/haproxy-slim.cfg:18]. Use only
> 'ssl-min-ver' and 'ssl-max-ver' to fix.
> Note: setting global.maxconn to 2000.
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result FAILED
> Total: 3 (2 usable), will use epoll.
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> Using epoll() as the polling mechanism.
> 00000000:tls-termination.accept(0004)=0005 from [10.0.0.4:56156]
> 00000000:tls-termination.clireq[0005:ffffffff]: GET /robots.txx HTTP/1.1
> 00000000:tls-termination.clihdr[0005:ffffffff]: Host: temp.lan.ltri.eu
> 00000000:tls-termination.clihdr[0005:ffffffff]: User-Agent: curl/7.48.0
> 00000000:tls-termination.clihdr[0005:ffffffff]: Accept: */*
> *** Error in `/usr/sbin/haproxy': malloc(): memory corruption:
> 0x0000000000a419c0 ***
> ======= Backtrace: =========
> /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff71bb7e5]
> /lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7ffff71c613e]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7ffff71c8184]
> /usr/sbin/haproxy[0x523d66]
> /usr/sbin/haproxy[0x522154]
> /usr/sbin/haproxy[0x522732]
> /usr/sbin/haproxy[0x532480]
> /usr/sbin/haproxy[0x4120b6]
> /usr/sbin/haproxy[0x4d89dd]
> /usr/sbin/haproxy[0x4d9a95]
> /usr/sbin/haproxy[0x4fa433]
> /usr/sbin/haproxy[0x5116a2]
> /usr/sbin/haproxy[0x4d4a75]
> /usr/sbin/haproxy[0x4d5d67]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffff7164830]
> /usr/sbin/haproxy[0x4055a9]
> ======= Memory map: ========
> 00400000-007ae000 r-xp 00000000 ca:02 40972
> /usr/sbin/haproxy
> 009ae000-009ce000 r--p 003ae000 ca:02 40972
> /usr/sbin/haproxy
> 009ce000-009ea000 rw-p 003ce000 ca:02 40972
> /usr/sbin/haproxy
> 009ea000-00abf000 rw-p 00000000 00:00 0
> [heap]
> 7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0
> 7ffff0021000-7ffff4000000 ---p 00000000 00:00 0
> 7ffff6b0e000-7ffff6b24000 r-xp 00000000 ca:02 24641
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7ffff6b24000-7ffff6d23000 ---p 00016000 ca:02 24641
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7ffff6d23000-7ffff6d24000 rw-p 00015000 ca:02 24641
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> 7ffff6d24000-7ffff7144000 rw-p 00000000 00:00 0
> 7ffff7144000-7ffff7304000 r-xp 00000000 ca:02 26350
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7ffff7304000-7ffff7504000 ---p 001c0000 ca:02 26350
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7ffff7504000-7ffff7508000 r--p 001c0000 ca:02 26350
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7ffff7508000-7ffff750a000 rw-p 001c4000 ca:02 26350
> /lib/x86_64-linux-gnu/libc-2.23.so
> 7ffff750a000-7ffff750e000 rw-p 00000000 00:00 0
> 7ffff750e000-7ffff7526000 r-xp 00000000 ca:02 24805
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7ffff7526000-7ffff7725000 ---p 00018000 ca:02 24805
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7ffff7725000-7ffff7726000 r--p 00017000 ca:02 24805
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7ffff7726000-7ffff7727000 rw-p 00018000 ca:02 24805
> /lib/x86_64-linux-gnu/libpthread-2.23.so
> 7ffff7727000-7ffff772b000 rw-p 00000000 00:00 0
> 7ffff772b000-7ffff7799000 r-xp 00000000 ca:02 24672
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7ffff7799000-7ffff7999000 ---p 0006e000 ca:02 24672
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7ffff7999000-7ffff799a000 r--p 0006e000 ca:02 24672
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7ffff799a000-7ffff799b000 rw-p 0006f000 ca:02 24672
> /lib/x86_64-linux-gnu/libpcre.so.3.13.2
> 7ffff799b000-7ffff799e000 r-xp 00000000 ca:02 26330
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7ffff799e000-7ffff7b9d000 ---p 00003000 ca:02 26330
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7ffff7b9d000-7ffff7b9e000 r--p 00002000 ca:02 26330
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7ffff7b9e000-7ffff7b9f000 rw-p 00003000 ca:02 26330
> /lib/x86_64-linux-gnu/libdl-2.23.so
> 7ffff7b9f000-7ffff7ba8000 r-xp 00000000 ca:02 24741
> /lib/x86_64-linux-gnu/libcrypt-2.23.so
> 7ffff7ba8000-7ffff7da7000 ---p 00009000 ca:02 24741
> /lib/x86_64-linux-gnu/libcrypt-2.23.so
> 7ffff7da7000-7ffff7da8000 r--p 00008000 ca:02 24741
> /lib/x86_64-linux-gnu/libcrypt-2.23.so
> 7ffff7da8000-7ffff7da9000 rw-p 00009000 ca:02 24741
> /lib/x86_64-linux-gnu/libcrypt-2.23.so
> 7ffff7da9000-7ffff7dd7000 rw-p 00000000 00:00 0
> 7ffff7dd7000-7ffff7dfd000 r-xp 00000000 ca:02 24651
> /lib/x86_64-linux-gnu/ld-2.23.so
> 7ffff7fec000-7ffff7ff0000 rw-p 00000000 00:00 0
> 7ffff7ff5000-7ffff7ff8000 rw-p 00000000 00:00 0
> 7ffff7ff8000-7ffff7ffa000 r--p 00000000 00:00 0
> [vvar]
> 7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0
> [vdso]
> 7ffff7ffc000-7ffff7ffd000 r--p 00025000 ca:02 24651
> /lib/x86_64-linux-gnu/ld-2.23.so
> 7ffff7ffd000-7ffff7ffe000 rw-p 00026000 ca:02 24651
> /lib/x86_64-linux-gnu/ld-2.23.so
> 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
> 7fffffede000-7ffffffff000 rw-p 00000000 00:00 0
> [stack]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
>
> Program received signal SIGABRT, Aborted.
> 0x00007ffff7179428 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0 0x00007ffff7179428 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x00007ffff717b02a in __GI_abort () at abort.c:89
> #2 0x00007ffff71bb7ea in __libc_message (do_abort=2,
> fmt=fmt@entry=0x7ffff72d4e98 "*** Error in `%s': %s: 0x%s ***\n") at
> ../sysdeps/posix/libc_fatal.c:175
> #3 0x00007ffff71c613e in malloc_printerr (ar_ptr=0x7ffff7508b20
> <main_arena>, ptr=0xa419c0, str=0x7ffff72d1cff "malloc(): memory corruption",
> action=<optimized out>) at malloc.c:5006
> #4 _int_malloc (av=av@entry=0x7ffff7508b20 <main_arena>,
> bytes=bytes@entry=16472) at malloc.c:3474
> #5 0x00007ffff71c8184 in __GI___libc_malloc (bytes=16472) at malloc.c:2913
> #6 0x0000000000523d66 in ssl3_setup_write_buffer ()
> #7 0x0000000000522154 in do_ssl3_write ()
> #8 0x0000000000522732 in ssl3_write_bytes ()
> #9 0x0000000000532480 in SSL_write ()
> #10 0x00000000004120b6 in ssl_sock_from_buf (conn=0xa43ce0, buf=0xa5e240,
> flags=1) at src/ssl_sock.c:4974
> #11 0x00000000004d89dd in si_conn_send (conn=0xa43ce0) at
> src/stream_interface.c:658
> #12 0x00000000004d9a95 in si_conn_send_cb (conn=0xa43ce0) at
> src/stream_interface.c:1295
> #13 0x00000000004fa433 in conn_fd_handler (fd=5) at src/connection.c:118
> #14 0x00000000005116a2 in fd_process_cached_events () at src/fd.c:240
This one should theorically not be caused by an issue in the task scheduler,
unless we're reusing something already freed. We could retry it with -dM
and/or -DDEBUG_MEMORY to force earlier corruption to pop up.
Thanks Lukas!
Willy
