Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Lukas Tribus]

Am 04.07.2017 um 23:18 schrieb Willy Tarreau:
> On Tue, Jul 04, 2017 at 10:57:08PM +0200, Lukas Tribus wrote:
>> The call trace doesn't really look different when I used -dM or 
>> -DDEBUG_MEMORY.
>>
>> I was able to get a different trace by actually connecting to a backend 
>> however,
>> (instead of showing an haproxy internal 403 error):
> (...)
>
> Thank you Lukas, let's hope this will help.
>
>

Another bisect (this time with -dM or -DDEBUG_MEMORY), another commit...
Now it points to 23e9e931 (MINOR: log: Add logurilen tunable).


(gdb) bt
#0  0x77179428 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x7717b02a in __GI_abort () at abort.c:89
#2  0x771bb7ea in __libc_message (do_abort=do_abort@entry=2, 
fmt=fmt@entry=0x772d4e98 "*** Error in `%s': %s: 0x%s ***\n") at 
../sysdeps/posix/libc_fatal.c:175
#3  0x771c437a in malloc_printerr (ar_ptr=, 
ptr=, str=0x772d4fc8 "double free or corruption (!prev)", 
action=3) at malloc.c:5006
#4  _int_free (av=, p=, have_lock=0) at 
malloc.c:3867
#5  0x771c853c in __GI___libc_free (mem=) at 
malloc.c:2968
#6  0x0053758e in SSL_SESSION_free ()
#7  0x0053324b in SSL_free ()
#8  0x00412055 in ssl_sock_close (conn=0xa56a10) at src/ssl_sock.c:4971
#9  0x004d6a90 in conn_xprt_close (conn=0xa56a10) at 
include/proto/connection.h:96
#10 0x004d6b01 in conn_full_close (conn=0xa56a10) at 
include/proto/connection.h:140
#11 0x004d927f in stream_int_shutw_conn (si=0xaaad08) at 
src/stream_interface.c:883
#12 0x004d7cd7 in si_shutw (si=0xaaad08) at 
include/proto/stream_interface.h:322
#13 0x004d866c in stream_int_notify (si=0xaaad08) at 
src/stream_interface.c:459
#14 0x004d8ad9 in si_conn_wake_cb (conn=0xa56a10) at 
src/stream_interface.c:581
#15 0x004f7fad in conn_fd_handler (fd=5) at src/connection.c:158
#16 0x0050f1cb in fd_process_cached_events () at src/fd.c:240
#17 0x004d4ddd in run_poll_loop () at src/haproxy.c:2193
#18 0x004d6092 in main (argc=5, argv=0x7fffe648) at 
src/haproxy.c:2700
(gdb) quit

and

(gdb) bt
#0  0x77179428 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x7717b02a in __GI_abort () at abort.c:89
#2  0x771bb7ea in __libc_message (do_abort=2, 
fmt=fmt@entry=0x772d4e98 "*** Error in `%s': %s: 0x%s ***\n") at 
../sysdeps/posix/libc_fatal.c:175
#3  0x771c613e in malloc_printerr (ar_ptr=0x77508b20 , 
ptr=0xa3f410, str=0x772d1cff "malloc(): memory corruption", 
action=) at malloc.c:5006
#4  _int_malloc (av=av@entry=0x77508b20 , bytes=bytes@entry=64) 
at malloc.c:3474
#5  0x771c8184 in __GI___libc_malloc (bytes=64) at malloc.c:2913
#6  0x00503755 in pool_refill_alloc (pool=0xa30cd0, avail=1) at 
src/memory.c:117
#7  0x004eb3be in pool_alloc_dirty (pool=0xa30cd0) at 
include/common/memory.h:151
#8  0x004eb3e0 in pool_alloc2 (pool=0xa30cd0) at 
include/common/memory.h:165
#9  0x004ec10a in flt_stream_add_filter (s=0xaaa650, fconf=0xa3d5b0, 
flags=1) at src/filters.c:324
#10 0x004ec646 in flt_set_stream_backend (s=0xaaa650, be=0xa490e0) at 
src/filters.c:448
#11 0x004e1d63 in stream_set_backend (s=0xaaa650, be=0xa490e0) at 
src/proxy.c:1256
#12 0x00484a8e in process_switching_rules (s=0xaaa650, req=0xaaa660, 
an_bit=32) at src/stream.c:1140
#13 0x00486243 in process_stream (t=0xa4f710) at src/stream.c:1795
#14 0x005063b6 in process_runnable_tasks () at src/task.c:238
#15 0x004d4d6e in run_poll_loop () at src/haproxy.c:2175
#16 0x004d6092 in main (argc=5, argv=0x7fffe648) at 
src/haproxy.c:2700
(gdb)

Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Willy Tarreau]

On Tue, Jul 04, 2017 at 10:57:08PM +0200, Lukas Tribus wrote:
> The call trace doesn't really look different when I used -dM or 
> -DDEBUG_MEMORY.
> 
> I was able to get a different trace by actually connecting to a backend 
> however,
> (instead of showing an haproxy internal 403 error):
(...)

Thank you Lukas, let's hope this will help.

Willy

Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Lukas Tribus]

Hi,


Am 04.07.2017 um 22:35 schrieb Willy Tarreau:
>
> This one should theorically not be caused by an issue in the task scheduler,
> unless we're reusing something already freed. We could retry it with -dM
> and/or -DDEBUG_MEMORY to force earlier corruption to pop up.

The call trace doesn't really look different when I used -dM or -DDEBUG_MEMORY.

I was able to get a different trace by actually connecting to a backend however,
(instead of showing an haproxy internal 403 error):

(gdb) bt
#0  0x77179428 in __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x7717b02a in __GI_abort () at abort.c:89
#2  0x771bb7ea in __libc_message (do_abort=2, 
fmt=fmt@entry=0x772d4e98 "*** Error in `%s': %s: 0x%s ***\n") at 
../sysdeps/posix/libc_fatal.c:175
#3  0x771c2913 in malloc_printerr (ar_ptr=0x77508b20 , 
ptr=0xa83220, str=0x772d1c35 "corrupted size vs. prev_size", 
action=) at malloc.c:5006
#4  malloc_consolidate (av=av@entry=0x77508b20 ) at 
malloc.c:4175
#5  0x771c5cde in _int_malloc (av=av@entry=0x77508b20 , 
bytes=bytes@entry=4448) at malloc.c:3450
#6  0x771c8184 in __GI___libc_malloc (bytes=4448) at malloc.c:2913
#7  0x005d658e in CRYPTO_zalloc ()
#8  0x00536172 in SSL_new ()
#9  0x0041143a in ssl_sock_init (conn=0xa3f9a0) at src/ssl_sock.c:4515
#10 0x004fcc58 in conn_xprt_init (conn=0xa3f9a0) at 
include/proto/connection.h:79
#11 0x004fde02 in session_accept_fd (l=0xa432f0, cfd=5, 
addr=0x7fffe320) at src/session.c:152
#12 0x004e44c6 in listener_accept (fd=4) at src/listener.c:493
#13 0x005116a2 in fd_process_cached_events () at src/fd.c:240
#14 0x004d4a75 in run_poll_loop () at src/haproxy.c:2186
#15 0x004d5d67 in main (argc=7, argv=0x7fffe638) at 
src/haproxy.c:2701
(gdb)

Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Willy Tarreau]

On Tue, Jul 04, 2017 at 10:29:27PM +0200, Lukas Tribus wrote:
> > On Tue, Jul 04, 2017 at 09:56:09PM +0200, Lukas Tribus wrote:
> >> Hi Emeric,
> >>
> >>
> >> since 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show
> >> map/acl' on cli") my setup crashes when a request comes in
> >> going through SSL termination.
> >>
> >> memory corruption, invalid pointers, double free is what haproxy
> >> randomly crashes with.
> > Hmmm bad! Do you want me to revert it now or can this wait for
> > Emeric to try to spot the issue ? William, I've seen you've already
> > backported it into 1.7, be careful not to release the next version
> > before this one is fixed!
> 
> No, I got it wrong, its a different commit that is 1.8-only, sorry.

Ah cool! Emeric already spent a few days trying to sort out this
really painful one, I feared to find him hung in the office after
discovering he has to do it again :-)  So William that's OK for 1.7.

> The commit is 019489 (MAJOR: task: task scheduler rework).

It could very well make sense especially on certain corner cases where
tasks don't quit cleanly. I insisted that this one was merged early so
that potential issues emerge. The other sensitive one is the scheduler
applet rework (no crash expected but possibly unexpected freezes in
bogus applets).

> I hope I got the right one this time (I don't know why, but I bisected to the
> wrong commit twice now).

It happens to me all the time. You press up arrow enter and realize too
late you got "good" or "bad" wrong.

> root@www:/usr/sbin# haproxy -vv
> HA-Proxy version 1.8-dev2-019489-36 2017/06/27
> Copyright 2000-2017 Willy Tarreau 
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
>   OPTIONS = USE_GETADDRINFO=1 USE_SLZ=1 USE_OPENSSL=1 USE_PCRE=1 
> USE_PCRE_JIT=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with OpenSSL version : OpenSSL 1.1.0g-dev  xx XXX 
> Running on OpenSSL version : OpenSSL 1.1.0g-dev  xx XXX 
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
> IP_FREEBIND
> Built with network namespace support.
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"), 
> raw-deflate("deflate"), gzip("gzip")
> Encrypted password support via crypt(3): yes
> Built with PCRE version : 8.38 2015-11-23
> Running on PCRE version : 8.38 2015-11-23
> PCRE library supports JIT : yes
> 
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> 
> root@www:/usr/sbin# gdb -ex=r --args haproxy -f /etc/haproxy/haproxy-slim.cfg 
> -d
> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from haproxy...done.
> Starting program: /usr/sbin/haproxy -f /etc/haproxy/haproxy-slim.cfg -d
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> [WARNING] 184/22 (30237) : config : log format ignored for frontend 
> 'tls-termination' since it has no log address.
> [WARNING] 184/22 (30237) : Proxy 'tls-termination': no-sslv3/no-tlsv1x 
> are ignored for bind ':443' at [/etc/haproxy/haproxy-slim.cfg:18]. Use only 
> 'ssl-min-ver' and 'ssl-max-ver' to fix.
> Note: setting global.maxconn to 2000.
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result FAILED
> Total: 3 (2 usable), will use epoll.
> 
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> Using epoll() as the polling mechanism.
> :tls-termination.accept(0004)=0005 from [10.0.0.4:56156]
> :tls-termination.clireq[0005:]: GET /robots.txx HTTP/1.1
> 

Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Lukas Tribus]

Hi Willy,


Am 04.07.2017 um 22:24 schrieb Willy Tarreau:
> Hi Lukas,
>
> On Tue, Jul 04, 2017 at 09:56:09PM +0200, Lukas Tribus wrote:
>> Hi Emeric,
>>
>>
>> since 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show
>> map/acl' on cli") my setup crashes when a request comes in
>> going through SSL termination.
>>
>> memory corruption, invalid pointers, double free is what haproxy
>> randomly crashes with.
> Hmmm bad! Do you want me to revert it now or can this wait for
> Emeric to try to spot the issue ? William, I've seen you've already
> backported it into 1.7, be careful not to release the next version
> before this one is fixed!

No, I got it wrong, its a different commit that is 1.8-only, sorry.


The commit is 019489 (MAJOR: task: task scheduler rework).
I hope I got the right one this time (I don't know why, but I bisected to the
wrong commit twice now).



root@www:/usr/sbin# haproxy -vv
HA-Proxy version 1.8-dev2-019489-36 2017/06/27
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
  OPTIONS = USE_GETADDRINFO=1 USE_SLZ=1 USE_OPENSSL=1 USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0g-dev  xx XXX 
Running on OpenSSL version : OpenSSL 1.1.0g-dev  xx XXX 
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : yes

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

root@www:/usr/sbin# gdb -ex=r --args haproxy -f /etc/haproxy/haproxy-slim.cfg -d
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from haproxy...done.
Starting program: /usr/sbin/haproxy -f /etc/haproxy/haproxy-slim.cfg -d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[WARNING] 184/22 (30237) : config : log format ignored for frontend 
'tls-termination' since it has no log address.
[WARNING] 184/22 (30237) : Proxy 'tls-termination': no-sslv3/no-tlsv1x are 
ignored for bind ':443' at [/etc/haproxy/haproxy-slim.cfg:18]. Use only 
'ssl-min-ver' and 'ssl-max-ver' to fix.
Note: setting global.maxconn to 2000.
Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Using epoll() as the polling mechanism.
:tls-termination.accept(0004)=0005 from [10.0.0.4:56156]
:tls-termination.clireq[0005:]: GET /robots.txx HTTP/1.1
:tls-termination.clihdr[0005:]: Host: temp.lan.ltri.eu
:tls-termination.clihdr[0005:]: User-Agent: curl/7.48.0
:tls-termination.clihdr[0005:]: Accept: */*
*** Error in `/usr/sbin/haproxy': malloc(): memory corruption: 
0x00a419c0 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x771bb7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x771c613e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x771c8184]
/usr/sbin/haproxy[0x523d66]
/usr/sbin/haproxy[0x522154]
/usr/sbin/haproxy[0x522732]
/usr/sbin/haproxy[0x532480]
/usr/sbin/haproxy[0x4120b6]
/usr/sbin/haproxy[0x4d89dd]
/usr/sbin/haproxy[0x4d9a95]
/usr/sbin/haproxy[0x4fa433]
/usr/sbin/haproxy[0x5116a2]
/usr/sbin/haproxy[0x4d4a75]
/usr/sbin/haproxy[0x4d5d67]

Re: ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Willy Tarreau]

Hi Lukas,

On Tue, Jul 04, 2017 at 09:56:09PM +0200, Lukas Tribus wrote:
> Hi Emeric,
> 
> 
> since 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show
> map/acl' on cli") my setup crashes when a request comes in
> going through SSL termination.
> 
> memory corruption, invalid pointers, double free is what haproxy
> randomly crashes with.

Hmmm bad! Do you want me to revert it now or can this wait for
Emeric to try to spot the issue ? William, I've seen you've already
backported it into 1.7, be careful not to release the next version
before this one is fixed!

Thanks,
Willy

ssl: crashing since 8d85aa (BUG/MAJOR: map: fix segfault ...)

[Lukas Tribus]

Hi Emeric,


since 8d85aa4 ("BUG/MAJOR: map: fix segfault during 'show
map/acl' on cli") my setup crashes when a request comes in
going through SSL termination.

memory corruption, invalid pointers, double free is what haproxy
randomly crashes with.


Here 2 crashes with full backtrace:

*** Error in `/usr/sbin/haproxy': double free or corruption (!prev): 
0x00a42590 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x771bb7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x771c437a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x771c853c]
/usr/sbin/haproxy[0x53a64e]
/usr/sbin/haproxy[0x53630b]
/usr/sbin/haproxy[0x4124de]
/usr/sbin/haproxy[0x48103a]
/usr/sbin/haproxy[0x482f09]
/usr/sbin/haproxy[0x4891af]
/usr/sbin/haproxy[0x50910f]
/usr/sbin/haproxy[0x4d5159]
/usr/sbin/haproxy[0x4d64ba]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x77164830]
/usr/sbin/haproxy[0x4055a9]
=== Memory map: 
0040-007af000 r-xp  ca:02 40972  
/usr/sbin/haproxy
009af000-009cf000 r--p 003af000 ca:02 40972  
/usr/sbin/haproxy
009cf000-009eb000 rw-p 003cf000 ca:02 40972  
/usr/sbin/haproxy
009eb000-00ac5000 rw-p  00:00 0  [heap]
7000-70021000 rw-p  00:00 0
70021000-7400 ---p  00:00 0
76b0e000-76b24000 r-xp  ca:02 24641  
/lib/x86_64-linux-gnu/libgcc_s.so.1
76b24000-76d23000 ---p 00016000 ca:02 24641  
/lib/x86_64-linux-gnu/libgcc_s.so.1
76d23000-76d24000 rw-p 00015000 ca:02 24641  
/lib/x86_64-linux-gnu/libgcc_s.so.1
76d24000-77144000 rw-p  00:00 0
77144000-77304000 r-xp  ca:02 26350  
/lib/x86_64-linux-gnu/libc-2.23.so
77304000-77504000 ---p 001c ca:02 26350  
/lib/x86_64-linux-gnu/libc-2.23.so
77504000-77508000 r--p 001c ca:02 26350  
/lib/x86_64-linux-gnu/libc-2.23.so
77508000-7750a000 rw-p 001c4000 ca:02 26350  
/lib/x86_64-linux-gnu/libc-2.23.so
7750a000-7750e000 rw-p  00:00 0
7750e000-77526000 r-xp  ca:02 24805  
/lib/x86_64-linux-gnu/libpthread-2.23.so
77526000-77725000 ---p 00018000 ca:02 24805  
/lib/x86_64-linux-gnu/libpthread-2.23.so
77725000-77726000 r--p 00017000 ca:02 24805  
/lib/x86_64-linux-gnu/libpthread-2.23.so
77726000-77727000 rw-p 00018000 ca:02 24805  
/lib/x86_64-linux-gnu/libpthread-2.23.so
77727000-7772b000 rw-p  00:00 0
7772b000-77799000 r-xp  ca:02 24672  
/lib/x86_64-linux-gnu/libpcre.so.3.13.2
77799000-77999000 ---p 0006e000 ca:02 24672  
/lib/x86_64-linux-gnu/libpcre.so.3.13.2
77999000-7799a000 r--p 0006e000 ca:02 24672  
/lib/x86_64-linux-gnu/libpcre.so.3.13.2
7799a000-7799b000 rw-p 0006f000 ca:02 24672  
/lib/x86_64-linux-gnu/libpcre.so.3.13.2
7799b000-7799e000 r-xp  ca:02 26330  
/lib/x86_64-linux-gnu/libdl-2.23.so
7799e000-77b9d000 ---p 3000 ca:02 26330  
/lib/x86_64-linux-gnu/libdl-2.23.so
77b9d000-77b9e000 r--p 2000 ca:02 26330  
/lib/x86_64-linux-gnu/libdl-2.23.so
77b9e000-77b9f000 rw-p 3000 ca:02 26330  
/lib/x86_64-linux-gnu/libdl-2.23.so
77b9f000-77ba8000 r-xp  ca:02 24741  
/lib/x86_64-linux-gnu/libcrypt-2.23.so
77ba8000-77da7000 ---p 9000 ca:02 24741  
/lib/x86_64-linux-gnu/libcrypt-2.23.so
77da7000-77da8000 r--p 8000 ca:02 24741  
/lib/x86_64-linux-gnu/libcrypt-2.23.so
77da8000-77da9000 rw-p 9000 ca:02 24741  
/lib/x86_64-linux-gnu/libcrypt-2.23.so
77da9000-77dd7000 rw-p  00:00 0
77dd7000-77dfd000 r-xp  ca:02 24651  
/lib/x86_64-linux-gnu/ld-2.23.so
77fec000-77ff rw-p  00:00 0
77ff5000-77ff8000 rw-p  00:00 0
77ff8000-77ffa000 r--p  00:00 0  [vvar]
77ffa000-77ffc000 r-xp  00:00 0  [vdso]
77ffc000-77ffd000 r--p 00025000 ca:02 24651  
/lib/x86_64-linux-gnu/ld-2.23.so
77ffd000-77ffe000 rw-p 00026000 ca:02 24651  
/lib/x86_64-linux-gnu/ld-2.23.so
77ffe000-77fff000 rw-p  00:00 0
7fede000-7000 rw-p  00:00 0  [stack]
ff60-ff601000 r-xp  00:00 0  
[vsyscall]

Program received 

Re: Any more detail on the plan for multi-threading and or multi-process support?

[Willy Tarreau]

Hi Malcolm,

On Tue, Jul 04, 2017 at 03:27:51PM +0100, Malcolm Turnbull wrote:
> It looks like some exciting stuff is on the way in haproxy-1.8-dev!

Let's hope so!

> Could I ask for a bit more detail on the plans for multi-threading or
> multi-process options?

I'll respond myself so that it doesn't divert them too long from their
keyboards (Emeric & Chris, feel free to correct me if you disagree).

> For the vast majority of HTTP applications, binding one cluster to one
> front end & process is fine.
> 
> But for high traffic sites on newer hardware (Think 24 cores & 2GHz -
> Not 4 core & 4GHz) you can quickly run into performance issues.
> 
> Will it be possible for one front end to access one stick table - and
> yet use multiple threads or processes?

That's essentially the idea behind this. The goal with threads is to go
back to the unified memory model we used to have with nbproc=1 for over
a decade where everything has access to everything, but at the same time
have the ability to scale on newer CPUs.

Over the long term, the goal is to be able to assign a thread mask to
all bind lines (like we currently do with processes), and that accepted
connections try to stay affine to the thread that created them, unless
there's a significant load unbalance or there are certain painful tasks
to deal with (like SSL handshake) in which case a thread migration could
be decided for certain sessions/streams/connections/you-name-it. Doing
this will ensure that all assigned cores can be used in parallel, and
that workloads sensitive to latency can continue to dedicate some cores
to low-latency stuff and queue the high latency stuff on other cores.
For the mid-term, we're doing baby steps. Thread migration for SSL might
definitely not be the first thing implemented for example!

> And if it is planned we'd be very interested in helping with testing
> and or code.

I hope that the guys will "soon" (ie before the 1.8 release) have something
to show and share. At the moment they're still focused on reverse-engineering
a lot of stuff accumulated over the last 16 years which sometimes has some
strong impacts and causes some fairly long head-scratching sessions. But
eventually all limitations are addressed one at a time and the progress is
encouraging. I've seen some traffic flow through haproxy running on multiple
cores, with a bunch of stuff disabled, but this is promising.

> Sorry if this has already been discussed and I missed the conversation.

No it was not really discussed, primarly to avoid putting pressure on the
team so that all options could sanely be considered. We've even had some
false starts due to me discussing too much with them and accidently driving
them into corners. I suspect that after the holidays they'll start to be
more confident in their current code (ie will not throw all of it to the
trash again) and may start to discuss about how it works :-)

Stay tuned!
Willy

Any more detail on the plan for multi-threading and or multi-process support?

[Malcolm Turnbull]

It looks like some exciting stuff is on the way in haproxy-1.8-dev!

Could I ask for a bit more detail on the plans for multi-threading or
multi-process options?
For the vast majority of HTTP applications, binding one cluster to one
front end & process is fine.

But for high traffic sites on newer hardware (Think 24 cores & 2GHz -
Not 4 core & 4GHz) you can quickly run into performance issues.

Will it be possible for one front end to access one stick table - and
yet use multiple threads or processes?

And if it is planned we'd be very interested in helping with testing
and or code.

Sorry if this has already been discussed and I missed the conversation.


--
Regards,

Malcolm Turnbull

Loadbalancer.org Ltd.
 +44 (0)330 380 1064

Re: Logging TLVs from PROXY protocol v2

[Willy Tarreau]

On Tue, Jul 04, 2017 at 07:17:05PM +1000, Marc Boschma wrote:
> My circumstance is that I have an environment of a very large enterprise
> inter-network with a large number of intranets (many with overlapping address
> space) that need to be proxied through a series of proxies to a shared /
> inter-network service.Being able to track / log flows across the series of
> proxies and or knowing the origin of a request is very useful from an
> operations perspective. 
> 
> There is also a security requirement to track the client source IP address
> and origin network - thus the interest in the namespace TLV which in our
> circumstance would represent the origin network.

Actually this makes a lot of sense, and I didn't consider the case where
you have multiple networks with the same addresses (which is a common problem
for large enterprise networks and partner VPNs). It's true that a proxy (or
a series of proxies) sometimes make it possible to interconnect overlapping
networks. And a single address has a different meaning for different nodes,
so being able to transmit sort of a node ID or network ID to be used as a
namespace makes quite some sense. Now one of the issues is what to do with
such information once relayed between proxies. Ideally only the first one
filling the entry should set this element and other ones should simply pass
it unmodified. This could be seen as some extension to the existing namespace
system (including internally in the connections) to be able to deal with an
element set on the bind lines for example. For example if we had a "netid"
parameter on the bind line, we could imagine logging it and passing it in
headers along with the source address.

Maybe adding a 32-bit field into struct connection to store such an element
would be a good start. I was thinking about abusing the existing ns_entry
for this until I realized that containers might overlap between machines and
furthermore require having their own netid as well.

So in the end I tend to like your idea :-)

Willy

Re: Logging TLVs from PROXY protocol v2

[Marc Boschma]

Thanks Willy for responding. I’ll read over the code soon to see if it makes 
sense to expose some of the TLVs that aren’t already.

I shared your residence to abuse the protocol and maintain transparency.

My circumstance is that I have an environment of a very large enterprise 
inter-network with a large number of intranets (many with overlapping address 
space) that need to be proxied through a series of proxies to a shared / 
inter-network service.Being able to track / log flows across the series of 
proxies and or knowing the origin of a request is very useful from an 
operations perspective. 

There is also a security requirement to track the client source IP address and 
origin network - thus the interest in the namespace TLV which in our 
circumstance would represent the origin network.

The service right now isn’t proxy protocol aware, but that is being explored to 
pass the origin information into the service for security policies.

I’m trying to avoid terminating the TLS or otherwise interfere with it since 
most flows have client certificates. Server certificate replication is fairly 
simple, however a large number of client certificate proxying isn’t somewhere 
I’d want to go.

Regards,

Marc


> On 4 Jul 2017, at 6:12 pm, Willy Tarreau  wrote:
> 
> On Mon, Jul 03, 2017 at 10:11:29PM +1000, Marc Boschma wrote:
>> Is there anyway to log details from the PROXY protocol version 2, such as
>> namespace or any other TLVs? Especially custom...
>> 
>> On TLVs I'm looking at someway to pass through a unique ID across a series
>> of proxies. So being able to inspect TLV and or passed in namespace values
>> and make processing decisions based on that; even if having to utilise Lua
>> to effect the inspection...
> 
> There's no such thing for now. The proxy protocol is used to pass front
> connection information to the next hop, so in general, the next hop will
> replace its own information extraction methods with the information
> retrieved from the proxy protocol header. It's possible that haproxy is
> still missing certain sample fetch methods to extract certain information
> (I don't think we can extract a namespace ID for example, but I could be
> wrong ; maybe that just needs to be added). So once the information is
> properly available, what you pass over proxy protocol should be
> transparently available.
> 
> Now, regarding the ability to pass extra information, I'm not fond of this
> idea. Not for the idea itself, it would be great. It's regarding the purpose
> of the proxy protocol. It's not meant to be used to transport metadata for a
> connection, just to forward connection information. So in general you should
> keep in mind that it must not do something that would not be possible without
> the proxy in the middle, if the server would simply receive the original
> connection itself. If we start to do this, we'll see people abuse this
> protocol and use it everywhere as an intermediary layer to carry extra
> information and that opens a new can of worms.
> 
> Regards,
> Willy

Re: Fix building haproxy with recent LibreSSL

[Willy Tarreau]

On Tue, Jul 04, 2017 at 11:12:20AM +0300, Dmitry Sivachenko wrote:
> >> https://www.mail-archive.com/haproxy@formilux.org/msg25819.html
> > 
> > 
> > Do you know if the patch applies to 1.8 (it was mangled so I didn't try).
> 
> 
> Sorry, hit reply too fast:  no, one chunk fails against 1.8-dev2 (the one
> dealing with #ifdef SSL_CTX_get_tlsext_status_arg, it requires analysis
> because it is not simple surrounding context change).

OK thanks. Bernard, care to have a look and ensure it works for you ?

Thanks,
Willy

Re: Fix building haproxy with recent LibreSSL

[Willy Tarreau]

On Tue, Jul 04, 2017 at 11:07:31AM +0300, Dmitry Sivachenko wrote:
> On FreeBSD it does fix a build (though new warning appear which I can't 
> explain because of the lack of SSL knowledge):
> 
> src/ssl_sock.c:803:2: warning: incompatible integer to pointer conversion
>  assigning to 'void (*)(void)' from 'long' [-Wint-conversion]
>SSL_CTX_get_tlsext_status_cb(ctx, );
>^~~~
> src/ssl_sock.c:801:6: note: expanded from macro 'SSL_CTX_get_tlsext_status_cb'
>  ...= SSL_CTX_ctrl(ctx,SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB,0, (void 
> (**)(void))cb)
> ^ 
> ~~
> 1 warning generated.

According to the man page, SSL_CTX_ctrl() returns a long. So the patch should
be changed so that the macro is defined this way :

#define SSL_CTX_get_tlsext_status_cb(ctx, cb) \
   *cb = (void *)SSL_CTX_ctrl(ctx,SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB,0, (void
(**)(void))cb)

> The patch was taken form OpenBSD, so in general it should be fine.

Aside the added warning ;-)

> Review from some SSL-aware guys on your side would be nice.

I'll ping Emeric about it, but I prefer to ensure it works before, it
will reduce his sighing ;-)

Cheers,
willy

Re: Fix building haproxy with recent LibreSSL

[Dmitry Sivachenko]

> On 04 Jul 2017, at 11:04, Willy Tarreau  wrote:
> 
> Hi Dmitry,
> 
> [CCing Bernard, the  patch's author]
> 
> On Mon, Jul 03, 2017 at 12:34:52AM +0300, Dmitry Sivachenko wrote:
>> Hello,
>> 
>> can you please take a look at proposed patch to fix build of haproxy with
>> recent version of LibreSSL?
>> 
>> https://www.mail-archive.com/haproxy@formilux.org/msg25819.html
> 
> 
> Do you know if the patch applies to 1.8 (it was mangled so I didn't try).


Sorry, hit reply too fast:  no, one chunk fails against 1.8-dev2 (the one 
dealing with #ifdef SSL_CTX_get_tlsext_status_arg, it requires analysis because 
it is not simple surrounding context change).

Re: Logging TLVs from PROXY protocol v2

[Willy Tarreau]

On Mon, Jul 03, 2017 at 10:11:29PM +1000, Marc Boschma wrote:
> Is there anyway to log details from the PROXY protocol version 2, such as
> namespace or any other TLVs? Especially custom...
> 
> On TLVs I'm looking at someway to pass through a unique ID across a series
> of proxies. So being able to inspect TLV and or passed in namespace values
> and make processing decisions based on that; even if having to utilise Lua
> to effect the inspection...

There's no such thing for now. The proxy protocol is used to pass front
connection information to the next hop, so in general, the next hop will
replace its own information extraction methods with the information
retrieved from the proxy protocol header. It's possible that haproxy is
still missing certain sample fetch methods to extract certain information
(I don't think we can extract a namespace ID for example, but I could be
wrong ; maybe that just needs to be added). So once the information is
properly available, what you pass over proxy protocol should be
transparently available.

Now, regarding the ability to pass extra information, I'm not fond of this
idea. Not for the idea itself, it would be great. It's regarding the purpose
of the proxy protocol. It's not meant to be used to transport metadata for a
connection, just to forward connection information. So in general you should
keep in mind that it must not do something that would not be possible without
the proxy in the middle, if the server would simply receive the original
connection itself. If we start to do this, we'll see people abuse this
protocol and use it everywhere as an intermediary layer to carry extra
information and that opens a new can of worms.

Regards,
Willy

Re: Fix building haproxy with recent LibreSSL

[Dmitry Sivachenko]

> On 04 Jul 2017, at 11:04, Willy Tarreau  wrote:
> 
> Hi Dmitry,
> 
> [CCing Bernard, the  patch's author]
> 
> On Mon, Jul 03, 2017 at 12:34:52AM +0300, Dmitry Sivachenko wrote:
>> Hello,
>> 
>> can you please take a look at proposed patch to fix build of haproxy with
>> recent version of LibreSSL?
>> 
>> https://www.mail-archive.com/haproxy@formilux.org/msg25819.html
> 
> I personally have no opinion on this one, as long as it doesn't break the
> build for other versions. Do you see the problem on your FreeBSD builds ?
> Do you know if the patch applies to 1.8 (it was mangled so I didn't try).
> We could relatively easily apply Bernard's patch as his description can
> be used as a commit message.



On FreeBSD it does fix a build (though new warning appear which I can't explain 
because of the lack of SSL knowledge):

src/ssl_sock.c:803:2: warning: incompatible integer to pointer conversion
 assigning to 'void (*)(void)' from 'long' [-Wint-conversion]
   SSL_CTX_get_tlsext_status_cb(ctx, );
   ^~~~
src/ssl_sock.c:801:6: note: expanded from macro 'SSL_CTX_get_tlsext_status_cb'
 ...= SSL_CTX_ctrl(ctx,SSL_CTRL_GET_TLSEXT_STATUS_REQ_CB,0, (void (**)(void))cb)
^ ~~
1 warning generated.


The patch was taken form OpenBSD, so in general it should be fine.

Review from some SSL-aware guys on your side would be nice.

Re: [PATCH] DOC: fix references to the section about time format

[Willy Tarreau]

On Tue, Jul 04, 2017 at 10:50:59AM +0300, Jarno Huuskonen wrote:
> Hi,
> 
> Small patch updating configuration.txt stick table time format(section
> 2.4) cross references.

Applied, thank you Jarno!

Willy

Re: Fix building haproxy with recent LibreSSL

[Willy Tarreau]

Hi Dmitry,

[CCing Bernard, the  patch's author]

On Mon, Jul 03, 2017 at 12:34:52AM +0300, Dmitry Sivachenko wrote:
> Hello,
> 
> can you please take a look at proposed patch to fix build of haproxy with
> recent version of LibreSSL?
> 
> https://www.mail-archive.com/haproxy@formilux.org/msg25819.html

I personally have no opinion on this one, as long as it doesn't break the
build for other versions. Do you see the problem on your FreeBSD builds ?
Do you know if the patch applies to 1.8 (it was mangled so I didn't try).
We could relatively easily apply Bernard's patch as his description can
be used as a commit message.

Thanks,
Willy

[PATCH] DOC: fix references to the section about time format

[Jarno Huuskonen]

Hi,

Small patch updating configuration.txt stick table time format(section
2.4) cross references.

-Jarno

-- 
Jarno Huuskonen
>From 32a999d2f41cf57eae479c6d4c6514a847d9d788 Mon Sep 17 00:00:00 2001
From: Jarno Huuskonen 
Date: Tue, 4 Jul 2017 10:35:12 +0300
Subject: [PATCH] DOC: fix references to the section about time format.

Time format is documented in section 2.4, not 2.2.
---
 doc/configuration.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index d7ffd3f..7cac3ca 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -8519,7 +8519,7 @@ stick-table type {ip | integer | string [len ] | 
binary [len ]}
was last created, refreshed or matched. The expiration delay is
defined using the standard time format, similarly as the various
timeouts. The maximum duration is slightly above 24 days. See
-   section 2.2 for more information. If this delay is not 
specified,
+   section 2.4 for more information. If this delay is not 
specified,
the session won't automatically expire, but older entries will
be removed once full. Be sure not to use the "nopurge" parameter
if not expiration delay is specified.
@@ -8667,7 +8667,7 @@ stick-table type {ip | integer | string [len ] | 
binary [len ]}
 # computed over a sliding window of 30 seconds.
 stick-table type ip size 1m expire 5m store gpc0,conn_rate(30s)
 
-  See also : "stick match", "stick on", "stick store-request", section 2.2
+  See also : "stick match", "stick on", "stick store-request", section 2.4
  about time format and section 7 about ACLs.
 
 
-- 
1.8.3.1

Re: Restrict HAProxy stats page to a specific IP

[Jarno Huuskonen]

Hi,

On Mon, Jul 03, Kaushal Shriyan wrote:
> Thanks for the email and it is working as expected. I will appreciate if
> you let me know to allow multiple office IP address.
> 
> For example http://IP1:1936 and http://IP2:1936
> 
> http://IP1:1936 -> http://192.168.1.10:1936
> http://IP2:1936 -> http://192.168.1.20:1936

Not quite sure what you're asking, but it's possible to have multiple
addresses/networks in acl:
acl office_addr src officeIP officeIP2
(http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-acl
http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#7)

-Jarno

-- 
Jarno Huuskonen

Re: Reg: HAProxy 1.6.12 on RHEL7.2 (MAXCONN in FRONT-END/LISTEN BLOCK)

[Jarno Huuskonen]

Hi,

On Tue, Jul 04, Igor Cicimov wrote:
> On Tue, Jul 4, 2017 at 1:34 PM, Velmurugan Dhakshnamoorthy <
> dvel@gmail.com> wrote:
> 
> > Thanks much for detailed explanation.
> >
> > Once the limit of 100 sessions are reached, note we are talking about *100
> > sessions in Weblogic* and *NOT 100 connections to the backend*, what is
> > the Weblogic server going to do? We need to understand what happens on
> > Weblogic side once the 101st session is accepted. You get error 500
> > straight away or something else happens? Maybe nothing and the request gets
> > dropped after sitting in the Weblogic queue for some time?
> >
> > [Vel] once the limit(100) is reached in weblogic, 101 user will receive
> > error 500, OOM (OutOfMeory) error in weblogic back-end.when there is OOM
> > occurs, even connected users responses will be impacted.
> >
> > Regards,
> > Vel
> >
> >
> ​Well, the biggest issue you have is that number or connections is not the
> same as number of sessions. Lets say you have reached your 100 connections
> limit which corresponds to 100 sessions in WL and one client's browser
> starts closing its connections. For HAP the number of connections will drop
> below 100, lets say to 96, and it is possible that in that moment a new
> user gets connected which will cause WL to create a new 101th session and
> crash. So how are you going to solve this dependency between connections on
> the HAP side and sessions in the WL side?
> 
> Another thing is that modern browsers can open up to 6-7 connections to a
> single domain name which potentially leaves you with less than ​20 users
> during the overload period with the limitation of 100 connections in HAP.
> Also depending on the users activity, every time the user is active WL will
> restart the session timer to 20 min, you might end up with less than 20
> users connected for a long long time.

What "might" work is track(stick table) the session cookie WL uses for
logged in users (does WL use cookie (JSESSIONID?)) and when
table_cnt gt 100 send users custom errorfile.

It's probaly quite tricky to remove entry from stick table when user
clicks logout (instead of just waiting for 20min stick table entry to
timeout).

-Jarno

-- 
Jarno Huuskonen - System Administrator |  jarno.huuskonen atsign uef.fi