url_param not matching key-only params (also testcases for fetchers)

[Robin H. Johnson]

I looked in tests & reg-tests, but didn't see any clear way to add tests
for verifying that fetchers work correctly.

I think my co-worker found an edge-case on smp_fetch_url_param/smp_fetch_param.

Trying to identify URLs that have a URL parameter set, that MIGHT not
have a value. This is commonly found in AWS S3 sub-resource URLs.

Sample inputs that are all expected to match below.

# Desired code:
acl bucket_website_crud urlp(website) -m found

# Should_match, Input
1,/?website
1,/?website=a
1,/?website&
1,/?website
1,/?website=a
1,/?website
1,/?website=1
1,/?website=a=1
1,/?website=1
1,/?foo
1,/?foo=a
1,/?foo&
1,/?foo=
1,/?foo==a
1,/?foo=&

# Workaround:
acl bucket_website_crud url ?website
acl bucket_website_crud url 

-- 
Robin Hugh Johnson
E-Mail : robb...@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85


signature.asc
Description: Digital signature

haproxy 1.8.12 / 1.9- 20180623 / stopping process hangs with threads (100% cpu) on -sf reload / FreeBSD

[PiBa-NL]

Hi List,

With a build of 1.8.12 (and the 1.9 snapshot of 20180623 ) im getting 
the 'old' haproxy process take up 100% cpu usage when using 3 threads in 
the config and reloading with -sf parameter. I'm using FreeBSD.. (It 
also happens with the 14-7 snapshot.)


It seems to happen after 1 thread quits, one of the others gets out of 
control.

Most of the time it happens after the first reload.:
haproxy -f /var/etc/haproxy/haproxy.cfg -D
haproxy -f /var/etc/haproxy/haproxy.cfg -D -sf 19110

The main features i use are:  ssl offloading / lua / threads
Only a little to no traffic passing through though, im seeing this 
behavior also on my in-active production node, the strange part sofar 
though is that i could not reproduce it yet on my test machine.
If someone has got a idea on how to patch or what direction to search 
for a fix i'm happy to try.


If there is nothing obvious that can be spotted with the info from the 
stack-traces of both 1.8 and 1.9 below ill try and dig further tomorrow 
:).Thanks in advance for anyone's time :).


Regards,
PiBa-NL (Pieter)

p.s.
I CC'ed Christopher as he seems to have made the last 2 patches going 
into 20180623. Im hoping he has a clue on what to do next :).


## Below stack traces are from 1.8.12.. ##

[2.4.3-RELEASE][admin@pfsense_3]/root: /usr/local/bin/gdb --pid 2136 
/usr/local/sbin/haproxy

GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/haproxy...done.
Attaching to program: /usr/local/sbin/haproxy, process 2136
[New LWP 101099 of process 2136]
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols 
found)...done.

Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libssl.so.8...(no debugging symbols 
found)...done.
Reading symbols from /lib/libcrypto.so.8...(no debugging symbols 
found)...done.
Reading symbols from /usr/local/lib/liblua-5.3.so...(no debugging 
symbols found)...done.

Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols 
found)...done.

[Switching to LWP 100345 of process 2136]
0x000800f0f91c in ?? () from /lib/libthr.so.3
(gdb) thread info
Invalid thread ID: info
(gdb) info thread
  Id   Target Id Frame
* 1    LWP 100345 of process 2136 0x000800f0f91c in ?? () from 
/lib/libthr.so.3
  2    LWP 101099 of process 2136 thread_sync_barrier (barrier=0x8bc4e0 
) at src/hathreads.c:112

(gdb) bt full
#0  0x000800f0f91c in ?? () from /lib/libthr.so.3
No symbol table info available.
#1  0x000800f0bf97 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2  0x0050bd4f in main (argc=6, argv=0x7fffec70) at 
src/haproxy.c:3077

    tids = 0x80249cf40
    threads = 0x802487240
    i = 2
    old_sig = {__bits = {1073741824, 0, 0, 0}}
    blocked_sig = {__bits = {4227856759, 4294967295, 4294967295, 
4294967295}}

    err = 0
    retry = 200
    limit = {rlim_cur = 2282, rlim_max = 2282}
    errmsg = 
"\000\354\377\377\377\177\000\000\250\354\377\377\377\177\000\000p\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\270\030\241\374+/\213\032`e\213\000\000\000\000\000h\354\377\377\377\177\000\000\250\354\377\377\377\177\000\000p\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\020\354\377\377\377\177\000\000r3\340\001\b\000\000\000\001\000\000"

    pidfd = 27
(gdb) thread 2
[Switching to thread 2 (LWP 101099 of process 2136)]
#0  thread_sync_barrier (barrier=0x8bc4e0 ) at 
src/hathreads.c:112

112 src/hathreads.c: No such file or directory.
(gdb) bt full
#0  thread_sync_barrier (barrier=0x8bc4e0 ) at 
src/hathreads.c:112

    old = 7
#1  0x005ae23f in thread_exit_sync () at src/hathreads.c:151
    barrier = 7
#2  0x0051260d in sync_poll_loop () at src/haproxy.c:2391
    stop = 1
#3  0x00512533 in run_poll_loop () at src/haproxy.c:2438
    next = -1636293614
    exp = -1636293614
#4  0x0050f10b in run_thread_poll_loop (data=0x80249cf48) at 
src/haproxy.c:2470
    start_lock = 

Re: Building HAProxy 1.8 fails on Solaris

[Thrawn]

 Ah, indeed, the GCC version provided on our server is 3.4.3. But the readme on 
https://github.com/haproxy/haproxy says "GCC between 2.95 and 4.8". Can the 
build be changed to continue supporting older GCC, or do the docs need an 
update?
Thanks for the quick help.

On Monday, 16 July 2018, 8:58:40 pm AEST, Lukas Tribus  
wrote:  
 
 Hello,


On Mon, 16 Jul 2018 at 03:12, Thrawn  wrote:
>
> Update: If I disable threading with
>
> USE_THREAD=
>
> then the build gets much further, but still fails eventually with:
>
> gcc  -g -o haproxy src/ev_poll.o ebtree/ebtree.o ebtree/eb32sctree.o 
> ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o 
> ebtree/ebimtree.o ebtree/ebistree.o src/proto_http.o src/cfgparse.o 
> src/server.o src/stream.o src/flt_spoe.o src/stick_table.o src/stats.o 
> src/mux_h2.o src/checks.o src/haproxy.o src/log.o src/dns.o src/peers.o 
> src/standard.o src/sample.o src/cli.o src/stream_interface.o src/proto_tcp.o 
> src/backend.o src/proxy.o src/tcp_rules.o src/listener.o src/flt_http_comp.o 
> src/pattern.o src/cache.o src/filters.o src/vars.o src/acl.o src/payload.o 
> src/connection.o src/raw_sock.o src/proto_uxst.o src/flt_trace.o 
> src/session.o src/ev_select.o src/channel.o src/task.o src/queue.o 
> src/applet.o src/map.o src/frontend.o src/freq_ctr.o src/lb_fwlc.o 
> src/mux_pt.o src/auth.o src/fd.o src/hpack-dec.o src/memory.o src/lb_fwrr.o 
> src/lb_chash.o src/lb_fas.o src/hathreads.o src/chunk.o src/lb_map.o 
> src/xxhash.o src/regex.o src/shctx.o src/buffer.o src/action.o src/h1.o 
> src/compression.o src/pipe.o src/namespace.o src/sha1.o src/hpack-tbl.o 
> src/hpack-enc.o src/uri_auth.o src/time.o src/proto_udp.o src/arg.o 
> src/signal.o src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o 
> src/mailers.o src/h2.o src/base64.o src/hash.o -lnsl -lsocket  -lcrypt 
> -L../pcre/lib -Wl,-Bstatic -lpcreposix -lpcre -Wl,-Bdynamic
> Undefined                      first referenced
>  symbol                            in file
> __sync_sub_and_fetch                src/cache.o
> __sync_val_compare_and_swap        src/cache.o
> __sync_lock_test_and_set            src/cache.o
> ld: fatal: symbol referencing errors. No output written to haproxy
> collect2: ld returned 1 exit status
> gmake: *** [haproxy] Error 1

Are you sure you did a "make clean" before building with USE_THREAD= ?


Also please provide the output of "gcc --version".


lukas


On Monday, 16 July 2018, 11:12:34 pm AEST, Olivier Houchard 
 wrote:

Hi,

I think your gcc is just too old. Those appeared around gcc 4.1 or so.

Regards,

Olivier
  

Re: SSL: double free on reload

[Willy Tarreau]

Hi Thierry,

On Fri, Jul 06, 2018 at 04:28:22PM +0200, Thierry Fournier wrote:
> Hi list,
> 
> I caught a double-free whien I reload haproxy-1.8:
> 
> writev(2, [{"*** Error in `", 14}, {"/opt/o3-haproxy/sbin/haproxy", 28}, 
> {"': ", 3}, {"double free or corruption (!prev)", 33}, {": 0x", 4}, 
> {"1cec2ab0", 16}, {" ***\n", 5}], 7) = 103
> 
> Decoded:
> 
> *** Error in `/opt/o3-haproxy/sbin/haproxy': double free or corruption 
> (!prev): 0x1cec2ab0 ***
> 
> Gdb says:
> 
>#0  0x7f4bac88b067 in __GI_raise (sig=sig@entry=6) at 
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>#1  0x7f4bac88c448 in __GI_abort () at abort.c:89
>#2  0x7f4bac8c91b4 in __libc_message (do_abort=do_abort@entry=1, 
>fmt=fmt@entry=0x7f4bac9be210 "*** Error in `%s': %s: 0x%s ***\n")
>at ../sysdeps/posix/libc_fatal.c:175
>#3  0x7f4bac8ce98e in malloc_printerr (action=1, 
>str=0x7f4bac9be318 "double free or corruption (!prev)", ptr= out>) at malloc.c:4996
>#4  0x7f4bac8cf696 in _int_free (av=, p= out>, have_lock=0) at malloc.c:3840
>#5  0x0042af56 in ssl_sock_destroy_bind_conf 
> (bind_conf=0x1d27e810) at src/ssl_sock.c:4819
>#6  0x004b1390 in deinit () at src/haproxy.c:2240
>#7  0x0041b83c in main (argc=, argv=0x7ffc22f6b4d8) 
> at src/haproxy.c:3094
> 
> I use the last 1.8.12 version.

This one looks a bit strange. I looked at it a little bit and it corresponds
to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is no
other line in the code appearing to perfom a free on this element, and when
passing through this code the key_ref is destroyed and properly nulled. I
checked if it was possible for this element not to be allocated and I don't
see how that could happen either. Thus I'm seeing only three possibilities :

  - this element was duplicated and appears at multiple places (multiple list
elements) leading to a real double free

  - there is a memory corruption somewhere possibly resulting in this element
being corrupted and not in fact victim of a double free

  - I can't read code and there is another free that I failed to detect.

Are you able to trigger this on a trivial config ? Maybe it only happens
when certain features you have in your config are enabled ?

willy

Re: SSL: double free on reload

[Janusz Dziemidowicz]

pon., 16 lip 2018 o 08:02 Willy Tarreau  napisał(a):
> This one looks a bit strange. I looked at it a little bit and it corresponds
> to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is no
> other line in the code appearing to perfom a free on this element, and when
> passing through this code the key_ref is destroyed and properly nulled. I
> checked if it was possible for this element not to be allocated and I don't
> see how that could happen either. Thus I'm seeing only three possibilities :
>
>   - this element was duplicated and appears at multiple places (multiple list
> elements) leading to a real double free
>
>   - there is a memory corruption somewhere possibly resulting in this element
> being corrupted and not in fact victim of a double free
>
>   - I can't read code and there is another free that I failed to detect.
>
> Are you able to trigger this on a trivial config ? Maybe it only happens
> when certain features you have in your config are enabled ?

I've reported this some time ago :)
https://www.mail-archive.com/haproxy@formilux.org/msg30093.html

-- 
Janusz Dziemidowicz

Re: Using LUA to redirect a connection based-upon initial content and to redirect upon target disconnection

[thierry . fournier]

Hi,

On Sun, 15 Jul 2018 17:14:01 +0100
Alistair Lowe  wrote:

> Hi folks,
> 
> I'm looking to use LUA in HAProxy to enable two use cases:
> 
>1. Redirect a connection based upon inspecting initial traffic for a
>server name.
>2. Redirect a connection when the target server closes its connection.
> 
> My understanding is that both of these scenarios should be possible if I
> inspect a connection for its duration, however I'm also wishing to build
> something high-performance and wish for connections to remain primarily in
> TCP pass-through mode, most of the time.
> 
> My questions are:
> 
>1. What's the most CPU friendly way to inspect a connection for a
>time-limited period (i.e. what configuration and commands in LUA should I
>be looking at etc).


I'm not sure to understand your question. You want to analyse received data
before taking a decision ? the Lua receive function doesn't consume CPU while
is waiting for data. Once data are available the function receive() returns
and ou can process these data.


>2. Is there anyway to trigger a LUA action upon TCP active close from
>the target?


If you're using http, once the first bytes are send to the client, you can't
send you own response. So the only one case is when th server close without
sending anything.

Maybe you can use a "tcp-response content" Lua function and call dup() function
(https://www.arpalert.org/src/haproxy-lua-api/1.8/index.html#Channel.dup). If
you receive data, it's ok, if the channel is close you receive 'nil' reponse,
and you can send your redirect.

I'm not sure that the dup() function wait for data, and I'm not sure that
HAPRoxy call the Lua function if the client close the connection.

Thierry

Re: SSL: double free on reload

[Thierry Fournier]

On Mon, 16 Jul 2018 08:00:48 +0200
Willy Tarreau  wrote:

> Hi Thierry,
> 
> On Fri, Jul 06, 2018 at 04:28:22PM +0200, Thierry Fournier wrote:
> > Hi list,
> > 
> > I caught a double-free whien I reload haproxy-1.8:
> > 
> > writev(2, [{"*** Error in `", 14}, {"/opt/o3-haproxy/sbin/haproxy", 
> > 28}, {"': ", 3}, {"double free or corruption (!prev)", 33}, {": 0x", 4}, 
> > {"1cec2ab0", 16}, {" ***\n", 5}], 7) = 103
> > 
> > Decoded:
> > 
> > *** Error in `/opt/o3-haproxy/sbin/haproxy': double free or corruption 
> > (!prev): 0x1cec2ab0 ***
> > 
> > Gdb says:
> > 
> >#0  0x7f4bac88b067 in __GI_raise (sig=sig@entry=6) at 
> > ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> >#1  0x7f4bac88c448 in __GI_abort () at abort.c:89
> >#2  0x7f4bac8c91b4 in __libc_message (do_abort=do_abort@entry=1, 
> >fmt=fmt@entry=0x7f4bac9be210 "*** Error in `%s': %s: 0x%s ***\n")
> >at ../sysdeps/posix/libc_fatal.c:175
> >#3  0x7f4bac8ce98e in malloc_printerr (action=1, 
> >str=0x7f4bac9be318 "double free or corruption (!prev)", 
> > ptr=) at malloc.c:4996
> >#4  0x7f4bac8cf696 in _int_free (av=, p= > out>, have_lock=0) at malloc.c:3840
> >#5  0x0042af56 in ssl_sock_destroy_bind_conf 
> > (bind_conf=0x1d27e810) at src/ssl_sock.c:4819
> >#6  0x004b1390 in deinit () at src/haproxy.c:2240
> >#7  0x0041b83c in main (argc=, 
> > argv=0x7ffc22f6b4d8) at src/haproxy.c:3094
> > 
> > I use the last 1.8.12 version.
> 
> This one looks a bit strange. I looked at it a little bit and it corresponds
> to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is no
> other line in the code appearing to perfom a free on this element, and when
> passing through this code the key_ref is destroyed and properly nulled. I
> checked if it was possible for this element not to be allocated and I don't
> see how that could happen either. Thus I'm seeing only three possibilities :
> 
>   - this element was duplicated and appears at multiple places (multiple list
> elements) leading to a real double free
> 
>   - there is a memory corruption somewhere possibly resulting in this element
> being corrupted and not in fact victim of a double free
> 
>   - I can't read code and there is another free that I failed to detect.
> 
> Are you able to trigger this on a trivial config ? Maybe it only happens
> when certain features you have in your config are enabled ?


Reproduced ! unfortunately, I can't reproduce it without systemd. Check the
tls-keys path. With relative path, you must force the start path in the 
systemd config file, or give the fullpath.

The bug seems to be linked with multiple bind line. The followng has no sense,
but the bug appens (on by original conf, I use multi process, avec each bind
line is associated with one process).

Maybe each bind line is duplicated on each process, the tls-key is commun for
each lines, and double-free when the second bind try to release memory.

I guess that systemd is not a cause of the crash, but if I start the process
with -Ws on command line, and I sent kill -USER2, the bug is not trigerred.

test.cfg:
-

   global

   frontend frt
  bind *:443  ssl crt default.pem tls-ticket-keys tls-keys
  bind *:443  ssl crt default.pem tls-ticket-keys tls-keys

tls-keys


   WRGMXEZMeqZzeY7bJTLsfWvrlBKszxDuZ+2WlSP3YFOqUq4dbzBpH+8nvwforYej
   b2dwxCxZsV02/8bmEv+q/QjMllu/4bOSCYFWn6CuTtwiQExG8SLYnwBMevOUjVpL
   cOGgEy6YK4K3h8rS9jSEiu8xWjHP4iMT+IRhHkwYaKPmgwbmvARzvoPkMDnyw5gq


/lib/systemd/system/test.service:
-
   [Service]
   LimitCORE=infinity
   Environment="PIDFILE=/run/test.pid"
   WorkingDirectory=/etc/o3-haproxy
   ExecStart=/opt/o3-haproxy/sbin/haproxy -Ws -f test.cfg
   ExecReload=/bin/kill -USR2 $MAINPID


Thierry
-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

RE: TLS handshake works with certificate name mismatch using "verify required" and "verifyhost"

[Martin RADEL]

Hi,

I think we found the issue:
Seems that there was a misunderstanding from us regarding the haproxy 
documentation with the "verifyhost" option.

If I get it right, the documentation says that if we have a haproxy config that
- Has "verify required"
- Does not use SNI
- Has no "verifyhost"
Then HAProxy will simply ignore whatever hostname the server sends back in its 
certificate and the handshake will be OK.

If the "verifyhost" option is set and it does match the pattern, SSL handshake 
will also be OK.
If the "verifyhost" option is set and it does not match the pattern, SSL 
handshake will fail.


We tested this with two different HAProxy configs now and I can confirm that 
it's exactly like that.
(Server is always presenting the same certificate with "*.foo.bar" in it's 
common name / subject)

TESTBACKEND1 config (WORKING) looks like this:
# --- TESTBACKEND1
backend TESTBACKEND1
option  forwardfor except 127.0.0.0/8
server TESTBACKEND1-server 10.1.1.1:443 check inter 30s  verify required 
ssl verifyhost www.foo.bar ca-file 
/etc/haproxy/certs/backend-ca-certificates.crt crt 
/etc/haproxy/certs/frontend-server-certificate.pem


TESTBACKEND2 config (NOT WORKING) looks like this:
# --- TESTBACKEND2
backend TESTBACKEND2
option  forwardfor except 127.0.0.0/8
server TESTBACKEND2-server 10.1.1.1:443 check inter 30s  verify required 
ssl verifyhost www.ham.eggs ca-file 
/etc/haproxy/certs/backend-ca-certificates.crt crt 
/etc/haproxy/certs/frontend-server-certificate.pem


Please can you confirm that our understanding of HAProxy documentation is 
correct?
If so, then we could mark this topic as "solved" :-)


BR
Martin


-Original Message-
From: lu...@ltri.eu [mailto:lu...@ltri.eu]
Sent: Samstag, 14. Juli 2018 11:35
To: Martin RADEL 
Cc: haproxy@formilux.org
Subject: Re: TLS handshake works with certificate name mismatch using "verify 
required" and "verifyhost"

Hello Martin,


> we have a strange situation with our HAProxy, running on Version 1.8.8 with 
> OpenSSL.

Please share the output of haproxy -vv. Did you build openssl yourself or is 
this a distribution provided openssl lib? I am asking because build issues can 
lead to very strange behavior.



> server BACKEND1-server 10.1.1.1:443 check inter 30s  verify required
> ssl verifyhost *.foo.bar

*.foo.bar is not a valid hostname. It is a valid wildcard representation in a 
cert's SAN, yes, but not a hostname. Use real hostname for verifyhost instead, 
like www.foo.bar

Also, lets confirm the backend is really configured as per expectations, by 
running requests via curl from the haproxy box:

This should work:
curl -v --cacert /etc/haproxy/certs/backend-ca-certificates.crt
--resolve www.foo.bar:443:10.1.1.1 https://www.foo.bar/

This should fail:
curl -v --cacert /etc/haproxy/certs/backend-ca-certificates.crt
--resolve www.foo.fail:443:10.1.1.1 https://www.foo.bar/



cheers,
lukas
This message and any attachment ("the Message") are confidential. If you have 
received the Message in error, please notify the sender immediately and delete 
the Message from your system, any use of the Message is forbidden. 
Correspondence via e-mail is primarily for information purposes. RBI neither 
makes nor accepts legally binding statements via e-mail unless explicitly 
agreed otherwise. Information pursuant to § 14 Austrian Companies Code: 
Raiffeisen Bank International AG; Registered Office: Am Stadtpark 9, 1030 
Vienna,Austria; Company Register Number: FN 122119m at the Commercial Court of 
Vienna (Handelsgericht Wien).

RE: TLS handshake works with certificate name mismatch using "verify required" and "verifyhost"

[Martin RADEL]

Hi,

The certificate subject and subject alternate name are set to “*.foo.bar” (I’m 
replacing real DNS name here with foo.bar here because of security reasons).
There is no IP address included in the server’s certificate.

We are not using SNI on our clients.

BR
Martin


From: ig...@encompasscorporation.com 
[mailto:ig...@encompasscorporation.com]
Sent: Freitag, 13. Juli 2018 03:27
To: Martin RADEL 
mailto:martin.ra...@rbinternational.com>>
Cc: haproxy@formilux.org
Subject: Re: TLS handshake works with certificate name mismatch using "verify 
required" and "verifyhost"

On Fri, Jul 13, 2018 at 11:08 AM, Igor Cicimov 
mailto:ig...@encompasscorporation.com>> wrote:
Hi Martin,

On Thu, Jul 12, 2018 at 6:55 PM, Martin RADEL 
mailto:martin.ra...@rbinternational.com>> 
wrote:
Hi all,

we have a strange situation with our HAProxy, running on Version 1.8.8 with 
OpenSSL.
(See the details in the setup listed below - some lines are missing by 
intention. It’s a config snippet with just the interesting parts mentioned)

Initial situation:
We run a HAProxy instance which enforces mutual TLS on the frontend, allowing 
only clients to connect to it when they will present a specific certificate.
The HAPRoxy also does mutual TLS to the backend, presenting its frontend server 
certificate to the backend as a client certificate.
The backend only allows connections when the HAProxy’s certificate is presented 
to it.
To have a proper TLS handshake to the backend, and to be able to identify a 
man-in-the-middle scenario, we use the “verify required” directive together 
with the “verifyhost” directive.

The HAProxy is not able to resolve the backend’s real DNS-hostname, so it’s 
using the IP of the server instead (10.1.1.1)
The backend is presenting a wildcard server certificate with a DNS-hostname 
looking like “*.foo.bar”


In this configuration, one could assume that there is always a certificate name 
mismatch with the TLS handshake:
Backend server will present its server certificate with a proper DNS hostname 
in it, and the HAProxy will find out that it doesn’t match the initially used 
connection name “10.1.1.1”.


​Just checking if the IP hasn't been by any chance included in the certificate 
subjectAlternateNames ?
​

Issue:
In fact the connection to the backend works all the time, even when there is a 
name mismatch and even if we use the “verify required” option together with 
“verifyhost”.
Seems as if HAProxy completely ignores the mismatch, as if we would use the 
option “verify none”.


According to HAProxy documentation, this is clearly a not-expected behavior:
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-verify


Can somebody please share some knowledge why this is working, or can confirm 
that this is a bug?


#-
# Global settings
#-
global
log /dev/log local2
pidfile /run/haproxy/haproxy.pid
maxconn 2
ssl-default-bind-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:!RC4
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
stats socket /var/lib/haproxy/stats

#-
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#-
defaults
modehttp
log global
option  http-server-close
option  redispatch
retries 3
maxconn 2
errorfile 503   /etc/haproxy/errorpage.html
default-server  init-addr last,libc,none

# 
#  HAPROXY CONFIG WITH WILDCARD CERTIFICATE ON BACKEND
# 
# --- FRONTEND1 (TLS with mutual authentication) ---
frontend FRONTEND1
option  forwardfor except 
127.0.0.0/8
acl authorizedClient ssl_c_s_dn(cn) -m str -f 
/etc/haproxy/authorized_clients.cfg
bind *:443 ssl crt /etc/haproxy/certs/frontend-server-certificate.pem 

Re: SSL: double free on reload

[Willy Tarreau]

On Mon, Jul 16, 2018 at 08:32:31AM +0200, Janusz Dziemidowicz wrote:
> pon., 16 lip 2018 o 08:02 Willy Tarreau  napisal(a):
> > This one looks a bit strange. I looked at it a little bit and it corresponds
> > to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is 
> > no
> > other line in the code appearing to perfom a free on this element, and when
> > passing through this code the key_ref is destroyed and properly nulled. I
> > checked if it was possible for this element not to be allocated and I don't
> > see how that could happen either. Thus I'm seeing only three possibilities :
> >
> >   - this element was duplicated and appears at multiple places (multiple 
> > list
> > elements) leading to a real double free
> >
> >   - there is a memory corruption somewhere possibly resulting in this 
> > element
> > being corrupted and not in fact victim of a double free
> >
> >   - I can't read code and there is another free that I failed to detect.
> >
> > Are you able to trigger this on a trivial config ? Maybe it only happens
> > when certain features you have in your config are enabled ?
> 
> I've reported this some time ago :)
> https://www.mail-archive.com/haproxy@formilux.org/msg30093.html

Ah thank you Janusz, and I notice that your report matches Thierry's second
e-mail very closely.

I'm CCing Nenad who added the tls-ticket-keys in case he has any idea
on the subject, based on how the bind line is initialized maybe.

thanks,
Willy

Re: SSL: double free on reload

[Nenad Merdanovic]

Hello,

On 7/16/2018 10:46 AM, Willy Tarreau wrote:

On Mon, Jul 16, 2018 at 08:32:31AM +0200, Janusz Dziemidowicz wrote:

pon., 16 lip 2018 o 08:02 Willy Tarreau  napisal(a):

This one looks a bit strange. I looked at it a little bit and it corresponds
to the line "free(bind_conf->keys_ref->tlskeys);". Unfortunately, there is no
other line in the code appearing to perfom a free on this element, and when
passing through this code the key_ref is destroyed and properly nulled. I
checked if it was possible for this element not to be allocated and I don't
see how that could happen either. Thus I'm seeing only three possibilities :

  - this element was duplicated and appears at multiple places (multiple list
elements) leading to a real double free

  - there is a memory corruption somewhere possibly resulting in this element
being corrupted and not in fact victim of a double free

  - I can't read code and there is another free that I failed to detect.

Are you able to trigger this on a trivial config ? Maybe it only happens
when certain features you have in your config are enabled ?


I've reported this some time ago :)
https://www.mail-archive.com/haproxy@formilux.org/msg30093.html


Ah thank you Janusz, and I notice that your report matches Thierry's second
e-mail very closely.

I'm CCing Nenad who added the tls-ticket-keys in case he has any idea
on the subject, based on how the bind line is initialized maybe.


Ugh, this was a long time ago. [FROM MEMORY] The element should not be 
duplicated as far as I can remember. The references are stored in an 
ebtree in order to prevent duplication and to provide consistent view 
when updated dynamically.


I just pulled HEAD and cannot reproduce this with either of these 
configs. The "good" thing is that I get a crash every time I reload, 
with different stack traces for each config.


One of them starts like:
#5  0x7f271cce0847 in _int_free (av=0x7f271d015c40 , 
p=0x562e01935460, have_lock=) at malloc.c:4362

size = 195488
fb = 
nextchunk = 0x562e01944f30
nextsize = 131280
nextinuse = 
prevsize = 
bck = 
fwd = 
__PRETTY_FUNCTION__ = "_int_free"
#6  0x562e0011a7bb in deinit_pollers () at src/fd.c:554
bp = 
p = 
#7  0x562e00024c77 in main (argc=, 
argv=0x7fffc630ec38) at src/haproxy.c:3095

err = 
retry = 
limit = {rlim_cur = 4012, rlim_max = 4012}
errmsg = 
"\000\000\000\000\000\000\000\000\000\377\330q\356\336\342\345\370\351\060\306\377\177\000\000\000\t\216\001.V\000\000\230\351\060\306\377\177\000\000\261\000\000\000\000\000\000\000\262\000\000\000\000\000\000\000\330\352\060\306\377\177\000\000\370\351\060\306\377\177\000\000\346z\r\000.V\000\000y+\025\000.V\000\000\330\352\060\306\377\177\000\000\000\000\000"

pidfd = 

I'll poke around it more tomorrow as it's quite late here.

Regards,
Nenad



thanks,
Willy

Re: SSL: double free on reload

[Willy Tarreau]

Hi Nenad,

On Tue, Jul 17, 2018 at 03:37:37AM +0200, Nenad Merdanovic wrote:
> Ugh, this was a long time ago. [FROM MEMORY] The element should not be
> duplicated as far as I can remember. The references are stored in an ebtree
> in order to prevent duplication and to provide consistent view when updated
> dynamically.

OK. Then maybe the elements are freed after scanning the ebtree as well,
and we're meeting the node again after it was freed. I'll run a test
with the memory debugging options to see if it crashes on first dereference.

> I just pulled HEAD and cannot reproduce this with either of these configs.
> The "good" thing is that I get a crash every time I reload, with different
> stack traces for each config.

This could then indeed indicate a corrupted memory somewhere, and maybe
the tls-ticket part is just the first victim.

> I'll poke around it more tomorrow as it's quite late here.

Thank you Nenad. I'll try as well on my side.

Cheers,
Willy

Re: Building HAProxy 1.8 fails on Solaris

[Lukas Tribus]

Hello,


On Mon, 16 Jul 2018 at 03:12, Thrawn  wrote:
>
> Update: If I disable threading with
>
> USE_THREAD=
>
> then the build gets much further, but still fails eventually with:
>
> gcc  -g -o haproxy src/ev_poll.o ebtree/ebtree.o ebtree/eb32sctree.o 
> ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o 
> ebtree/ebimtree.o ebtree/ebistree.o src/proto_http.o src/cfgparse.o 
> src/server.o src/stream.o src/flt_spoe.o src/stick_table.o src/stats.o 
> src/mux_h2.o src/checks.o src/haproxy.o src/log.o src/dns.o src/peers.o 
> src/standard.o src/sample.o src/cli.o src/stream_interface.o src/proto_tcp.o 
> src/backend.o src/proxy.o src/tcp_rules.o src/listener.o src/flt_http_comp.o 
> src/pattern.o src/cache.o src/filters.o src/vars.o src/acl.o src/payload.o 
> src/connection.o src/raw_sock.o src/proto_uxst.o src/flt_trace.o 
> src/session.o src/ev_select.o src/channel.o src/task.o src/queue.o 
> src/applet.o src/map.o src/frontend.o src/freq_ctr.o src/lb_fwlc.o 
> src/mux_pt.o src/auth.o src/fd.o src/hpack-dec.o src/memory.o src/lb_fwrr.o 
> src/lb_chash.o src/lb_fas.o src/hathreads.o src/chunk.o src/lb_map.o 
> src/xxhash.o src/regex.o src/shctx.o src/buffer.o src/action.o src/h1.o 
> src/compression.o src/pipe.o src/namespace.o src/sha1.o src/hpack-tbl.o 
> src/hpack-enc.o src/uri_auth.o src/time.o src/proto_udp.o src/arg.o 
> src/signal.o src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o 
> src/mailers.o src/h2.o src/base64.o src/hash.o -lnsl -lsocket  -lcrypt 
> -L../pcre/lib -Wl,-Bstatic -lpcreposix -lpcre -Wl,-Bdynamic
> Undefined   first referenced
>  symbol in file
> __sync_sub_and_fetchsrc/cache.o
> __sync_val_compare_and_swap src/cache.o
> __sync_lock_test_and_setsrc/cache.o
> ld: fatal: symbol referencing errors. No output written to haproxy
> collect2: ld returned 1 exit status
> gmake: *** [haproxy] Error 1

Are you sure you did a "make clean" before building with USE_THREAD= ?


Also please provide the output of "gcc --version".


lukas

Re: Bug when passing variable to mapping function

[Lukas Tribus]

Hello,



On Fri, 29 Jun 2018 at 07:15, Jarno Huuskonen  wrote:
>
> Hi,
>
> On Thu, Jun 28, Jarno Huuskonen wrote:
> > I think this is the commit that breaks map_regm in this case:
> > b5997f740b21ebb197e10a0f2fe9dc13163e1772 (MAJOR: threads/map: Make
> > acls/maps thread safe).
> >
> > If I revert this commit from pattern.c:pattern_exec_match
> > then the map_regm \1 backref seems to work.
>
> I think I found what's replacing the \000 as first char:
> in (map.c) sample_conv_map:
> /* In the regm case, merge the sample with the input. */
> if ((long)private == PAT_MATCH_REGM) {
> str = get_trash_chunk();
> str->len = exp_replace(str->str, str->size, 
> smp->data.u.str.str,
>pat->data->u.str.str,
>(regmatch_t *)smp->ctx.a[0]);
>
> Before call to get_trash_chunk() smp->data.u.str.str is for example
> 'distri.com' and after get_trash_chunk() smp->data.u.str.str
> is '\000istri.com'.
>
> At the moment I don't have time to dig deeper, but hopefully this
> helps a little bit.

Thanks for the detailed analysis, relaying to Emeric ...



Lukas

Re: TLS handshake works with certificate name mismatch using "verify required" and "verifyhost"

[Lukas Tribus]

On Mon, 16 Jul 2018 at 11:57, Martin RADEL
 wrote:
>
> Hi,
>
> I think we found the issue:
> Seems that there was a misunderstanding from us regarding the haproxy 
> documentation with the "verifyhost" option.
>
> If I get it right, the documentation says that if we have a haproxy config 
> that
> - Has "verify required"
> - Does not use SNI
> - Has no "verifyhost"
> Then HAProxy will simply ignore whatever hostname the server sends back in 
> its certificate and the handshake will be OK.

Yes, that is correct, also see the verify docs:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-verify

Not sure how we ended up in this situation though. I remember there
was a vivid discussion about whether "verify" should default to none
or required. We opted for "required", to be "secure by default", but
this is totally useless given that it requires verifyhost or sni, and
will silently disable cert verification when those option are not
given. That's probably the worst thing we can do in this case; this
configuration should be rejected, imho. People that don't care about
cert verification should simply set "verify none". But here we are
now, and this is documented behavior :(

I think this was introduced in 2ab88675, maybe we can change this in 1.9.



> Please can you confirm that our understanding of HAProxy documentation is 
> correct?
> If so, then we could mark this topic as "solved" :-)

Yes, but I don't understand, you reported that verification is not
happening *with* verifyhost:

> the connection to the backend works all the time, even when there is a name 
> mismatch and even if we use the “verify required” option together with 
> “verifyhost”.


"verify required ssl verifyhost www.ham.eggs" fails as expected for
you now, correct?



Thanks,
Lukas

Re: Building HAProxy 1.8 fails on Solaris

[Olivier Houchard]

Hi,

On Mon, Jul 16, 2018 at 01:12:18AM +, Thrawn wrote:
>  Update: If I disable threading with
> USE_THREAD=
> then the build gets much further, but still fails eventually with:
> gcc  -g -o haproxy src/ev_poll.o ebtree/ebtree.o ebtree/eb32sctree.o 
> ebtree/eb32tree.o ebtree/eb64tree.o ebtree/ebmbtree.o ebtree/ebsttree.o 
> ebtree/ebimtree.o ebtree/ebistree.o src/proto_http.o src/cfgparse.o 
> src/server.o src/stream.o src/flt_spoe.o src/stick_table.o src/stats.o 
> src/mux_h2.o src/checks.o src/haproxy.o src/log.o src/dns.o src/peers.o 
> src/standard.o src/sample.o src/cli.o src/stream_interface.o src/proto_tcp.o 
> src/backend.o src/proxy.o src/tcp_rules.o src/listener.o src/flt_http_comp.o 
> src/pattern.o src/cache.o src/filters.o src/vars.o src/acl.o src/payload.o 
> src/connection.o src/raw_sock.o src/proto_uxst.o src/flt_trace.o 
> src/session.o src/ev_select.o src/channel.o src/task.o src/queue.o 
> src/applet.o src/map.o src/frontend.o src/freq_ctr.o src/lb_fwlc.o 
> src/mux_pt.o src/auth.o src/fd.o src/hpack-dec.o src/memory.o src/lb_fwrr.o 
> src/lb_chash.o src/lb_fas.o src/hathreads.o src/chunk.o src/lb_map.o 
> src/xxhash.o src/regex.o src/shctx.o src/buffer.o src/action.o src/h1.o 
> src/compression.o src/pipe.o src/namespace.o src/sha1.o src/hpack-tbl.o 
> src/hpack-enc.o src/uri_auth.o src/time.o src/proto_udp.o src/arg.o 
> src/signal.o src/protocol.o src/lru.o src/hdr_idx.o src/hpack-huff.o 
> src/mailers.o src/h2.o src/base64.o src/hash.o -lnsl -lsocket  -lcrypt 
> -L../pcre/lib -Wl,-Bstatic -lpcreposix -lpcre -Wl,-Bdynamic Undefined         
>               first referenced symbol                             in 
> file__sync_sub_and_fetch                
> src/cache.o__sync_val_compare_and_swap         
> src/cache.o__sync_lock_test_and_set            src/cache.old: fatal: symbol 
> referencing errors. No output written to haproxycollect2: ld returned 1 exit 
> statusgmake: *** [haproxy] Error 1

I think your gcc is just too old. Those appeared around gcc 4.1 or so.

Regards,

Olivier

What to look out for when going from 1.6 to 1.8?

[Tim Verhoeven]

Hello all,

We have been running the 1.6 branch of HAProxy, without any issues, for a
while now. And reading the updates around 1.8 here in the mailing list it
looks like its time to upgrade to this branch.

So I was wondering if there are any things I need to look of for when doing
this upgrade? We are not doing anything special with HAProxy (I think). We
run it as a single process, we use SSL/TLS termination, some ACL's and a
bunch of backends. We only use HTTP 1.1 and TCP connections.

>From what I've been able to gather my current config will works just as
good with 1.8. But some extra input from all the experts here is always
appreciated.

Thanks,
Tim

Re: What to look out for when going from 1.6 to 1.8?

[Alex Evonosky]

Tim-

I can speak from a production point of view that we had HAproxy on the 1.6
branch inside docker containers for mesos load balancing with pretty much
the same requirements as you spoke of.  After compiling Haproxy to the 1.8x
branch the same config worked without issues.

-Alex


On Mon, Jul 16, 2018 at 9:39 AM, Tim Verhoeven 
wrote:

> Hello all,
>
> We have been running the 1.6 branch of HAProxy, without any issues, for a
> while now. And reading the updates around 1.8 here in the mailing list it
> looks like its time to upgrade to this branch.
>
> So I was wondering if there are any things I need to look of for when
> doing this upgrade? We are not doing anything special with HAProxy (I
> think). We run it as a single process, we use SSL/TLS termination, some
> ACL's and a bunch of backends. We only use HTTP 1.1 and TCP connections.
>
> From what I've been able to gather my current config will works just as
> good with 1.8. But some extra input from all the experts here is always
> appreciated.
>
> Thanks,
> Tim
>

RE: TLS handshake works with certificate name mismatch using "verify required" and "verifyhost"

[Martin RADEL]

Hi Lukas,

Right, "verify required ssl verifyhost www.ham.eggs" fails now as expected.

My initial report that it doesn't work with "verifyhost" option was not 
completely right,
because in fact we never tried what would happen if we set a non-matching 
pattern in the "verifyhost" directive.

We tested without the verifyhost because of lack of knowledge that this would 
be mandatory. Then it always worked (even with the name mismatch),
And we tested with verifyhost directive, but only with a matching pattern, also 
always worked.

Now with the new information, it's clear why this always worked and what we 
have to do to achieve a correct haproxy config.
-> always use "verify required" *together* with "verifyhost"

I would also vote to change the HAProxy default behavior to more 
security-oriented when there are some directives not passed.
This would on the one hand generate more questions "why is it not working?", 
but on the other hand would have a stronger security out of the box.

Thanks for your help!

BR
Martin

-Original Message-
From: lu...@ltri.eu [mailto:lu...@ltri.eu]
Sent: Montag, 16. Juli 2018 14:11
To: Martin RADEL 
Cc: haproxy@formilux.org; w...@1wt.eu; m...@gandi.net
Subject: Re: TLS handshake works with certificate name mismatch using "verify 
required" and "verifyhost"

On Mon, 16 Jul 2018 at 11:57, Martin RADEL 
 wrote:
>
> Hi,
>
> I think we found the issue:
> Seems that there was a misunderstanding from us regarding the haproxy 
> documentation with the "verifyhost" option.
>
> If I get it right, the documentation says that if we have a haproxy
> config that
> - Has "verify required"
> - Does not use SNI
> - Has no "verifyhost"
> Then HAProxy will simply ignore whatever hostname the server sends back in 
> its certificate and the handshake will be OK.

Yes, that is correct, also see the verify docs:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-verify

Not sure how we ended up in this situation though. I remember there was a vivid 
discussion about whether "verify" should default to none or required. We opted 
for "required", to be "secure by default", but this is totally useless given 
that it requires verifyhost or sni, and will silently disable cert verification 
when those option are not given. That's probably the worst thing we can do in 
this case; this configuration should be rejected, imho. People that don't care 
about cert verification should simply set "verify none". But here we are now, 
and this is documented behavior :(

I think this was introduced in 2ab88675, maybe we can change this in 1.9.



> Please can you confirm that our understanding of HAProxy documentation is 
> correct?
> If so, then we could mark this topic as "solved" :-)

Yes, but I don't understand, you reported that verification is not happening 
*with* verifyhost:

> the connection to the backend works all the time, even when there is a name 
> mismatch and even if we use the “verify required” option together with 
> “verifyhost”.


"verify required ssl verifyhost www.ham.eggs" fails as expected for you now, 
correct?



Thanks,
Lukas
This message and any attachment ("the Message") are confidential. If you have 
received the Message in error, please notify the sender immediately and delete 
the Message from your system, any use of the Message is forbidden. 
Correspondence via e-mail is primarily for information purposes. RBI neither 
makes nor accepts legally binding statements via e-mail unless explicitly 
agreed otherwise. Information pursuant to § 14 Austrian Companies Code: 
Raiffeisen Bank International AG; Registered Office: Am Stadtpark 9, 1030 
Vienna,Austria; Company Register Number: FN 122119m at the Commercial Court of 
Vienna (Handelsgericht Wien).

Re: What to look out for when going from 1.6 to 1.8?

[Dave Chiluk]

We have the same use case as Alex *(mesos load balancing), and also confirm
that our config worked without change 1.6->1.8.

Given our testing you should consider the seamless reload -x option, and
the dynamic server configuration apis.  Both have greatly alleviated issues
we've faced in our microservices-based cloud.

Dave.

On Mon, Jul 16, 2018 at 8:47 AM Alex Evonosky 
wrote:

> Tim-
>
> I can speak from a production point of view that we had HAproxy on the 1.6
> branch inside docker containers for mesos load balancing with pretty much
> the same requirements as you spoke of.  After compiling Haproxy to the 1.8x
> branch the same config worked without issues.
>
> -Alex
>
>
> On Mon, Jul 16, 2018 at 9:39 AM, Tim Verhoeven  > wrote:
>
>> Hello all,
>>
>> We have been running the 1.6 branch of HAProxy, without any issues, for a
>> while now. And reading the updates around 1.8 here in the mailing list it
>> looks like its time to upgrade to this branch.
>>
>> So I was wondering if there are any things I need to look of for when
>> doing this upgrade? We are not doing anything special with HAProxy (I
>> think). We run it as a single process, we use SSL/TLS termination, some
>> ACL's and a bunch of backends. We only use HTTP 1.1 and TCP connections.
>>
>> From what I've been able to gather my current config will works just as
>> good with 1.8. But some extra input from all the experts here is always
>> appreciated.
>>
>> Thanks,
>> Tim
>>
>
>