Re: Debugging ssl handshake failures
Corrected build instructions attached. openssl-2.2.2.2 should be haproxy-2.2.3. Regards, Bruno apt-debuild Description: Binary data
Re: Debugging ssl handshake failures
Hi, > I take it that means theres no internal debug logging for the tls errors that > we can just expose via logfile? Proof of concept patches are attached with build instructions. You may wish to edit the haproxy-2.2.3/rules/debian folder to increase the -j setting to your current number of cores. The "disambiguate-ssl-handshake-errors-1.patch" only adds additional error messages for the initial ClientHello processing - realistically, it's only useful to see if there is no SNI being sent (bots, healthchecks are the usual offenders). The "disambiguate-ssl-handshake-errors-2.patch" implements everything the first patch implements, adds a trash chunk for logging additional error data to the conn structure, and reuses the SSL error logging logic from ssl_sock_dump_errors Practically, this means that memory usage is higher - if I recall correctly (and it's way to late/early at this point) it's a 16KB overhead per connection (echo "show pools" | socat stdio /var/run/haproxy.sock will have a more detailed breakdown). Watching the output of show pools is recommended - while I haven't noticed a memory leak, keeping an eye on the trash pool is a good idea. The fcgi protocol is also affected by the addition of the extra_err_data. I have to do a smoke test if "proto fcgi" behaves as expected, or if there's a potential segfault. The patch works, but it requires more extensive testing. Sharing it as-is since I might not be able to pursue this further in a significant way for some time. The mapping between the error messages and the potential causes can be a bit obscure, but it's still useful. E.g. an invalid SNI when using strict-sni maps to: tls_post_process_client_hello: no shared cipher If there's a cipher mismatch, this also maps to the above error message. A protocol version mismatch (e.g. trying TLS1 when only TLS1.2 is supported) results in: tls_early_post_process_client_hello: unsupported protocol. The list of error codes is available upstream at https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/err/openssl.txt#L2774 . Regarding the packet capture question - exporting libpcap data via SPOE might be possible. It's an ongoing topic. Regards, Brunodiff -ur a/include/haproxy/connection.h b/include/haproxy/connection.h --- a/include/haproxy/connection.h 2020-09-10 05:49:02.705917730 +0200 +++ b/include/haproxy/connection.h 2020-09-10 05:12:30.287797636 +0200 @@ -382,8 +382,10 @@ struct connection *conn; conn = pool_alloc(pool_head_connection); + conn->extra_err_data = alloc_trash_chunk(); if (likely(conn != NULL)) - conn_init(conn); + if(likely(conn->extra_err_data != NULL)) + conn_init(conn); return conn; } @@ -458,7 +460,7 @@ sess->idle_conns--; session_unown_conn(sess, conn); } - + free_trash_chunk(conn->extra_err_data); sockaddr_free(>src); sockaddr_free(>dst); @@ -697,8 +699,12 @@ case CO_ER_SSL_CRT_FAIL: return "SSL client certificate not trusted"; case CO_ER_SSL_MISMATCH: return "Server presented an SSL certificate different from the configured one"; case CO_ER_SSL_MISMATCH_SNI: return "Server presented an SSL certificate different from the expected one"; - case CO_ER_SSL_HANDSHAKE: return "SSL handshake failure"; + case CO_ER_SSL_HANDSHAKE:return "SSL handshake failure"; case CO_ER_SSL_HANDSHAKE_HB: return "SSL handshake failure after heartbeat"; + case CO_ER_SSL_HSHK_CL_SNI_GBRSH: return "SSL handshake failure: ClientHello server name (SNI) null, too long, or invalid"; + case CO_ER_SSL_HSHK_CL_S_SNI: return "SSL handshake failure: ClientHello server name (SNI) missing. SNI is required when strict-sni is used"; + case CO_ER_SSL_HSHK_CL_CIPHERS: return "SSL handshake failure: ClientHello ciphers are invalid"; + case CO_ER_SSL_HSHK_CL_SIGALGS: return "SSL handshake failure: ClientHello signature algorithms are invalid"; case CO_ER_SSL_KILLED_HB: return "Stopped a TLSv1 heartbeat attack (CVE-2014-0160)"; case CO_ER_SSL_NO_TARGET: return "Attempt to use SSL on an unknown target (internal error)"; diff -ur a/include/haproxy/connection-t.h b/include/haproxy/connection-t.h --- a/include/haproxy/connection-t.h2020-09-10 05:49:02.705917730 +0200 +++ b/include/haproxy/connection-t.h2020-09-10 05:13:00.007799264 +0200 @@ -233,6 +233,10 @@ CO_ER_SSL_MISMATCH_SNI, /* Server presented an SSL certificate different from the expected one */ CO_ER_SSL_HANDSHAKE,/* SSL error during handshake */ CO_ER_SSL_HANDSHAKE_HB, /* SSL error during handshake with heartbeat present */ + CO_ER_SSL_HSHK_CL_SNI_GBRSH, /* SSL handshake failure: ClientHello server name (SNI) null, too long, or invalid */ + CO_ER_SSL_HSHK_CL_S_SNI, /* SSL handshake failure:
Re: Haproxy 2.2.3 source
On Wed, Sep 09, 2020 at 10:03:29PM +0200, Vincent Bernat wrote: > ? 9 septembre 2020 19:31 +02, Willy Tarreau: > > >> Feel free to pick this patch if that helps for your builds, I'm going > >> to backport it to 2.2 once all platforms are happy. > > > > All builds are OK now, the commit was backported to 2.2 and the patch > > can be retrieved here: > > > > http://git.haproxy.org/?p=haproxy-2.2.git;a=commitdiff_plain;h=10c627ab > > All good on Debian: > https://buildd.debian.org/status/package.php?p=haproxy=experimental Great, thank you Vincent for the quick update! Willy
Re: Haproxy 2.2.3 source
Thank you Willy! A On Wed, Sep 9, 2020 at 1:31 PM Willy Tarreau wrote: > On Wed, Sep 09, 2020 at 07:20:17PM +0200, Willy Tarreau wrote: > > Feel free to pick this patch if that helps for your builds, I'm going > > to backport it to 2.2 once all platforms are happy. > > All builds are OK now, the commit was backported to 2.2 and the patch > can be retrieved here: > > http://git.haproxy.org/?p=haproxy-2.2.git;a=commitdiff_plain;h=10c627ab > > Sorry for the mess :-/ > > Willy >
Re: Haproxy 2.2.3 source
❦ 9 septembre 2020 19:31 +02, Willy Tarreau: >> Feel free to pick this patch if that helps for your builds, I'm going >> to backport it to 2.2 once all platforms are happy. > > All builds are OK now, the commit was backported to 2.2 and the patch > can be retrieved here: > > http://git.haproxy.org/?p=haproxy-2.2.git;a=commitdiff_plain;h=10c627ab All good on Debian: https://buildd.debian.org/status/package.php?p=haproxy=experimental -- Small things make base men proud. -- William Shakespeare, "Henry VI"
Re: `ssl_fc_has_early` fetcher and 0rtt
> > That seems strange indeed but looking at the code that's what I'm > > seeing. Was your access to ssl_fc_has_early placed before or after the > > rule above ? If it's after it must indeed report false. fetcher is placed before the rule > > I seem to remember there was one but can't find it, so I may have been > > confused. With this said, it doesn't provide a big information since > > once the handshake is completed, it's exactly identical to a regular > > one. But it can be nice for statistics at least. > > > > Pretty sure the original version always returned 1 if there were early > data, but we changed it so that it would return 0 once the handshake was > done, as we thought it was more useful, that may be what you're thinking > about. Yes I indeed found the commit which changed the behaviour. In fact I probably mixed the true meaning of the definition in the doc and the old blog post about it which is vague. Now everything is clear. >From our point of view, it's interesting to check some behavioural changes we can make on the L4 layer (e.g. hash including source port or not). > That indeed tells me something. However I checked if CO_FL_EARLY_DATA > persists, and apparently we kill it very early once we have everything > so it's not as if we could trivially add a new sample fetch function to > report its past status. > > Also I seem to remember that we concluded that depending on the timing > and how data were aggregated and processed in the SSL lib, we couldn't > reliably report the use of early data if those were converted very early. At some point I was asking myself whether this could be done but simply to give a signal without any guarantee, but you seemed to conclude that it's not reliable enough to become a new fetcher. -- William
Re: Haproxy 2.2.3 source
On Wed, Sep 09, 2020 at 07:20:17PM +0200, Willy Tarreau wrote: > Feel free to pick this patch if that helps for your builds, I'm going > to backport it to 2.2 once all platforms are happy. All builds are OK now, the commit was backported to 2.2 and the patch can be retrieved here: http://git.haproxy.org/?p=haproxy-2.2.git;a=commitdiff_plain;h=10c627ab Sorry for the mess :-/ Willy
Re: Haproxy 2.2.3 source
On Wed, Sep 09, 2020 at 05:49:50PM +0200, Willy Tarreau wrote:
> On Wed, Sep 09, 2020 at 05:40:05PM +0200, Vincent Bernat wrote:
> > ? 9 septembre 2020 16:58 +02, Willy Tarreau:
> >
> > > Ah I'm really angry because I tested on many platforms, *including* armhf,
> > > but now I'm not seeing it, so either I failed on one test or it depends
> > > on the compiler combination :-(
> >
> > I am getting it on Debian Unstable (gcc 10.2.0, glibc 2.31), Ubuntu
> > Focal (gcc 9.3.0, glibc 2.31) and Ubuntu Bionic (gcc 7.5.0, glibc 2.27).
>
> Thanks. Now I'm trying to work on it but can't reproduce the loading
> problem anymore... That's a heisenbug... The trick I'm trying right now
> is to really create a thread that exits on startup. This way we'll be
> certain pthread_exit() was already called once. It's much uglier than
> the first work around but should be more portable and more reliable.
OK, I've pushed this one into -dev:
f734ebfac ("BUILD: threads: better workaround for late loading of libgcc_s")
I found a more reliable and safer way to address this, exactly the same
as the one we did for backtrace(), which consists in calling the offending
function once at boot. So what we do now very early during initialization
is that we create a new thread that exits using pthread_exit(). This way
we're certain the function is usable before going further, and we don't
care if libgcc_s or whatever else provides the functionality.
I tested it on armhf, aarch64, x86_64, mipsel, ppc64le, and sparc64 with
success. It was also tested on musl which uses a different threading
library and met not issue. On Travis it's OK on all platforms. On Cirrus
and Cygwin it's still in progrss but Centos is already OK so I'm quite
confident.
Feel free to pick this patch if that helps for your builds, I'm going
to backport it to 2.2 once all platforms are happy.
Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 05:43:08PM +0200, Olivier Houchard wrote: > > I seem to remember there was one but can't find it, so I may have been > > confused. With this said, it doesn't provide a big information since > > once the handshake is completed, it's exactly identical to a regular > > one. But it can be nice for statistics at least. > > > > Pretty sure the original version always returned 1 if there were early > data, but we changed it so that it would return 0 once the handshake was > done, as we thought it was more useful, that may be what you're thinking > about. That indeed tells me something. However I checked if CO_FL_EARLY_DATA persists, and apparently we kill it very early once we have everything so it's not as if we could trivially add a new sample fetch function to report its past status. Also I seem to remember that we concluded that depending on the timing and how data were aggregated and processed in the SSL lib, we couldn't reliably report the use of early data if those were converted very early. Willy
Re: Haproxy 2.2.3 source
On Wed, Sep 09, 2020 at 05:40:05PM +0200, Vincent Bernat wrote: > ? 9 septembre 2020 16:58 +02, Willy Tarreau: > > > Ah I'm really angry because I tested on many platforms, *including* armhf, > > but now I'm not seeing it, so either I failed on one test or it depends > > on the compiler combination :-( > > I am getting it on Debian Unstable (gcc 10.2.0, glibc 2.31), Ubuntu > Focal (gcc 9.3.0, glibc 2.31) and Ubuntu Bionic (gcc 7.5.0, glibc 2.27). Thanks. Now I'm trying to work on it but can't reproduce the loading problem anymore... That's a heisenbug... The trick I'm trying right now is to really create a thread that exits on startup. This way we'll be certain pthread_exit() was already called once. It's much uglier than the first work around but should be more portable and more reliable. Willy
Re: Haproxy 2.2.3 source
It is not cross-built. Debian builds armhf from arm64 builders. It seems Ubuntu is also using arm64 hardware to build armhf. An alternative that could work is to use QEMU user emulation. You can directly use "qemu-debootstrap --arch=armhf" to get a working chroot. -- Format a program to help the reader understand it. - The Elements of Programming Style (Kernighan & Plauger) ――― Original Message ――― From: Илья Шипицин Sent: 9 septembre 2020 20:38 +05 Subject: Re: Haproxy 2.2.3 source To: Willy Tarreau Cc: Vincent Bernat; Alex Evonosky; haproxy@formilux.org > how do you build armh ? can you share details ? > if that's cross build, we can easily add to github actions, for example. > > unfortunately, it is hard to get armh native CI. > > ср, 9 сент. 2020 г. в 20:01, Willy Tarreau : > >> On Tue, Sep 08, 2020 at 11:47:25PM +0200, Vincent Bernat wrote: >> > ? 8 septembre 2020 16:13 -04, Alex Evonosky: >> > >> > > Just compiling 2.2.3 and getting this reference: >> > > >> > > >> > > /haproxy-2.2.3/src/thread.c:212: undefined reference to >> > > `_Unwind_Find_FDE' >> > >> > I am getting the same issue on armhf only. Other platforms don't get >> > this issue. On this platform, we only get: >> > >> > w DF *UND* GLIBC_2.4 __gnu_Unwind_Find_exidx >> > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException >> > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase >> > 00016e1c gDF .text 0022 GCC_4.3.0 _Unwind_Backtrace >> > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind >> > 00016dd4 gDF .text 0022 GCC_3.3 _Unwind_Resume_or_Rethrow >> > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase >> > 0001662c gDF .text 0036 GCC_3.5 _Unwind_VRS_Set >> > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume >> > 000169d8 gDF .text 02ba GCC_3.5 _Unwind_VRS_Pop >> > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart >> > 000165cc gDF .text 0002 GCC_3.5 _Unwind_Complete >> > 00017184 gDF .text 0012 GCC_3.0 >> _Unwind_GetLanguageSpecificData >> > 000165dc gDF .text 0036 GCC_3.5 _Unwind_VRS_Get >> > 000164f0 gDF .text 0004 GCC_3.3 _Unwind_GetCFA >> > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException >> > >> > So, older symbols are: >> > >> > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException >> > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase >> > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind >> > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase >> > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume >> > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart >> > 00017184 gDF .text 0012 GCC_3.0 >> _Unwind_GetLanguageSpecificData >> > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException >> > >> > Moreover, comment says _Unwind_Find_DFE doesn't take arguments, but the >> > signature I have in glibc is: >> > >> > fde * >> > _Unwind_Find_FDE (void *pc, struct dwarf_eh_bases *bases) >> >> Ah I'm really angry because I tested on many platforms, *including* armhf, >> but now I'm not seeing it, so either I failed on one test or it depends >> on the compiler combination :-( >> >> The principle was just to force to load libgcc_s that tends to cause >> aborts on reload for some users in chroots when threads exit, because >> pthread_exit() tends to be the first one to require libgcc_s and it's >> quite too late. >> >> In the mean time you can probably get away with this: >> >> diff --git a/src/thread.c b/src/thread.c >> index 5eb68e33a..167e26e9d 100644 >> --- a/src/thread.c >> +++ b/src/thread.c >> @@ -201,7 +201,7 @@ static void __thread_init(void) >> exit(1); >> } >> >> -#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && >> !defined(__clang__) >> +#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && >> !defined(__clang__) && !defined(__arm__) >> /* make sure libgcc_s is already loaded, because pthread_exit() may >> * may need it on exit after the chroot! _Unwind_Find_FDE() is >> defined >> * there since gcc 3.0, has no side effect, doesn't take any >> argument >> >> >> But I'd really like to find a *reliable* way to force libgcc_s to be loaded >> if available and required (i.e. not if statically built). I thought about >> causing a 64/64 bit divide that usually is done via divsi3 and friend but >> on x86_64 (where the problem was encountered) it will not do it. >> >> I'm thinking about something: maybe I can have a look around >> pthread_getspecific() and pthread_key_create(). I would be very surprised >> if they didn't rely on some compiler-specific help from libgcc_s. >> >> I'll keep testing and will get back to you guys. >> >> Willy >> >>
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 05:35:28PM +0200, Willy Tarreau wrote: > On Wed, Sep 09, 2020 at 04:57:58PM +0200, William Dauchy wrote: > > > I think it's not easy to reproduce these tests, you need a high enough > > > latency between haproxy and the client so that the handshake is not > > > already completed when you evaluate the rule, and of course you need > > > to make sure the client sends using early data. I don't remember how > > > Olivier used to run his tests but I remember that it was a bit tricky, > > > so it's very possible that you never fall into the situation where you > > > can see the unvalidated early data yet. > > > > It means my understanding of this fetcher was wrong indeed. > > For me the protection was here: > > http-request wait-for-handshake if ! METH_GET > > and the fetcher here to log whether it was a 0rtt request or not. > > In reality, it means all our requests have completed the handshake > > when the rule is evaluated (which is surprising looking at the number > > we have). > > That seems strange indeed but looking at the code that's what I'm > seeing. Was your access to ssl_fc_has_early placed before or after the > rule above ? If it's after it must indeed report false. > > > So maybe we can possibly work on an alternative fetcher to know > > whether this was a 0rtt request? Or is there another way? > > I seem to remember there was one but can't find it, so I may have been > confused. With this said, it doesn't provide a big information since > once the handshake is completed, it's exactly identical to a regular > one. But it can be nice for statistics at least. > Pretty sure the original version always returned 1 if there were early data, but we changed it so that it would return 0 once the handshake was done, as we thought it was more useful, that may be what you're thinking about. Olivier
Re: Haproxy 2.2.3 source
❦ 9 septembre 2020 16:58 +02, Willy Tarreau: > Ah I'm really angry because I tested on many platforms, *including* armhf, > but now I'm not seeing it, so either I failed on one test or it depends > on the compiler combination :-( I am getting it on Debian Unstable (gcc 10.2.0, glibc 2.31), Ubuntu Focal (gcc 9.3.0, glibc 2.31) and Ubuntu Bionic (gcc 7.5.0, glibc 2.27). -- Sometimes I wonder if I'm in my right mind. Then it passes off and I'm as intelligent as ever. -- Samuel Beckett, "Endgame"
Re: Haproxy 2.2.3 source
how do you build armh ? can you share details ? if that's cross build, we can easily add to github actions, for example. unfortunately, it is hard to get armh native CI. ср, 9 сент. 2020 г. в 20:01, Willy Tarreau : > On Tue, Sep 08, 2020 at 11:47:25PM +0200, Vincent Bernat wrote: > > ? 8 septembre 2020 16:13 -04, Alex Evonosky: > > > > > Just compiling 2.2.3 and getting this reference: > > > > > > > > > /haproxy-2.2.3/src/thread.c:212: undefined reference to > > > `_Unwind_Find_FDE' > > > > I am getting the same issue on armhf only. Other platforms don't get > > this issue. On this platform, we only get: > > > > w DF *UND* GLIBC_2.4 __gnu_Unwind_Find_exidx > > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException > > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase > > 00016e1c gDF .text 0022 GCC_4.3.0 _Unwind_Backtrace > > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind > > 00016dd4 gDF .text 0022 GCC_3.3 _Unwind_Resume_or_Rethrow > > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase > > 0001662c gDF .text 0036 GCC_3.5 _Unwind_VRS_Set > > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume > > 000169d8 gDF .text 02ba GCC_3.5 _Unwind_VRS_Pop > > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart > > 000165cc gDF .text 0002 GCC_3.5 _Unwind_Complete > > 00017184 gDF .text 0012 GCC_3.0 > _Unwind_GetLanguageSpecificData > > 000165dc gDF .text 0036 GCC_3.5 _Unwind_VRS_Get > > 000164f0 gDF .text 0004 GCC_3.3 _Unwind_GetCFA > > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException > > > > So, older symbols are: > > > > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException > > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase > > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind > > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase > > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume > > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart > > 00017184 gDF .text 0012 GCC_3.0 > _Unwind_GetLanguageSpecificData > > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException > > > > Moreover, comment says _Unwind_Find_DFE doesn't take arguments, but the > > signature I have in glibc is: > > > > fde * > > _Unwind_Find_FDE (void *pc, struct dwarf_eh_bases *bases) > > Ah I'm really angry because I tested on many platforms, *including* armhf, > but now I'm not seeing it, so either I failed on one test or it depends > on the compiler combination :-( > > The principle was just to force to load libgcc_s that tends to cause > aborts on reload for some users in chroots when threads exit, because > pthread_exit() tends to be the first one to require libgcc_s and it's > quite too late. > > In the mean time you can probably get away with this: > > diff --git a/src/thread.c b/src/thread.c > index 5eb68e33a..167e26e9d 100644 > --- a/src/thread.c > +++ b/src/thread.c > @@ -201,7 +201,7 @@ static void __thread_init(void) > exit(1); > } > > -#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && > !defined(__clang__) > +#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && > !defined(__clang__) && !defined(__arm__) > /* make sure libgcc_s is already loaded, because pthread_exit() may > * may need it on exit after the chroot! _Unwind_Find_FDE() is > defined > * there since gcc 3.0, has no side effect, doesn't take any > argument > > > But I'd really like to find a *reliable* way to force libgcc_s to be loaded > if available and required (i.e. not if statically built). I thought about > causing a 64/64 bit divide that usually is done via divsi3 and friend but > on x86_64 (where the problem was encountered) it will not do it. > > I'm thinking about something: maybe I can have a look around > pthread_getspecific() and pthread_key_create(). I would be very surprised > if they didn't rely on some compiler-specific help from libgcc_s. > > I'll keep testing and will get back to you guys. > > Willy > >
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 09, 2020 at 04:57:58PM +0200, William Dauchy wrote: > > I think it's not easy to reproduce these tests, you need a high enough > > latency between haproxy and the client so that the handshake is not > > already completed when you evaluate the rule, and of course you need > > to make sure the client sends using early data. I don't remember how > > Olivier used to run his tests but I remember that it was a bit tricky, > > so it's very possible that you never fall into the situation where you > > can see the unvalidated early data yet. > > It means my understanding of this fetcher was wrong indeed. > For me the protection was here: > http-request wait-for-handshake if ! METH_GET > and the fetcher here to log whether it was a 0rtt request or not. > In reality, it means all our requests have completed the handshake > when the rule is evaluated (which is surprising looking at the number > we have). That seems strange indeed but looking at the code that's what I'm seeing. Was your access to ssl_fc_has_early placed before or after the rule above ? If it's after it must indeed report false. > So maybe we can possibly work on an alternative fetcher to know > whether this was a 0rtt request? Or is there another way? I seem to remember there was one but can't find it, so I may have been confused. With this said, it doesn't provide a big information since once the handshake is completed, it's exactly identical to a regular one. But it can be nice for statistics at least. Willy
Re: Haproxy 2.2.3 source
On Tue, Sep 08, 2020 at 11:47:25PM +0200, Vincent Bernat wrote: > ? 8 septembre 2020 16:13 -04, Alex Evonosky: > > > Just compiling 2.2.3 and getting this reference: > > > > > > /haproxy-2.2.3/src/thread.c:212: undefined reference to > > `_Unwind_Find_FDE' > > I am getting the same issue on armhf only. Other platforms don't get > this issue. On this platform, we only get: > > w DF *UND* GLIBC_2.4 __gnu_Unwind_Find_exidx > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase > 00016e1c gDF .text 0022 GCC_4.3.0 _Unwind_Backtrace > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind > 00016dd4 gDF .text 0022 GCC_3.3 _Unwind_Resume_or_Rethrow > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase > 0001662c gDF .text 0036 GCC_3.5 _Unwind_VRS_Set > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume > 000169d8 gDF .text 02ba GCC_3.5 _Unwind_VRS_Pop > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart > 000165cc gDF .text 0002 GCC_3.5 _Unwind_Complete > 00017184 gDF .text 0012 GCC_3.0 _Unwind_GetLanguageSpecificData > 000165dc gDF .text 0036 GCC_3.5 _Unwind_VRS_Get > 000164f0 gDF .text 0004 GCC_3.3 _Unwind_GetCFA > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException > > So, older symbols are: > > 000165d0 gDF .text 000c GCC_3.0 _Unwind_DeleteException > d1f6 gDF .text 0002 GCC_3.0 _Unwind_GetTextRelBase > 00016df8 gDF .text 0022 GCC_3.0 _Unwind_ForcedUnwind > d1f0 gDF .text 0006 GCC_3.0 _Unwind_GetDataRelBase > 00016db0 gDF .text 0022 GCC_3.0 _Unwind_Resume > 00017178 gDF .text 000a GCC_3.0 _Unwind_GetRegionStart > 00017184 gDF .text 0012 GCC_3.0 _Unwind_GetLanguageSpecificData > 00016d8c gDF .text 0022 GCC_3.0 _Unwind_RaiseException > > Moreover, comment says _Unwind_Find_DFE doesn't take arguments, but the > signature I have in glibc is: > > fde * > _Unwind_Find_FDE (void *pc, struct dwarf_eh_bases *bases) Ah I'm really angry because I tested on many platforms, *including* armhf, but now I'm not seeing it, so either I failed on one test or it depends on the compiler combination :-( The principle was just to force to load libgcc_s that tends to cause aborts on reload for some users in chroots when threads exit, because pthread_exit() tends to be the first one to require libgcc_s and it's quite too late. In the mean time you can probably get away with this: diff --git a/src/thread.c b/src/thread.c index 5eb68e33a..167e26e9d 100644 --- a/src/thread.c +++ b/src/thread.c @@ -201,7 +201,7 @@ static void __thread_init(void) exit(1); } -#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && !defined(__clang__) +#if defined(__GNUC__) && (__GNUC__ >= 3) && defined(__GNU_LIBRARY__) && !defined(__clang__) && !defined(__arm__) /* make sure libgcc_s is already loaded, because pthread_exit() may * may need it on exit after the chroot! _Unwind_Find_FDE() is defined * there since gcc 3.0, has no side effect, doesn't take any argument But I'd really like to find a *reliable* way to force libgcc_s to be loaded if available and required (i.e. not if statically built). I thought about causing a 64/64 bit divide that usually is done via divsi3 and friend but on x86_64 (where the problem was encountered) it will not do it. I'm thinking about something: maybe I can have a look around pthread_getspecific() and pthread_key_create(). I would be very surprised if they didn't rely on some compiler-specific help from libgcc_s. I'll keep testing and will get back to you guys. Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
Hello Willy, Thank you for your answer, On Wed, Sep 9, 2020 at 4:39 PM Willy Tarreau wrote: > If I remember well, the principle consists in detecting whether or not > the request was received using TLS early data (0-rtt) before the handshake > was completed. The problem is that early data may trivially be captured > and replayed, so you don't necessarily want to accept all of them, only > replay-safe requests. Typically a login page that is limited to 3 > attempts before blocking should not be allowed, but fetching a favicon > is totally safe. > > Once the handshake ends you'll know whether it was safe or not, so you > can actually decide to wait on this function to return false to indicate > that the request is complete and not replayed, or just use it to return > a "425 too early" response for certain sensitive resources. > > I think it's not easy to reproduce these tests, you need a high enough > latency between haproxy and the client so that the handshake is not > already completed when you evaluate the rule, and of course you need > to make sure the client sends using early data. I don't remember how > Olivier used to run his tests but I remember that it was a bit tricky, > so it's very possible that you never fall into the situation where you > can see the unvalidated early data yet. It means my understanding of this fetcher was wrong indeed. For me the protection was here: http-request wait-for-handshake if ! METH_GET and the fetcher here to log whether it was a 0rtt request or not. In reality, it means all our requests have completed the handshake when the rule is evaluated (which is surprising looking at the number we have). So maybe we can possibly work on an alternative fetcher to know whether this was a 0rtt request? Or is there another way? Thanks, -- William
Re: `ssl_fc_has_early` fetcher and 0rtt
Hi William! On Wed, Sep 09, 2020 at 12:02:03PM +0200, William Dauchy wrote: > On Wed, Sep 9, 2020 at 10:48 AM William Dauchy wrote: > > I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm > > unable to find a single request where it returns 1. > > (sorry, forgot to mention, all of these tests were done on v2.2.x) If I remember well, the principle consists in detecting whether or not the request was received using TLS early data (0-rtt) before the handshake was completed. The problem is that early data may trivially be captured and replayed, so you don't necessarily want to accept all of them, only replay-safe requests. Typically a login page that is limited to 3 attempts before blocking should not be allowed, but fetching a favicon is totally safe. Once the handshake ends you'll know whether it was safe or not, so you can actually decide to wait on this function to return false to indicate that the request is complete and not replayed, or just use it to return a "425 too early" response for certain sensitive resources. I think it's not easy to reproduce these tests, you need a high enough latency between haproxy and the client so that the handshake is not already completed when you evaluate the rule, and of course you need to make sure the client sends using early data. I don't remember how Olivier used to run his tests but I remember that it was a bit tricky, so it's very possible that you never fall into the situation where you can see the unvalidated early data yet. Hoping this helps, Willy
Re: `ssl_fc_has_early` fetcher and 0rtt
On Wed, Sep 9, 2020 at 10:48 AM William Dauchy wrote: > I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm > unable to find a single request where it returns 1. (sorry, forgot to mention, all of these tests were done on v2.2.x) -- William
Re: [RFC PATCH] MAJOR: ssl: Support for validating backend certificates with URI SANs (subjectAltName)
On Tue, 8 Sep 2020 at 12:39, Teo Klestrup Röijezon wrote: > > Hey Willy, sorry about the delay.. managed to get sick right after that stuff. > > > I don't understand what you mean here in that it does not make sense to > > you. Actually it's not even about overriding verifyhost, it's more that > > we match that the requested host (if any) is indeed supported by the > > presented certificate. The purpose is to make sure that the connection > > is not served by a server presenting a valid cert which doesn't match > > the authority we're asking for. And if we don't send any servername, > > then we can still enforce the check against a hard-coded servername > > presented in verifyhost. > > To my mind, `verifyhost` is more or less an acknowledgement that "no, this > isn't quite set up perfectly, but we can at least verify with some caveats". It's about proper certificate validation. > Otherwise, the host could just be taken from the address in the `connect` > keyword before SNI? No, because the address often is an actual IP address, not a hostname, and if it is a hostname, then it often does not match the actual certificate SAN. In a common setup we'd be serving a HTTPS site like www.example.org, so that is the certificate. However the backend servers that haproxy accesses is not www.example.org - because www.example.org would actually point to haproxy, not a backend server. Backend servers could be www1.example.org and www2.example.org, so there is a mismatch. So in the most common configuration, those hostnames do not match, which is why we need to specify it, with the intention to fully validate it. Lukas
`ssl_fc_has_early` fetcher and 0rtt
Hello, I'm trying to understand `ssl_fc_has_early` fetcher behavior as I'm unable to find a single request where it returns 1. Our config has 0rtt enabled and it is as follow: ``` global log 127.0.0.1 format rfc5424 local0 info daemon stats socket /var/lib/haproxy/stats level admin mode 600 user haproxy group haproxy expose-fd listeners stats timeout 2m maxconn 524288 user haproxy group haproxy set-dumpable tune.bufsize 33792 tune.runqueue-depth 1200 tune.sched.low-latency on tune.ssl.cachesize 0 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES256-SHA ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384 ssl-server-verify none hard-stop-after 183s nbthread 16 cpu-map auto:1/1-16 0-15 spread-checks 5 chroot /etc/haproxy/chroot strict-limits defaults mode http log global option httplog option http-keep-alive option forwardfor except 127.0.0.0/8 option redispatch 1 option http-ignore-probes retries 3 retry-on conn-failure empty-response response-timeout 0rtt-rejected timeout http-request 10s timeout queue 1s timeout connect 10s timeout client 300s timeout server 300s timeout http-keep-alive 10s timeout check 5s maxconn 524288 balance roundrobin http-reuse always default-server inter 10s fastinter 1s fall 3 slowstart 20s observe layer7 error-limit 5 on-error fastinter pool-purge-delay 10s tfo allow-0rtt pool-low-conn 32 http-check expect rstatus ^(200|429) frontend fe_foo bind x:80 name http_ip process 1/all tfo bind x:443 name https_ip process 1/all ssl crt /etc/haproxy/tls/fe_foo alpn h2,http/1.1 tfo allow-0rtt log-format-sd [fc_rtt=\"%[fc_rtt]\"\ fc_0rtt=\"%[ssl_fc_has_early]\"\ fc_resumed=\"%[ssl_fc_is_resumed]\"] http-request disable-l7-retry if ! METH_GET http-request wait-for-handshake if ! METH_GET use_backend bar ``` - Am I right that tune.ssl.cachesize is not involved here? - What's odd is that I do find requests which returns 1 with `ssl_fc_is_resumed` fetcher. - I looked at `ssl_fc_has_early` and I did not found anything strange. - I tried to remove `wait-for-handshake` and `tune.ssl.cachesize 0` without success >From the client point of view I'm able to test it through: openssl s_client -tls1_3 -state -connect x:443 -servername foo.com -keylogfile keylogfile.log -sess_out ~/ssl/session.data openssl s_client -tls1_3 -state -connect x:443 -servername foo.com -keylogfile keylogfile.log -sess_in ~/ssl/session.data -early_data httpget.txt The second time I'm able to see: Reused, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was accepted Verify return code: 0 (ok) So to me, from the client point of view I'm able to make use of 0rtt. Any idea how to explain `ssl_fc_has_early` fetcher behaviour? Am I missing something in my config? Does someone have a different behaviour with a similar config? Best, -- William
Re: Not sure if my mails to haproxy mailing lists are being blocked.
Badari, Am 09.09.20 um 04:10 schrieb Badari Prasad: > Hi Admin, > Need help here , not sure if my mails to the mailing lists are being > blocked. Can you kindly check. > I am not the Admin, but I can confirm that your emails reach the list just fine. You can easily check yourself using the mailing list archives: https://www.mail-archive.com/haproxy@formilux.org/ You should be a bit more patient, though. Bumping your thread after just roughly 24 hours is not nice. I'm however not able to help with your issue, that's why I did not reply. Best regards Tim Düsterhus
