Hello HAProxy Team! I have a question about one of your posts...
Hi HAProxy Team, Vanessa from Toptal here! I promise this will be quick. I just read one of your posts - https://www.haproxy.com/blog/power-your-consul-service-mesh-with-haproxy/ and I was hoping that we could connect. I totally agree when you mention that Kubernetes makes configuring a service mesh easier tactically because you can run multiple containers inside a single pod, which is often referred to as running sidecar containers. When discussing microservices architecture and containerization, one set of production-proven tools has captured most of the attention in recent years: the service mesh. Would you be interested in adding this as an additional resource to your article? It will be a great update especially to your readers that are interested in the Kubernetes service. Here is the link for your reference: https://www.toptal.com/kubernetes/service-mesh-comparison Looking forward to hearing your thoughts on this and stay safe always. Regards, Vanessa
Re: CVE-2021-40346, the Integer Overflow vulnerability
Hello Jonathan, On Wed, 8 Sept 2021 at 21:28, Jonathan Greig wrote: > > Hello! My name is Jonathan Greig and I'm a reporter for ZDNet. I'm > writing a story about CVE-2021-40346 and I was wondering if > Ha Proxy had any comment about the vulnerability. Just making sure you are aware that this is a public mailing list: https://www.mail-archive.com/haproxy@formilux.org/msg41140.html You can find the CVE-2021-40346 announcement with comments here on this mailing list: https://www.mail-archive.com/haproxy@formilux.org/msg41114.html Short blog article on haproxy.com: https://www.haproxy.com/blog/september-2021-duplicate-content-length-header-fixed/ Long Jfrog article with (lots) of technical details: https://jfrog.com/blog/critical-vulnerability-in-haproxy-cve-2021-40346-integer-overflow-enables-http-smuggling/ Hope this helps, Lukas
CVE-2021-40346, the Integer Overflow vulnerability
Hello! My name is Jonathan Greig and I'm a reporter for ZDNet. I'm writing a story about CVE-2021-40346 and I was wondering if Ha Proxy had any comment about the vulnerability. Thanks, Jonathan
Re: PPv2: clarify "protocol block" and how to ignore it?
On Fri, 2021-07-09 at 16:23 -0700, Jacob Champion wrote: > Hi all, > > [Please keep me CC'd on replies, if you would.] > > Quick interoperability question, regarding this paragraph in the PROXY > v2 spec: Hello, Just to double-check, is there a better way for me to have asked my question, or something I missed in the ML etiquette doc? Thanks! --Jacob
Re: I just broke opentracing :-(
On 09/08/2021 07:57 PM, Miroslav Zagorac wrote: On 09/08/2021 07:42 PM, Willy Tarreau wrote: No rush on this one, I'll let you think about it, just let me know if we need to temporarily disable it from the CI, or if you just want a day or two of checking before knowing if you can get a quick solution or if it will take more time. Thank you for understanding. For a start it would be good to exclude ot filter from the compilation process so as not to interfere with the development and testing of other code. Next, I could throw out the part related to context transfer via variables with #ifdef/#endif blocks, so that the ot filter code is compiled without it. After that the problem with variables could be solved without interfering with the rest of the coding/compilation process and the ot filter will be functional (except for the part with context transfer via variables, but the question is whether anyone uses it). The variable 'sess.ot.uuid' is just set, it is not used anywhere (as far as memory serves me). I imagined that this could be used if the user needs it, but it is not necessary. OK, it is still used but again only as an example that it can be used and not that it should be used. In our examples uuid is set as one of the data from the opentracing 'baggage' data group: test/ctx/ot.cfg: --- baggage "haproxy_id" var(sess.ot.uuid) --- -- Zaga What can change the nature of a man?
Re: I just broke opentracing :-(
On 09/08/2021 07:42 PM, Willy Tarreau wrote: No rush on this one, I'll let you think about it, just let me know if we need to temporarily disable it from the CI, or if you just want a day or two of checking before knowing if you can get a quick solution or if it will take more time. Thank you for understanding. For a start it would be good to exclude ot filter from the compilation process so as not to interfere with the development and testing of other code. Next, I could throw out the part related to context transfer via variables with #ifdef/#endif blocks, so that the ot filter code is compiled without it. After that the problem with variables could be solved without interfering with the rest of the coding/compilation process and the ot filter will be functional (except for the part with context transfer via variables, but the question is whether anyone uses it). The variable 'sess.ot.uuid' is just set, it is not used anywhere (as far as memory serves me). I imagined that this could be used if the user needs it, but it is not necessary. -- Zaga What can change the nature of a man?
Re: I just broke opentracing :-(
On Wed, Sep 08, 2021 at 07:23:59PM +0200, Miroslav Zagorac wrote: > On 09/08/2021 06:46 PM, Willy Tarreau wrote: > > I have no doubt about this, what I mean is that once such needs are > > identified, as much as possible we should try to move these parts into > > the proper location (vars), or if really needed, that must be clearly > > explained around the code so that someone else may try to figure what > > to change there. For example I've seen a function that parses scope > > names and turns them list heads, that's the perfect example of something > > that ought to have been adapted into vars.c if what was provided was not > > sufficient, because since then new scopes were added without anyone > > noticing they were listed there, so that function was not updated (there > > I doubt anyone cares but conceptually it's a problem). > > It is clear to me.. at the time when the code for the ot filter was > being developed, it was easier for me to write the functions I was > missing (ie the content I was missing in the existing functions) and > then see what to do with it .. and it remained like that after > completion. At the time, I didn't think that something more > significant could change in working with variables one day. I know how we end up like this, I'm not saying that in bad terms, just as something to be careful about in the future. The codebase is huge nowadays and no single person knows even 10% of it anymore, so if we continue to create hacks or temporary stuff in various corners it's unmaintainable and breaks all over every time someone tries to fix a design problem. And that's already what happened with these shitty variables that have been causing us some gray hair for a few years now. > I was wrong, obviously. No worries. > > > The moment we write the variables we know their > > > names but at the moment we read them we don't know how many there are > > > and their complete name, rather we take all the variables of a > > > particular prefix. > > > > Hmmm that may become quite problematic then since we don't have that > > name anymore :-/ Are these variables used only for internal stuff in > > OT or are they used to exchange with the rest of the configuration ? > > In opentracing there is a context of data, it is a group of data that > is transferred from one place where they are used to another place. > > This data is defined as a key:value, where the use of haproxy variables > suited me perfectly. The problem is that it is not known in advance > what the keys of this data are and how much of this data is, this is > set by the user in the opentracing configuration (which has nothing to > do with haproxy). OK thanks for explaining! Now I understand indeed why you reused variables for this. > The operations that handle this data are 'inject' and 'extract', the > first writes all the data from the opentracing context to the haproxy > variable and the second extracts the data from the haproxy variable to > the opentracing context. > > During the 'inject' operation, it is known which are the variables (or > their names) because they are generated from the 'key' part of the > opentracing data. > > During the 'extract' operation this is not known, ie only the prefix of > the variable under which one of the 'inject' operations of that > variable is written is known. Got it. It's just as if you'd serialize a blob that it extracted on the other side, in some sort. > > Do you have a concrete example of a use case for such prefixes on > > receipt ? For example if you're dumping all variables from your current > > request, maybe that's enough. Or if you need prefixes for dumping all > > variables of a particular scope, we could proceed differently. > > Context transfer is conceived in two ways, one is via variables (if > transferred within the same haproxy process) and the other is via http > header (if transferred between different processes, there does not have > to be haproxy as one of the processes). > > An example of this is the configuration from the test/ctx directory, > more precisely the test/ctx/ot.cfg file. The only explicit variable I'm seeing there is sess.ot.uuid, so I guess that the rest is totally transparent, and that if needed we can arrange so that sess.ot.uuid is always created on new sessions. > An example of context transfer via http header is in the test/fe and > test/be directories where data is transferred from the frontend process > via the http header to the backend process, which can then add > opentracing data to the trace initiated in the frontend process. > > In principle, context transfer via haproxy variables can be omitted > because this data can also be transferred via opentracing process > without interfering with its context, because this process is started > and completed in the same haproxy program and does not need to be > exposed to the outside. OK. > But maybe someone needs it; I implemented it because it can be done and > not because
Re: I just broke opentracing :-(
On Wed, Sep 08, 2021 at 06:30:15PM +0200, Miroslav Zagorac wrote: > there is a reason why i used some functions related to > reading/setting/searching variables. If I could use the original haproxy > functions, I wouldn't write these because duplicating the > code doesn't make sense. I have no doubt about this, what I mean is that once such needs are identified, as much as possible we should try to move these parts into the proper location (vars), or if really needed, that must be clearly explained around the code so that someone else may try to figure what to change there. For example I've seen a function that parses scope names and turns them list heads, that's the perfect example of something that ought to have been adapted into vars.c if what was provided was not sufficient, because since then new scopes were added without anyone noticing they were listed there, so that function was not updated (there I doubt anyone cares but conceptually it's a problem). > What I have seen so far is that the 'name' member of the vars structure > has actually been deleted and that the var_clear() function has been > given another parameter. Yep, previously, there was a "name" member pointing to a pre-allocated entry into the names table, that contained the part after the scope. The goal was to allocate names only once at boot time when variables couldn't be added at runtime. Now instead we don't keep this name and we only keep a 64-bit hash of that same part. > Generally, in the ot filter it is possible to transfer data via a group > of variables with a common prefix, so that in one place we set all the > variables and their values and in another we read all the variables > with a given prefix. OK I see, thanks for explaining! > The moment we write the variables we know their > names but at the moment we read them we don't know how many there are > and their complete name, rather we take all the variables of a > particular prefix. Hmmm that may become quite problematic then since we don't have that name anymore :-/ Are these variables used only for internal stuff in OT or are they used to exchange with the rest of the configuration ? Do you have a concrete example of a use case for such prefixes on receipt ? For example if you're dumping all variables from your current request, maybe that's enough. Or if you need prefixes for dumping all variables of a particular scope, we could proceed differently. > I need to take a closer look at how ot filter source can be redesigned > to retain functionality and use the new structure definition to store > variables. Just thinking loud, if you absolutely need to enumerate a list of variables when setting them, maybe it could be possible to enumerate their names into another well-known one and look up that one first on the other side? Thanks! Willy
Re: I just broke opentracing :-(
On 09/08/2021 04:41 PM, Willy Tarreau wrote: Hi Miroslav, I just discovered that the changes I've made to the totally broken variables API now broke opentracing because apparently it's using some of the variable code's internals for its own use: https://github.com/haproxy/haproxy/runs/3545475810 That's annoying because unbreaking the API was already extremely complicated due to the global name table limitation that became a feature on its own, but now I see that some upper level code outside of vars/ is also making intimate use of that broken API (e.g. in flt_ot_vars_unset), and I have no idea what it tries to do nor why. Most of the breakage comes from all the undocumented functions flt_ot_vars_* but I don't even understand why this does not use the vars_* API that Lua, CLI and SPOE already use. Note that it doesn't seem that the breakage is large, so most likely we can figure together a few changes to perform to fix it, but reversing all that is quite a pain, and even if I went through the process of trying to build it I wouldn't know if what I'm doing works or not :-/ The changes that broke are in the following commit range which tries hard to minimize the visible changes: 10080716b..55f8a830d Hello Willy, there is a reason why i used some functions related to reading/setting/searching variables. If I could use the original haproxy functions, I wouldn't write these because duplicating the code doesn't make sense. What I have seen so far is that the 'name' member of the vars structure has actually been deleted and that the var_clear() function has been given another parameter. Generally, in the ot filter it is possible to transfer data via a group of variables with a common prefix, so that in one place we set all the variables and their values and in another we read all the variables with a given prefix. The moment we write the variables we know their names but at the moment we read them we don't know how many there are and their complete name, rather we take all the variables of a particular prefix. I need to take a closer look at how ot filter source can be redesigned to retain functionality and use the new structure definition to store variables. -- Zaga What can change the nature of a man?
I just broke opentracing :-(
Hi Miroslav, I just discovered that the changes I've made to the totally broken variables API now broke opentracing because apparently it's using some of the variable code's internals for its own use: https://github.com/haproxy/haproxy/runs/3545475810 That's annoying because unbreaking the API was already extremely complicated due to the global name table limitation that became a feature on its own, but now I see that some upper level code outside of vars/ is also making intimate use of that broken API (e.g. in flt_ot_vars_unset), and I have no idea what it tries to do nor why. Most of the breakage comes from all the undocumented functions flt_ot_vars_* but I don't even understand why this does not use the vars_* API that Lua, CLI and SPOE already use. Note that it doesn't seem that the breakage is large, so most likely we can figure together a few changes to perform to fix it, but reversing all that is quite a pain, and even if I went through the process of trying to build it I wouldn't know if what I'm doing works or not :-/ The changes that broke are in the following commit range which tries hard to minimize the visible changes: 10080716b..55f8a830d I would highly appreciate it if you could have a look, figure what needs to be done, if possible using the public vars API, otherwise that we discuss what needs to be extended, and if you could document these functions, because, to be honest, the following doesn't help me much: /*** * NAME * flt_ot_vars_get - * * ARGUMENTS * s - * scope - * prefix - * opt- * err- * * DESCRIPTION * - * * RETURN VALUE * - */ struct otc_text_map *flt_ot_vars_get(struct stream *s, const char *scope, const char *prefix, uint opt, char **err) Thanks, Willy
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
On Wed, Sep 08, 2021 at 12:34:49PM +0200, Aleksandar Lazic wrote: > On 08.09.21 11:07, Willy Tarreau wrote: > > On Wed, Sep 08, 2021 at 01:58:00PM +0500, ??? wrote: > > > ??, 8 . 2021 ?. ? 13:54, Willy Tarreau : > > > > > > > On Wed, Sep 08, 2021 at 12:05:23PM +0500, ??? wrote: > > > > > Hello, Bob > > > > > > > > > > I tracked an issue https://github.com/haproxy/haproxy/issues/1386 > > > > > > > > > > > > > > > let's track activity there > > > > > > > > Quite frankly, I'm seriously wondering how long we'll want to keep > > > > supporting that constantly breaking library. Does it still provide > > > > > > > > > > by "let us track activity" I do not mean that we are going to maintain > > > BoringSSL :) > > > > > > people will come from time to time with BoringSSL support request. > > > Existing > > > github issue is good to redirect them to. > > > > Oh this is how I understood it as well, I just think that you and a > > handful of others have already spent a lot of energy on that lib and > > I was only encouraging you not to spend way more than what you find > > reasonable after this issue is created :-) > > Is there another library which have the quic stuff implemented which > can be used for quic development? Fred told me that he manages to build it using the fork of openssl that contains the proper stuff. Hopefully it should get merged soon. But with BoringSSL the problem is that something that works on monday suddenly fails to build on tuesday and there's no stable branch so you're just forced to change your API to adapt to it on the fly. I don't want to blame them too much because they always warned against this. It's been convenient to start on QUIC but as soon as we can avoid this pain it will be better to get rid of it. Willy
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
On 08.09.21 11:07, Willy Tarreau wrote: On Wed, Sep 08, 2021 at 01:58:00PM +0500, ??? wrote: ??, 8 . 2021 ?. ? 13:54, Willy Tarreau : On Wed, Sep 08, 2021 at 12:05:23PM +0500, ??? wrote: Hello, Bob I tracked an issue https://github.com/haproxy/haproxy/issues/1386 let's track activity there Quite frankly, I'm seriously wondering how long we'll want to keep supporting that constantly breaking library. Does it still provide by "let us track activity" I do not mean that we are going to maintain BoringSSL :) people will come from time to time with BoringSSL support request. Existing github issue is good to redirect them to. Oh this is how I understood it as well, I just think that you and a handful of others have already spent a lot of energy on that lib and I was only encouraging you not to spend way more than what you find reasonable after this issue is created :-) Is there another library which have the quic stuff implemented which can be used for quic development? Willy
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
On Wed, Sep 08, 2021 at 01:58:00PM +0500, ??? wrote: > ??, 8 . 2021 ?. ? 13:54, Willy Tarreau : > > > On Wed, Sep 08, 2021 at 12:05:23PM +0500, ??? wrote: > > > Hello, Bob > > > > > > I tracked an issue https://github.com/haproxy/haproxy/issues/1386 > > > > > > > > > let's track activity there > > > > Quite frankly, I'm seriously wondering how long we'll want to keep > > supporting that constantly breaking library. Does it still provide > > > > by "let us track activity" I do not mean that we are going to maintain > BoringSSL :) > > people will come from time to time with BoringSSL support request. Existing > github issue is good to redirect them to. Oh this is how I understood it as well, I just think that you and a handful of others have already spent a lot of energy on that lib and I was only encouraging you not to spend way more than what you find reasonable after this issue is created :-) Willy
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
ср, 8 сент. 2021 г. в 13:54, Willy Tarreau : > On Wed, Sep 08, 2021 at 12:05:23PM +0500, ??? wrote: > > Hello, Bob > > > > I tracked an issue https://github.com/haproxy/haproxy/issues/1386 > > > > > > let's track activity there > > Quite frankly, I'm seriously wondering how long we'll want to keep > supporting that constantly breaking library. Does it still provide > by "let us track activity" I do not mean that we are going to maintain BoringSSL :) people will come from time to time with BoringSSL support request. Existing github issue is good to redirect them to. > any value that I'm not aware of ? It's not even possible to have > the same version at the beginning and at the end of a maintenance > branch, it's designed to constantly change and documented as such. > > I'd be strongly in favor for dropping it once it becomes too painful > to deal with, and I sense that we're starting to get closer to that > point now. > > Willy >
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
On Wed, Sep 08, 2021 at 12:05:23PM +0500, ??? wrote: > Hello, Bob > > I tracked an issue https://github.com/haproxy/haproxy/issues/1386 > > > let's track activity there Quite frankly, I'm seriously wondering how long we'll want to keep supporting that constantly breaking library. Does it still provide any value that I'm not aware of ? It's not even possible to have the same version at the beginning and at the end of a maintenance branch, it's designed to constantly change and documented as such. I'd be strongly in favor for dropping it once it becomes too painful to deal with, and I sense that we're starting to get closer to that point now. Willy
Re: [ANNOUNCE] HTX vulnerability from 2.0 to 2.5-dev
❦ 8 September 2021 09:02 +02, Artur: > Hello, > > Thank you. > > Could you please explain the version numbering differences between official > haproxy release and the linux distributions > packages ? > > For example : 2.4.4 (official) -> 2.4.3-2~bpo10+1 (Debian 10 > backports) 2.4.3-2~bpo10+1 means this is based on upstream version 2.4.3, second revision for Debian (-2), backport to Debian 10 (~bpo10), first iteration of the backport (+1). The changelog (in ~doc/haproxy/changelog.Debian.gz) gives a hint of the deviation compared to official upstream version: haproxy (2.4.3-2~bpo10+1) buster-backports; urgency=medium * Rebuild for buster-backports. -- Vincent Bernat Sat, 04 Sep 2021 15:19:43 +0200 haproxy (2.4.3-2) experimental; urgency=high * d/patches: fix missing header name length check in HTX (CVE-2021-40346). -- Vincent Bernat Sat, 04 Sep 2021 11:56:31 +0200 haproxy (2.4.3-1~bpo10+1) buster-backports; urgency=medium * Rebuild for buster-backports. -- Vincent Bernat Sat, 21 Aug 2021 16:47:45 +0200 haproxy (2.4.3-1) experimental; urgency=medium * New upstream release. * d/patches: remove patches applied upstream. * d/patches: h2: match absolute-path not path-absolute for :path. -- Vincent Bernat Sat, 21 Aug 2021 16:32:25 +0200 Debian packages are not based on 2.4.4 because they were prepared in advance to be ready when the vulnerability is announced. Packages based on 2.4.4 will get available later this week. -- Instrument your programs. Measure before making "efficiency" changes. - The Elements of Programming Style (Kernighan & Plauger)
Re: BoringSSL commit dddb60e breaks compilation of HAProxy
Hello, Bob
I tracked an issue https://github.com/haproxy/haproxy/issues/1386
let's track activity there
вт, 7 сент. 2021 г. в 22:58, Zakharychev, Bob :
> BoringSSL commit dddb60e, "Make most of crypto/x509 opaque.", breaks
> compilation of HAProxy with the following errors (log from compiling
> HAProxy 2.4.4 with BoringSSL master branch commit a03c34c, but I suppose
> all other versions are equally affected):
>
>
> …
>
> CC src/ssl_sample.o
>
> In file included from include/haproxy/listener-t.h:37,
>
> from include/haproxy/server-t.h:36,
>
> from include/haproxy/lb_map-t.h:26,
>
> from include/haproxy/backend-t.h:30,
>
> from include/haproxy/proxy-t.h:35,
>
> from include/haproxy/applet-t.h:31,
>
> from include/haproxy/obj_type.h:26,
>
> from src/ssl_sample.c:27:
>
> include/haproxy/openssl-compat.h: In function ‘X509_OBJECT_get0_X509_CRL’:
>
> include/haproxy/openssl-compat.h:182:23: error: dereferencing pointer to
> incomplete type ‘X509_OBJECT’ {aka ‘const struct x509_object_st’}
>
> if (a == NULL || a->type != X509_LU_CRL) {
>
>^~
>
> src/ssl_sample.c: In function ‘smp_fetch_ssl_x_key_alg’:
>
> include/haproxy/openssl-compat.h:122:37: error: dereferencing pointer to
> incomplete type ‘X509’ {aka ‘struct x509_st’}
>
> #define X509_get_X509_PUBKEY(x) ((x)->cert_info->key)
>
> ^~
>
> src/ssl_sample.c:716:55: note: in expansion of macro ‘X509_get_X509_PUBKEY’
>
> X509_PUBKEY_get0_param(, NULL, NULL, NULL,
> X509_get_X509_PUBKEY(crt));
>
>^~~~
>
> make: *** [Makefile:945: src/ssl_sample.o] Error 1
>
>
>
> Indeed, BoringSSL commit dddb60e “unexported” these structs “aligning with
> OpenSSL” and directs to “Use the accessor APIs instead”. I couldn't figure
> out an easy fix to this - simply removing the two macros conditional on
> OPENSSL_IS_BORINGSSL in affected places doesn't fully help because while
> X509_get_X509_PUBKEY() accessor is now defined, X509_OBJECT_get0_X509_CRL()
> is still missing in BoringSSL. Therefore I defer the fix to HAProxy SSL
> experts - maybe it's actually BoringSSL that needs to be fixed by adding
> the missing accessor, or maybe the single loop in ssl_set_cert_crl_file()
> over all X509 store objects needs to be broken into two loops: one over
> certs returned by X509_STORE_get1_certs() and another over CRLs returned by
> X509_STORE_get1_crls().
>
> Thanks in advance for taking a stab at this,
> Bob
>
>
>
>
>
