Re: HAProxy Network Namespace Support issues, and I also found a security flaw.
Hello, On Tue, 20 Jul 2021 at 08:13, Peter Jin wrote: > 2. There is a stack buffer overflow found in one of the files. Not > disclosing it here because this email will end up on the public mailing > list. If there is a "security" email address I could disclose it to, > what is it? It's secur...@haproxy.org, it's somehow well hidden in doc/intro.txt (that is the *starter* guide). I would definitely suggest putting it on the website haproxy.org, and in the repository move it to a different file, like MAINTAINERS. Lukas
Re: Replying to spam [was: Some Spam Mail]
On Thu, 15 Jul 2021 at 11:27, Илья Шипицин wrote: > > I really wonder what they will suggest. > > I'm not a spam source, since we do not have "opt in" policy, anybody can send > mail. so they do. > please address the issue properly, either change list policy or be calm with > my experiments. It's about common sense, not list policy. Please do your SPAM responding experiments without the list in CC. Thank you, Lukas
Re: set mss on backend site on version 1.7.9
Hello Stefan, On Tue, 13 Jul 2021 at 14:10, Stefan Fuhrmann wrote: > > Hello all, > > > First, we can not change to newer version so fast within the project. > > We are having on old installation of haproxy (1.7.9) and we have the > need to configure tcp- mss- value on backend site. > > > > Is that possible to change the mss- value on backend site? How? No. You can set the MSS on the frontend socket, but not on the backend socket. You need to work with your OS/kernel configuration. Lukas
Re: [PATCH 0/1] Replace issue templates by issue forms
Hello, On Wed, 23 Jun 2021 at 22:25, Willy Tarreau wrote: > > Hi Tim, Max, > > On Wed, Jun 23, 2021 at 09:38:12PM +0200, Tim Duesterhus wrote: > > Hi Willy, Lukas, List! > > > > GitHub finally launched their next evolution of issue templates, called > > issue > > forms, as a public beta: > > https://github.blog/changelog/2021-06-23-issues-forms-beta-for-public-repositories/ > > > > Instead of prefilling the issue creation form with some markdown that can be > > ignored or accidentally deleted issue forms will create different textareas, > > automatically formatting the issue correctly. The end result will be regular > > markdown that can be edited as usual. > > > > Beta implies that they might still slightly change in the future, possibly > > requiring some further adjustments. However as the final result no longer > > depends on the form itself we are not locking ourselves into some feature > > for eternity. > > > > Max and I worked together to migrate the existing templates to issue forms, > > cleaning up the old stuff that is no longer required. > > > > You can find an example bug report here: > > > > https://github.com/TimWolla/haproxy/issues/7 > > > > It looks just like before! > > Indeed, and I like the issue description and the proposed fix :-) > > > The new forms can be tested here: > > https://github.com/TimWolla/haproxy/issues/new/choose. > > > > I have deleted the old 'Question' template, because it no longer is > > required, > > as the user can't simply delete the template from the field when there's > > separate fields :-) > > At first glance it indeed looks better than before. I'm personally fine > with the change. I'll wait for Lukas' Ack (or comments) before merging > though. Thanks for this, I like it! What I'm missing in the new UI is the possibility for the user to preview the *entire* post, I'm always extensively using preview features everywhere. So this feels like a loss, although the user can preview the content of the individual input box. But that's not reason enough to hold up this change, I just wish the "send us your feedback" button [1] would actually work. Full Ack from me for this change, this will be very beneficial as we get higher quality reports and people won't be required to navigate through raw markdown, which is not user-friendly at all. cheers, lukas [1] https://github.blog/changelog/2021-06-23-issues-forms-beta-for-public-repositories/
Re: SSL Labs says my server isn't doing ssl session resumption
Hello Shawn, On Sun, 20 Jun 2021 at 14:03, Shawn Heisey wrote: > > On 6/20/2021 1:52 AM, Lukas Tribus wrote: > > Can you try disabling threading, by putting nbthread 1 in your config? > > That didn't help. From testssl.sh: > > SSL Session ID support yes > Session Resumption Tickets: yes, ID: no It's a haproxy bug, affecting 2.4 releases, I've filed an issue in our tracker: https://github.com/haproxy/haproxy/issues/1297 Willy wrote: > I don't know if the config is responsible for this but I've just tested > on haproxy.org and it does work there: > > Session resumption (caching) Yes > Session resumption (tickets) Yes demo.haproxy.org suggests code running is quite old though: # curl -s http://demo.haproxy.org/ | grep released http://www.haproxy.org/; style="text-decoration: none;">HAProxy version 1.7.12-84aad5b, released 2019/10/25 # Regards, Lukas
Re: SSL Labs says my server isn't doing ssl session resumption
Hello Shawn, On Sun, 20 Jun 2021 at 08:39, Shawn Heisey wrote: > This is what SSL Labs now says for the thing that started this thread: > > Session resumption (caching)No (IDs assigned but not accepted) > Session resumption (tickets)Yes > > I'd like to get the caching item fixed, but I haven't figured that out > yet. Can you try disabling threading, by putting nbthread 1 in your config? An upgrade to 2.4.1 would also be advisable, it actually fixes a locking issue with SSL session cache (not sure whether that could really be the root cause though). Lukas
Re: SSL Labs says my server isn't doing ssl session resumption
On Wed, 16 Jun 2021 at 17:03, Илья Шипицин wrote: > > ssl sessions are for tls1.0 (disabled in your config) > tls1.2 uses tls tickets for resumption That is not true, you can disable TLS tickets and still get resumption on TLSv1.2. Disabling TLSv1.0 does not mean disabling Session ID caching. What do you see with testssl.sh ? Lukas
Re: [EXTERNAL] Re: built in ACL, REQ_CONTENT
Hello, On Tue, 8 Jun 2021 at 17:36, Godfrin, Philippe E wrote: > > Certainly, > > Postrgres sends this message across the wire: > > Jun 2 21:14:40 ip-172-31-77-193 haproxy[9031]: #0110x00: 00 00 00 4c 00 > 03 00 00 75 73 65 72 00 74 73 64 |...Luser.tsd| > Jun 2 21:14:40 ip-172-31-77-193 haproxy[9031]: #0110x10: 62 00 64 61 74 > 61 62 61 73 65 00 74 73 64 62 00 |b.database.tsdb.| > Jun 2 21:14:40 ip-172-31-77-193 haproxy[9031]: #0110x20: 61 70 70 6c 69 > 63 61 74 69 6f 6e 5f 6e 61 6d 65 |application_name| > Jun 2 21:14:40 ip-172-31-77-193 haproxy[9031]: #0110x30: 00 70 73 71 6c > 00 63 6c 69 65 6e 74 5f 65 6e 63 |.psql.client_enc| > Jun 2 21:14:40 ip-172-31-77-193 haproxy[9031]: #0110x40: 6f 64 69 6e 67 > 00 55 54 46 38 00 00 |oding.UTF8..| > > > > Bytes, 8 – are user\0 Byte 13 starts the userid. I would like to be able to > test that userid and make a routing decision on that. This is what the > HAProxy docs suggest: > > > > acl check-rw req.payload(8,32),hex -m sub 757365720074736462727700 And don't see how this is supposed to match? 62727700 is not what it's in your trace. Is the username tsdb, like in your trace, or is it tsdbrw, like in your ACL? Also, put a "tcp-request inspect-delay 5s" in front of the ACL (you can optimize performance later) and share the entire configuration. Please try to ask the actual question directly next time, so we can help you right away (https://xyproblem.info/). Thanks, Lukas
Re: built in ACL, REQ_CONTENT
Hello, On Mon, 7 Jun 2021 at 14:51, Godfrin, Philippe E wrote: > > Greetings! > > I can’t seem to find instructions on how to use this builtin ACL. Can someone > point me in the right direction, please? There is nothing specific about it, you use just like every other ACL. http-request deny if REQ_CONTENT http-request deny unless REQ_CONTENT Lukas
Re: how to write to a file safely in haproxy
Hello, On Wed, 26 May 2021 at 13:29, reshma r wrote: > > Hello all, > Periodically I need to write some configuration data to a file. > However I came across documentation that warned against writing to a file at > runtime. > Can someone give me advice on how I can achieve this safely? You'll have to elaborate what you are talking about. Are you talking about writing to the filesystem from a LUA scripts or other runtime code within haproxy? Then yes, don't do it, it will block the event loop and you will be in a world of hurt. Are you talking about writing and changing the configuration file, prior to a reload, manually or from a external process, that's not a problem at all. The issue is blocking filesystem access in the event-loop of the haproxy process itself. Explaining what the problem is you are trying to solve can get you more accurate proposals and solutions faster than asking about a particular solution (XY problem). Lukas
Re: haproxy hung with CPU usage at 100% Heeeelp, please!!!
The first thing I'd try is to disable multithreading (by putting nbthread 1 in the global section of the configuration), so if that helps. Lukas
Re: Table sticky counters decrementation problem
Hi Willy, On Tue, 30 Mar 2021 at 17:56, Willy Tarreau wrote: > > Guys, > > out of curiosity I wanted to check when the overflow happened: > > $ date --date=@$$(date +%s) * 1000) & -0x800) / 1000)) > Mon Mar 29 23:59:46 CEST 2021 > > So it only affects processes started since today. I'm quite tempted not > to wait further and to emit 2.3.9 urgently to fix this before other > people get trapped after reloading their process. Any objection ? No objection, but also: what a coincidence. I suggest you get a lottery ticket today. cheers, lukas
Re: Stick table counter not working after upgrade to 2.2.11
Hi Willy, On Tue, 23 Mar 2021 at 09:32, Willy Tarreau wrote: > > Guys, > > These two patches address it for me, and I could verify that they apply > on top of 2.2.11 and work there as well. This time I tested with two > counters at different periods 500 and 2000ms. Both Sander and Thomas now report that the issue is at least partially still there in 2.3.8 (which has all fixes, or 2.2.11 with patches), and that downgrading to 2.3.7 (which has none of the commits) works around the issue: https://www.mail-archive.com/haproxy@formilux.org/msg40093.html https://www.mail-archive.com/haproxy@formilux.org/msg40092.html Let's not yet publish stable bugfix releases, until this is properly diagnosed. Lukas
Re: Table sticky counters decrementation problem
Hello Thomas, this is a known issue in any release train other than 2.3 ... https://github.com/haproxy/haproxy/issues/1196 However neither 2.3.7 (does not contain the offending commits), nor 2.3.8 (contains all the fixes) should be affected by this. Are you absolutely positive that you are running 2.3.8 and not something like 2.2 or 2.0 ? Can you provide the full output of haproxy -vv? Thanks, Lukas
Re: zlib vs slz (perfoarmance)
Hello, On Mon, 29 Mar 2021 at 20:54, Илья Шипицин wrote: >> > Dear list, >> > >> > on browser load (html + js + css) I observe 80% of cpu spent on gzip. >> > also, I observe that zlib is probably one of the slowest implementation >> > my personal benchmark correlate with https://github.com/inikep/lzbench >> > >> > if so, should'n we switch to slz by default ? or am I missing something ? >> >> There is no default. >> >> zlib is optional. >> slz is optional. >> >> Haproxy is compiled with either zlib, slz or no compression library at all. >> >> >> What specifically are you suggesting to change in the haproxy source tree? > > > for example, docker image > https://hub.docker.com/_/haproxy So nothing we control directly. Docker images, package repositories, etc. This means filing requests at those places, convincing other people to switch from a well known library to an unknown library that isn't even packaged yet in most places, that those maintainers then have to support. I'm not sure how realistic that is. Like I said last year, this needs a marketing campaign: https://www.mail-archive.com/haproxy@formilux.org/msg38044.html What about the docker images from haproxytech? Are those zlib or slz based? Perhaps that would be a better starting point? https://hub.docker.com/r/haproxytech/haproxy-alpine Lukas
Re: Is there a way to deactivate this "message repeated x times"
Hello,
On Mon, 29 Mar 2021 at 15:25, Aleksandar Lazic wrote:
>
> Hi.
>
> I need to create some log statistics with awffull stats and I assume this
> messages
> means that only one line is written for 3 requests, is this assumption right?
>
> Mar 28 14:04:07 lb1 haproxy[11296]: message repeated 3 times: [
> ::::49445 [28/Mar/2021:14:04:07.234] https-in~ be_api/api_prim
> 0/0/0/13/13 200 2928 - - 930/900/8/554/0 0/0 {|Mozilla/5.0
> (Macintosh; Intel Mac OS X 10.13; rv:86.0) Gecko/20100101
> Firefox/86.0||128|TLS_AES_128_GCM_SHA256|TLSv1.3|} "GET
> https:/// HTTP/2.0"]
>
> Can this behavior be disabled?
This is not haproxy, this is your syslog server. Refer to the
documentation of the syslog server.
Lukas
Re: zlib vs slz (perfoarmance)
Hi Ilya, On Mon, 29 Mar 2021 at 15:34, Илья Шипицин wrote: > > Dear list, > > on browser load (html + js + css) I observe 80% of cpu spent on gzip. > also, I observe that zlib is probably one of the slowest implementation > my personal benchmark correlate with https://github.com/inikep/lzbench > > if so, should'n we switch to slz by default ? or am I missing something ? There is no default. zlib is optional. slz is optional. Haproxy is compiled with either zlib, slz or no compression library at all. What specifically are you suggesting to change in the haproxy source tree? Regards, Lukas
Re: HAProxy proxy protocol
Double post on discourse, please refrain from this practice in the future! https://discourse.haproxy.org/t/haproxy-proxy-protocol/6413/2 Thanks, Lukas
Re: [HAP 2.3.8] Is there a way to see why "" and "SSL handshake failure" happens
Hello, On Sat, 27 Mar 2021 at 11:52, Aleksandar Lazic wrote: > > Hi. > > I have a lot of such entries in my logs. > > ``` > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 > [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - > PR-- 1041/1011/0/0/0 0/0 "" > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23167 > [27/Mar/2021:11:48:20.523] https-in~ https-in/ -1/-1/-1/-1/0 0 0 - - > PR-- 1041/1011/0/0/0 0/0 "" Use show errors on the admin socket: https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.3-show%20errors > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166 > [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure > Mar 27 11:48:20 lb1 haproxy[14556]: ::::23166 > [27/Mar/2021:11:48:20.440] https-in/sock-1: SSL handshake failure That's currently a pain point: https://github.com/haproxy/haproxy/issues/693 Lukas
Fwd: OpenSSL Security Advisory
FYI -- Forwarded message - From: OpenSSL Date: Thu, 25 Mar 2021 at 15:03 Subject: OpenSSL Security Advisory To: , OpenSSL User Support ML , OpenSSL Announce ML -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 OpenSSL Security Advisory [25 March 2021] = CA certificate check bypass with X509_V_FLAG_X509_STRICT (CVE-2021-3450) Severity: High The X509_V_FLAG_X509_STRICT flag enables additional security checks of the certificates present in a certificate chain. It is not set by default. Starting from OpenSSL version 1.1.1h a check to disallow certificates in the chain that have explicitly encoded elliptic curve parameters was added as an additional strict check. An error in the implementation of this check meant that the result of a previous check to confirm that certificates in the chain are valid CA certificates was overwritten. This effectively bypasses the check that non-CA certificates must not be able to issue other certificates. If a "purpose" has been configured then there is a subsequent opportunity for checks that the certificate is a valid CA. All of the named "purpose" values implemented in libcrypto perform this check. Therefore, where a purpose is set the certificate chain will still be rejected even when the strict flag has been used. A purpose is set by default in libssl client and server certificate verification routines, but it can be overridden or removed by an application. In order to be affected, an application must explicitly set the X509_V_FLAG_X509_STRICT verification flag and either not set a purpose for the certificate verification or, in the case of TLS client or server applications, override the default purpose. OpenSSL versions 1.1.1h and newer are affected by this issue. Users of these versions should upgrade to OpenSSL 1.1.1k. OpenSSL 1.0.2 is not impacted by this issue. This issue was reported to OpenSSL on 18th March 2021 by Benjamin Kaduk from Akamai and was discovered by Xiang Ding and others at Akamai. The fix was developed by Tomáš Mráz. NULL pointer deref in signature_algorithms processing (CVE-2021-3449) = Severity: High An OpenSSL TLS server may crash if sent a maliciously crafted renegotiation ClientHello message from a client. If a TLSv1.2 renegotiation ClientHello omits the signature_algorithms extension (where it was present in the initial ClientHello), but includes a signature_algorithms_cert extension then a NULL pointer dereference will result, leading to a crash and a denial of service attack. A server is only vulnerable if it has TLSv1.2 and renegotiation enabled (which is the default configuration). OpenSSL TLS clients are not impacted by this issue. All OpenSSL 1.1.1 versions are affected by this issue. Users of these versions should upgrade to OpenSSL 1.1.1k. OpenSSL 1.0.2 is not impacted by this issue. This issue was reported to OpenSSL on 17th March 2021 by Nokia. The fix was developed by Peter Kästle and Samuel Sapalski from Nokia. Note OpenSSL 1.0.2 is out of support and no longer receiving public updates. Extended support is available for premium support customers: https://www.openssl.org/support/contracts.html OpenSSL 1.1.0 is out of support and no longer receiving updates of any kind. The impact of these issues on OpenSSL 1.1.0 has not been analysed. Users of these versions should upgrade to OpenSSL 1.1.1. References == URL for this Security Advisory: https://www.openssl.org/news/secadv/20210325.txt Note: the online version of the advisory may be updated with additional details over time. For details of OpenSSL severity classifications please see: https://www.openssl.org/policies/secpolicy.html -BEGIN PGP SIGNATURE- iQEzBAEBCAAdFiEEhlersmDwVrHlGQg52cTSbQ5gRJEFAmBcl6sACgkQ2cTSbQ5g RJGvnAgAtG6I7rfokDC9E5yB26KC3k0Vasfq5iH/aZz0CNRyOokWJBUyyNIVjqr0 2eZP7VsQT7zRM+tgh9c8MwH3FIghtpwJRJls4qZDHKoXts7JH4Ul4NLPd546x7xA GcKNwTD4NkZbTqtZ72NTgliInzrj0MCC8jqQrIIkcAIleGNzvZ0f64jdE+vBXoqX M2FOhWiA/JkAKtB3W7pthIt25qkOwHbrpTy+UUp/S5QD779NJ/EOYcsOFBRfLZiP gA6QILuW2L55lhG6Y2u+nVE3UI2hqd2hGgSAvDIPr2lVJxq0LQpgHca7Gj5bfIRo GLDz7n0FhN6n7NBqetP+nlHmYivcSg== =XIXK -END PGP SIGNATURE-
Re: Stick table counter not working after upgrade to 2.2.11
Hello, just a heads-up, this was also reported for 1.8: https://discourse.haproxy.org/t/counter-issues-on-1-8-29/6381/ Lukas On Tue, 23 Mar 2021 at 09:32, Willy Tarreau wrote: > > Guys, > > These two patches address it for me, and I could verify that they apply > on top of 2.2.11 and work there as well. This time I tested with two > counters at different periods 500 and 2000ms. > > Thanks, > Willy
Re: [ANNOUNCE] haproxy-1.6.16
Hello Willy,
On Sat, 20 Mar 2021 at 10:09, Willy Tarreau wrote:
> > 1.6 was EOL last year, I don't understand why there is a last release.
>
> There were some demands late last year and early this year to issue a
> last one with pending fixes to "flush the pipe" but it was terribly
> difficult to find enough time to go through the whole chain with the
> other nasty bugs that kept us busy.
>
> > Both 1.6 and 1.7 are marked for critical fixes but many fixes are pushed
> > in it. The risk is to introduce a late regression in this last version.
>
> There's always such a risk when doing backports unfortunately and it's
> always difficult to set the boundary between what is needed and what
> not. A lot of the issues I'm seeing there are crash-related, and
> others address not-so-funny recent changes in compilers behaviors
> leading to bad code generation. There are also some that were possibly
> not strictly necessary, but then they're obvious fixes (like the one
> on the timer that's incorrectly compared), and whose possible
> consequences are not always trivial to imagine (such as checks looping
> at 100% CPU every 24 days maybe depending on the tick's sign).
I agree that finding the sweet spot can be difficult, but I have to
say I share Vincent's concerns. I do feel like currently we backport
*a lot*, especially on those near-EOLed trees. When looking at the
list of backported patches, I don't feel like the age and remaining
lifetime is taken into consideration.
I don't want to be the monday morning quarterback, but in 1.7 we have
853926a9ac ("BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to
compare memory blocks") and I quote from the commit message:
> This is not correct and definitely confuses address sanitizers. In real
> life the problem doesn't have visible consequences.
> [...]
> there is no way such a multi-byte access will cross a page boundary
> and end up reading from an unallocated zone. This is why it was
> never noticed before.
This sounds like a backport candidate for "warm" stable branches
(maybe), but 1.7 and 1.8 feel too "cold" for this, even 8 - 9 months
ago.
This backport specifically causes a build failure of 1.7.13 on musl
(#760) - because of a missing backport, but that's just an example. 39
"MINOR" patches made it into 1.6.16, 62 patches in 1.7.13. While it is
true that a lot of "MINOR" tagged patches are actually important, this
is still a large number for a tree that is supposed to die so soon.
Very rarely do users build from source from such old trees anyway (and
those users would be especially conservative, definitely not
interested in generic, non-important improvements).
> But with this in mind, there were two options:
> - not releasing the latest fixes
You are talking about whether to publish a release or not for tree
X.Y, with the backports that are already in the tree. I don't think
that's the issue.
I think the discussion should be about what commits land in those old
trees in the first place. And I don't believe it is scalable to make
those decisions during your backporting sessions. Instead I believe we
should be more conservative when suggesting backports in the commit
message. Currently, we say "should/can be backported to X.Y" based on
whether it is *technically* possible to do so for supported trees, not
if it makes sense considering the age and lifetime of the suggested
tree. This is why I'm proposing a commit author should make such
considerations when suggesting backports. Avoiding backports to cold
trees of no-impact improvements and minor fixes for rare corner cases
should be a goal.
Unless we insist every single bug needs to be fixed on every single
supported release branch.
lukas
Re: [PATCH 1/1] MINOR: build: force CC to set a return code when probing options
Hello Bertrand, On Sun, 7 Mar 2021 at 00:53, Bertrand Jacquin wrote: > I am not proposing haproxy build-system to use -Werror here, I'm only > proposing to use -Werror when probing for options supported by the > compiler, as effectively clang return a code if 0 even if an option is > not supported. The fact haproxy does not use -Wno-clobbered today is > with clang build because of an internal bug in clang with how haproxy is > using stdin/stderr indirection. > > With the proposal above, Werror is only use to probe options, it is not > reflected in SPEC_CFLAGS. Got it, thanks for clarifying. Lukas
Re: [PATCH 1/1] MINOR: build: force CC to set a return code when probing options
Hello, On Sat, 6 Mar 2021 at 21:25, Bertrand Jacquin wrote: > > gcc returns non zero code if an option is not supported (tested > from 6.5 to 10.2). > > $ gcc -Wfoobar -E -xc - -o /dev/null < /dev/null > /dev/null 2>&1 ; echo $? > 1 > > clang always return 0 if an option in not recognized unless > -Werror is also passed, preventing a correct probing of options > supported by the compiler (tested with clang 6.0.1 to 11.1.0) -Werror is more than just "-Wunknown-warning-option" on clang. -Werror/ERR is specifically optional in the Makefile. If we want to change the haproxy build-system to do -Werror from now on we need a) discuss it and b) fix it all up. We can't hardcode -Werror and at the same time claim that it's an optional parameter. Lukas
Re: minconn, maxconn and fullconn (again, sigh!)
On Thu, 11 Feb 2021 at 05:31, Victor Sudakov wrote: > > Lukas Tribus wrote: > > > > On Wed, 10 Feb 2021 at 16:55, Victor Sudakov wrote: > > > > > > I can even phrase my question in simpler terms. What happens if the sum > > > total of all servers' maxconns in a backend is less than the maxconn > > > value in the frontend pointing to the said backend? > > > > Queueing for "timeout queue" amount of time, and then return 503 error > > And what happens in TCP mode, after the "timeout queue" amount of time? > I suppose the TCP connection with the client is reset? Yes, that's the only choice. > > > > > See: > > > > timeout queue > > https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-timeout%20queue > > > > maxconn > > https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#5.2-maxconn > > > > > > I really suggest you ignore minconn and fullconn, and focus on maxconn > > instead. The latter is a must-have (and must-understand). Really > > maxconn (global, frontend and per server ) is the single most > > important performance knob in haproxy. > > Maxconn is rather clear, especially when one is sure about two things: > > 1. A server's maxconn value is always a hard limit (notwithstanding if > there is a minconn configured for the server). Yes. > 2. Connections outnumbering the sum total of a backend's servers > maxconns are queued for the "timeout queue" amount of time and then > dropped. If, for any reason, we can't use another server with free slots, yes. > It would be nice however to understand minconn/fullconn too. If a > backend has several servers with identical minconn, maxconn and weight, > what's the point of having minconn? The load will be always distributed > evenly between all the servers notwithstanding minconn/fullconn, > correct? If the load is REALLY the same, sure. That's just never the case in real life for a number of reasons: - different load-balancing algorithms - different client behavior - session persistence - long-running TCP connections (websocket, et all) But yes, like I already mentioned, minconn/fullconn is addressing a very specific requirement that I don't think comes up very often. Lukas
Re: minconn, maxconn and fullconn (again, sigh!)
Hello Victor, On Wed, 10 Feb 2021 at 16:55, Victor Sudakov wrote: > > I can even phrase my question in simpler terms. What happens if the sum > total of all servers' maxconns in a backend is less than the maxconn > value in the frontend pointing to the said backend? Queueing for "timeout queue" amount of time, and then return 503 error (and this really is desirable as opposed to hitting maxconn on a frontend or even worse, global maxconn, because a few milliseconds of delay do not hurt and returning a proper HTTP error in distress is way better then some obscure connection timeout). See: timeout queue https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-timeout%20queue maxconn https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#5.2-maxconn I really suggest you ignore minconn and fullconn, and focus on maxconn instead. The latter is a must-have (and must-understand). Really maxconn (global, frontend and per server ) is the single most important performance knob in haproxy. Lukas
Re: TCP mode and ultra short lived connection
Hello, On Mon, 8 Feb 2021 at 18:14, Максим Куприянов wrote: > > Hi! > > I faced a problem dealing with l4 (tcp mode) haproxy-based proxy over > Graphite's component receiving metrics from clients and clients who are > connecting just to send one or two Graphite-metrics and disconnecting right > after. > > It looks like this > 1. Client connects to haproxy (SYN/SYN-ACK/ACK) > 2. Client sends one line of metric > 3. Haproxy acknowledges receiving this line (ACK to client) > 4. Client disconnects (FIN, FIN-ACK) > 5. Haproxy writes 1/-1/0/0 CC-termination state to log without even trying to > connect to a backend and send client's data to it. > 6. Metric is lost :( > > If the client is slow enough between steps 1 and 2 or it sends a bunch of > metrics so haproxy has time to connect to a backend – everything works like a > charm. The issue though is the client disconnect. If we delay the client disconnect, it could work. Try playing with tc by delaying the incoming FIN packets for a few hundred milliseconds (make sure you only apply this to this particular traffic, for example this particular destination port). > Example. First column is a time delta in seconds between packets It would be useful to have both front and backend tcp connections in the same output (and absolute time stamps - delta from the first packet, not the previous). You may also want to accelerate the server connect with options like: option tcp-smart-connect https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4-option%20tcp-smart-connect tfo (needs server support) https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#tfo%20%28Server%20and%20default-server%20options%29 > How can I deal with these send-and-forget clients? In TCP mode, we need to propagate the close from one side to the other, as we are not aware of the protocol. Not sure if it is possible (or a good idea) to keep sending buffer contents to the backend server when the client is already gone. "[no] option abortonclose" only affects HTTP, according to the docs. Maybe Willy can confirm/deny this. Lukas
Re: HAproxy soft reload timeout?
Hello Dominik, you are looking for hard-stop-after: http://cbonte.github.io/haproxy-dconv/2.2/configuration.html#hard-stop-after Regards, Lukas On Thu, 4 Feb 2021 at 11:40, Froehlich, Dominik wrote: > > Hi, > > > > I am currently experimenting with the HAproxy soft reload functionality (USR2 > to HAproxy master process). > > From what I understood, a new worker is started and takes over the listening > sockets while the established connections remain on the previous worker until > they are finished. > > The worker then terminates itself once all work is done. > > > > I’ve noticed there are some quite long-lived connections on my system (e.g. > websockets or tcp-keepalive connections from slow servers). So when I am > doing the reload, the previous instance > > of HAproxy basically lives as long as the last connection is still going. > Which potentially could go on forever. > > > > So when I keep reloading HAproxy because the config has changed, I could end > up with dozens, even hundreds of HAproxy instances running with old > connections. > > > > My question: Is there a way to tell the old haproxy instances how much time > they have to get done with their work and leave? > > I know it’s a tradeoff. I want my users to not be disrupted in their > connections but I also need to protect the ingress machines from overloading. > > > > Any best practices here? > > > > Cheers, > > D
Re: (possibly off topic) how to handle Chrome on SSL mass hosting ?
On Wed, 3 Feb 2021 at 18:47, Илья Шипицин wrote: >> while I do not mind to have such optimization, but when 'a.example.com" >> responds with http2 GOAWAY, that affects also "b.example.com" and " >> c.example.com". Chrome is not clever enough to open new connections instead >> of abandoned one. > > above approach works for Chrome (and does not work for Safari) > unfortunately we found Safari is using connection reuse just like Chrome, and > Safari does not handle 421 properly In which exact case is GOAWAY sent to the browser and how does that impact the browser behavior exactly? You are probably doing routing based on the host header, not the SNI value, so you wouldn't have a routing problem. I'm unsure what the actual problem is that you are trying to solve. Lukas
Re: SSL session resumption
Hello, On Wed, 3 Feb 2021 at 17:44, Илья Шипицин wrote: > > TLS1.2 uses tls tickets, when TLS1.0 uses ssl sessions. I believe this is incorrect, TLSv1.2 works just fine with Session ID's (RFC5246) and TLS 1.0 works fine with TLS tickets (RFC5077). I'm not aware of any restrictions between TLSv1.0/1.2 and session ID's vs TLS tickets. Lukas
Re: SSL session resumption
Hello Johan, we are gonna need the outputs of "haproxy -vv" from both situations, as well as at the very least *all* the ssl configuration parameters in haproxy that you are using. However, I do not believe it is likely that we can find the root cause, without access to those handshakes, since it cannot be reproduced by openssl s_client. What definitely changed in haproxy 2.2 is that the default minimum TLS version is now 1.2. To rollback to TLS 1.0 you can configure: global ssl-default-bind-options ssl-min-ver TLSv1.0 Regards, Lukas On Wed, 3 Feb 2021 at 13:36, Johan Andersson wrote: > > To whom it may concern > > > > We have recently upgraded out HAProxy version from 2.1.3 to 2.2.4. > > After the upgrade we got customer complaints that the data usage of their > devices had gone up. Our company sells proprietary hardware that logs data > and sends that to a web service which we host. These devices are often > deployed remotely and connected via shaky 3G connections with data-capped SIM > cards, so low data usage is very important. > > After some digging with Wireshark, we found that the SSL sessions are not > resumed. Instead a new handshake is initiated every time the device sends > data. Which is typically once an hour. > > We have set the global tune.ssl.lifetime parameter to 24h and the > tune.ssl.cachesize to 10 and this has worked since HAProxy version 1.6.9 > when we first introduced it. > > We have also tested with the latest 2.1.11 release of HAProxy and it behaves > the same way as the 2.1.3 version. We have also tested with 2.2.0 and 2.2.8 > and they behave the same as 2.2.4. > > > > We have tried reproducing this with openssl s_client, saving the session id > between requests but can’t reproduce it that way. > > We have also pored over the change logs between versions to see if there is > some change that could make HAProxy behave this way. > > > > We’re at a loss here, what could cause this behavior, and how can we fix it? > > > > > > Best regards > > > > Johan Andersson > > Development Engineer > > Global Platforms Cloud Team > > > > HMS Industrial Networks AB > > Stationsgatan 37, Box 4126 > > 300 04 Halmstad, Sweden > > > > Email: j...@hms-networks.com > > > > > > HALMSTAD | BARCELONA | BEIJING | BOSTON | BUCHEN | CHICAGO | COVENTRY | DEN > BOSCH | DUBAI | IGUALADA | > > KARLSRUHE | MILAN | MULHOUSE | NIVELLES | PUNE | RAVENSBURG | SEOUL | > SINGAPORE | TOKYO | WETZLAR > >
Re: How can I enable the HTTP/3 (QUIC) in HAProxy?
Jimmy, On Thu, 21 Jan 2021 at 09:45, Tim Düsterhus wrote: > > Hi List, > > Am 21.01.21 um 08:59 schrieb jimmy: > > I found the fact that HAProxy 2.3 higher supports HTTP/3 (QUIC) through > > [this > > link](https://www.haproxy.com/blog/announcing-haproxy-2-3/#connection-improvements). > This is a duplicate of this comment on the issue tracker: > > https://github.com/haproxy/haproxy/issues/680#issuecomment-764475902 You also cross-posted on discourse: https://discourse.haproxy.org/t/how-to-enable-http-3-quic-in-haproxy/6159 Understand that this is considered rude, try to avoid that please. Thanks, Lukas
Re: end all sessions for specific user
Hello, On Friday, 4 December 2020, Yossi Nachum wrote: > If I will change the map file via admin socket > Will it shutdown old/current sessions? Better, you don't need to shutdown anything, because HTTP authentication works on a HTTP transaction level, so each request is authenticated, even if it is an old session. You do need to modify the map file too though, otherwise you will reload back into the old configuration, next time you need to make a config change. Lukas
Re: end all sessions for specific user
Hello, On Thu, 3 Dec 2020 at 16:17, Yossi Nachum wrote: > > Hi, > I'm using haproxy 1.8 > This is my global and frontend configuration which include user auth: > [...] > acl network_allowed src,map_ip_int(/etc/haproxy/allowed_ips.lst,0) -m int > eq 1 > acl users_allowed hdr(MD5UP),map(/etc/haproxy/allowed_users.lst) -m found > http-request auth realm Bis if network_allowed BASIC_AUTH !users_allowed > http-request auth realm Bis if !users_allowed !network_allowed > http-request reject unless network_allowed || users_allowed I assume you are reloading haproxy to apply this change. This means that an older haproxy process will keep running with the old data. Some ideas: - restart instead of reloading, dropping all session immediately (but also killing in flight transactions) - configure hard-stop-after to an acceptable value for your, to limit the amount of time haproxy runs with old configurations - apply the changes to the map file via admin socket, instead of requiring a new haproxy process to spawn Haproxy can't know whether a session has an old password or not. This is handled at transaction level, not at session level. The only thing you can do is kill all sessions with an IP address that is not in network_allowed, manually. cheers, lukas
Re: end all sessions for specific user
Hello, On Thu, 3 Dec 2020 at 15:32, Yossi Nachum wrote: > > Hi, > > > > I have haproxy configuration that based on a file with username and password. > > When I disable a user his new sessions are blocked with 407 but his > old/current sessions are still processed Please share your configuration and haproxy release. I think you may be in tunnel mode, where haproxy does not have visibility to subsequent transactions. Lukas
Fwd: Forthcoming OpenSSL Release
FYI -- Forwarded message - From: Paul Nelson Date: Tue, 1 Dec 2020 at 11:15 Subject: Forthcoming OpenSSL Release To: The OpenSSL project team would like to announce the forthcoming release of OpenSSL version 1.1.1i. This release will be made available on Tuesday 8th December 2020 between 1300-1700 UTC. OpenSSL 1.1.i is a security-fix release. The highest severity issue fixed in this release is HIGH: https://www.openssl.org/policies/secpolicy.html#high Yours The OpenSSL Project Team
Re: Logging mTLS handshake errors
Hello Dominik, On Wed, 18 Nov 2020 at 15:06, Froehlich, Dominik wrote: > > Hi everyone, > > > > Some of our customers are using mTLS to authenticate clients. There have been > complaints that some certificates don’t work > > but we don’t know why. To shed some light on the matter, I’ve tried to add > more info to our log format regarding TLS validation: This is a know pain point: https://github.com/haproxy/haproxy/issues/693 Lukas
Re: Disable client keep-alive using ACL
Hi Tim, On Tue, 17 Nov 2020 at 13:35, Tim Düsterhus, WoltLab GmbH wrote: > > Hi > > Am 09.11.20 um 12:36 schrieb Tim Düsterhus, WoltLab GmbH: > > is it possible to reliably disable client keep-alive on demand based on > > the result of an ACL? > > > > I was successful for HTTP/1 requests by using: > > > > http-after-response set-header connection close if foo > > > > But apparently that has no effect for HTTP/2 requests. I was unable to > > find anything within the documentation with regard to this either. I don't think there is a way. In HTTP/2 you'd need to send a GOAWAY message to close the connection. There are no instructions in the HTTP headers regarding the connection. I *think/hope* we are actually sending GOAWAY messages when: - some timeouts are reached - hard-stop-after triggers - a "shutdown session ..." is triggered You could check if sending a "421 Misdirected Request" error to the client could achieve your goal, but it certainly behaves differently than a close in H1 (you can't get a successful answer to the client). It's also a workaround. Triggering GOAWAY/full H2 connection teardown dynamically would need to be implemented. I think in HTX all connection headers are immediately dropped (they are not "translated" and applied to the connection). cheers, lukas [1] https://tools.ietf.org/html/rfc7540#section-9.1.2
Re: do we want to keep CentOS 6 builds?
Hello Ilya, On Mon, 16 Nov 2020 at 22:48, Илья Шипицин wrote: > we run CI only for master branch. Exactly! > do all those people want to run latest unstable haproxy on oldish RHEL 6 ? No, but since we *only test* master, this is the only way we get *some* coverage for the changes we are backporting to stable branches. After all, a large percentage of them come from master. How do we know that a fix that we are backporting to 1.8 won't break the build on an older libc or gcc? There is a chance that this would have failed a test on master. This is *NOT* about CentOs6 specifically. This is about having at least one old linux system we are testing with, so we don't break things that *we don't want to break*. How sure are you that there are no supported OS's out there that still use gcc 4.4 or glibc 2.12, which we are testing here "for free" and before backporting the fix to 1.8? I am very sympathetic to drop support for old systems, *if the maintenance overhead becomes a burden* - and I don't set this bar high. My only point is that we should be discussing the problem we are trying to solve (effort that goes into supporting and testing an obsolete system?). I don't know how much hand holding the tests require - I can't quantify the effort that goes into this, which is why I would like this discussion to be about that as opposed to bikesheed around EOL's. So, is this about OpenSSL? Thanks, Lukas
Re: do we want to keep CentOS 6 builds?
Hello, On Sun, 15 Nov 2020 at 17:14, Илья Шипицин wrote: > > Hello, > > we still run cirrus-ci builds. > CentOS 6 is EOL. > > should we drop it? I think CentOs6 gives us good feedback about older operating systems that we may not necessarily want to break. The question for me is not so much whether we want to test with CentOs 6, it's more about do we want to be aware and fix build issues with those old systems? If the answer to the latter is yes, then we should keep the tests. If the answer is no, then we should drop them of course. How much of a burden is it to a) keep testing and b) keep supporting (by making sure it builds) on old kernels, libc and libraries? I don't have a strong opinion, but I would tend to keep the support (even though it's EOL). However if this is a continued pain in the ass, for example with openssl, then it's probably better to drop it. Lukas
Re: fronted/bind ordering
Hello, On Fri, 13 Nov 2020 at 21:21, Willy Tarreau wrote: > > > I'd suggest you run haproxy with noreuseport [1] temporarily, and > > > check if your kernel refuses to bind() to those IP's - it likely will. > > > This indicates an unsupported configuration (by your kernel, not by > > > haproxy). > > > > It indeed does fail. Hmmm, that's a shame as this was a really nice > > "feature" to have this fallback. I guess it's back to the drawing board > > unless you or anyone else have any other suggestions. > > OK so that confirms it. Well, you didn't lose anything it's just that > before the moment noreuseport was introduced, you simply did not benefit > from it. Note that for two sockets to support binding with SO_REUSEPORT, > *both* need to have it. So in your case if you value having it for whatever > reason, you may want to keep it enabled for one of the bind lines and > remove it on the other ones. I'm afraid noreuseport is a global option in haproxy. But the noreuseport was merely a suggestion to confirm that the kernel is not OK with this config. The change in behavior with conflicting sockets with SO_REUSEPORT enabled could be due to support for BPF (SO_ATTACH_REUSEPORT_CBPF) and eBPF (SO_ATTACH_REUSEPORT_EBPF) introduced in 4.5. Lukas
Re: fronted/bind ordering
Hello Bartosz, On Fri, 13 Nov 2020 at 10:08, Bartosz wrote: > > Are we really the only ones with this issue? Has no one else seen this > change in behaviour? Or otherwise have any idea where it's coming from? > > Or at least confirm whether they do or don't see the same behaviour. I don't think you can setup sockets this way, I think this is an incorrect configuration leading to undefined behavior. The fact the behavior then changes across different kernels is not a surprise. I'd suggest you run haproxy with noreuseport [1] temporarily, and check if your kernel refuses to bind() to those IP's - it likely will. This indicates an unsupported configuration (by your kernel, not by haproxy). Lukas [1] https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#noreuseport
Re: DNS Load balancing needs feedback and advice.
Hello Willy, On Fri, 6 Nov 2020 at 10:59, Willy Tarreau wrote: > > > hate the noise that some people regularly make about "UDP support" > > > > I am *way* more concerned about what to tell people when they report > > redundant production systems meltdowns because of the traps that we > > knew about for a long time and never improved. Like when the DNS > > response size surpasses accepted_payload_size and we don't have a TCP > > fallback, > > This one should be addressed once TCP support is implemented. But here > again, I'm not interested in implementing fallbacks. We're not supposed > to be dealing with unknown public servers when it comes to discovery > (which is the case where large responses are expected), so I'd rather > have resolvers configured for a simple resolving use case (e.g. server > address change, UDP) or discovery (TCP only). All DNS servers are supposed to support TCP, in my opinion using *only* a TCP DNS client would be fine (I'm not sure about using different code paths regarding address change and discovery). A part from the code changes though, this would be a big change that the user needs to know about. I'm not sure if a big item in the release notes is enough. I'm always concerned about changes that are not immediately obvious (because they don't cause configuration warnings or errors). > > or we don't force the users to specify the address-family > > for resolution, which is of course very wrong on a load-balancer. > > Are you suggesting that we should use IPv4 only unless IPv6 is set, and > that under no circumstance we switch from one to the other ? I remember > that this was a difficult choice initially because we were trying to deal > with servers migrating to another region and being available using a > different protocol (I'm not personally fan of mixing address families > for a same server in a load balancer, but I'm pretty certain we clearly > identified this need). But again while it's understandable for certain > (older?) cases, it's very likely that it makes absolutely no sense for > discovery. I'm suggesting: zero assumptions. Currently we have "resolve-prefer" to set a *prefered* address-family. I suggest keeping this keyword as is. However I also suggest that we introduce a keyword to restrict the resolver to a specific address-family. "resolve-family" [ipv4|ipv6] to only send either A or queries. But more importantly I suggest: reject a configuration that has neither "resolve-prefer", nor "resolve-family" (if implemented). This is a hard breaking change that can easily be fixed by adjusting the configuration. It could also be just a config warning for one major release and only become an error in the subsequent release (although I don't think that is needed, since the fix for the user is so easy). I'm confident that the only way to get out of this address-family mess is by stopping the assumptions altogether and forcing the user to make the decision. A load-balancer is not a browser, we must not do happy eyeballs and the current default behavior is pretty close to "undefined" (first valid response is accepted, on errors switching from one AF to another). Regards, Lukas
Re: DNS Load balancing needs feedback and advice.
Hello Willy, On Wed, 4 Nov 2020 at 15:36, Willy Tarreau wrote: > I think it's a reasonable tradeoff because those who insist on this are > also those who want to use so-called "modern" tools (placing "modern" > and DNS in the same sentence always leaves me a strange feeling that > something 37 years old is still modern). > > @Lukas, to respond to your concern, I don't like DNS either I don't think I got my point across. I never said I don't like DNS (the protocol). Let me be a little more blunt then: What I don't like are code/subsystems that are not sufficiently covered maintenance- and maintainer-wise (whatever the reason may be). In my opinion, the resolver code is like that today: - issues (including bugs) are open for years - it's riddled with traps for the users that will suddenly blow up in their faces (lack of TCP support, IPv4 vs IPv6) - important discussions have come to a halt It's obvious from the language in this thread (from Emeric and Willy), that YOU don't like DNS, and it's obvious from the condition of the existing dns subsystem that there is a complete lack of time for it as well. I'm not blaming Baptiste, I understand time is a rare resource, I'm just honestly describing the situation as I see it. I cannot help here (other than explaining why some current behaviours are bad and triaging the bugs on GH, which is also lacking: most dns issues do not even have the dns subsystem label). All this blunt critique without providing suggestions to improve the situation is rude, but since we are discussing DNS load-balancing (which sounds like adding new fuel to the fire to me), apparently with the same amount of resources and enthusiasm, I am concerned that we will end up in the same or worse situation, which is why I have to share my (negative) opinion about the current situation. > hate the noise that some people regularly make about "UDP support" I am *way* more concerned about what to tell people when they report redundant production systems meltdowns because of the traps that we knew about for a long time and never improved. Like when the DNS response size surpasses accepted_payload_size and we don't have a TCP fallback, or we don't force the users to specify the address-family for resolution, which is of course very wrong on a load-balancer. Of course I understand the DNS resolver code has nothing to do with future DNS load-balancing code. But the fact of the matter is that a new subsystems/featureset require sustained effort, time and frankly also interest. lukas
Re: DNS Load balancing needs feedback and advice.
Hello Emeric, On Mon, 2 Nov 2020 at 15:41, Emeric Brun wrote: > > Hi All, > > We are currently studying to develop a DNS messages load balancer (into > haproxy core) I find this a little surprising given that there already is a great DNS load-balancer out there (dnsdist) from the folks at powerdns and when I look at the status of the haproxy resolver, I don't feel like DNS sparkes a huge amount of developer interest. Loadbalancing DNS will certainly require more attention and enthusiasm than what the resolver code get's today, and even more important: long term maintenance. > After a global pass on RFCs (DNS, DNS over TCP, eDNS, DNSsec ...) we noticed > that practices on DNS have largely evolved > since stone age. > > Since the last brainstorm meeting I had with Baptiste Assmann and Willy > Tarreau, we were attempted to make some > assumptions and choices and we want to submit them to community to have your > thoughts. > > Reading RFCs, I notice multiple fallback cases (if server not support eEDNS > we should retry request without eDNS The edns fallback should be obsolete and has been disabled on the large public resolver on flagday 2019. https://dnsflagday.net/2019/ > or if response is truncated we should retry over TCP This is and always will be very necessary. Deploying the haproxy resolver feature would be a lot less dangerous if we would support this (or make all requests over TCP in the first place). > So we decide to make the assumption that nowadays, all modern DNS servers > support both TCP (and pipelined requests > as defined in rfc 7766) and eDNS. In this case the DNS loadbalancer will > forward messages received from clients in UDP > or TCP (supporting eDNS or not) to server via pipelined TCP conn. That's probably a good idea. You still have to handle all the UDP pain on the frontend though. > In addition, I had a more technical question: eDNS first purpose is clearly > to bypass the 512 bytes limitation of standard DNS over UDP, > but I did'nt find details about usage of eDNS over TCP which seems mandatory > if we want to perform DNSsec (since DNSsec > exloit some eDNS pseudo-header fields). The main question is how to handle > the payload size field of the eDNS pseudo header > if messages are exchanged over TCP. I'm not sure what the RFC specifically says, but I'd say don't send the "UDP payload size" field if the transport is TCP and ignore/filter it when received over TCP. not a dns expert here though, lukas
Re: IP binding and standby health-checks
Hello, On Tue, 20 Oct 2020 at 05:36, Dave Hall wrote: > HAProxy Active/Standby pair using keepalived and a virtual IP. > Load balance SSH connections to a group of user access systems (long-running > Layer 4 connections). > Using Fail2Ban to protect against password attacks, so using send-proxy-v2 > and go-mmproxy to present client IP to target servers. > > Our objective is to preserve connections through a fail-over. This is not possible today and I doubt it ever will. Haproxy is terminating the Layer 4 sessions on both ends and thus would have to migrate the sockets from one box to another. While linux does have "TCP connection repair" I'm not sure it's actually possible to do this in the load-balancer scenario, where the active box would just suddenly die (as opposed to a graceful and planned failover). You need to look at a solution that does not involve socket termination. Like IPVS Connection Synchronization for example. Or look at what the hyperscalers do nowadays. Google's Maglev, Github's glb-director or Facebook's katran probably can give some inspiration. Lukas
Re: Removal / obsolescence of keywords in 2.3 and future
Hello, On Wed, 14 Oct 2020 at 15:29, Willy Tarreau wrote: > For "nbproc", given that I had no response in the previous question and > I anticipate some surprises if I play games with it, I'll probably apply > William's suggestion, consisting in starting to emit a warning about it, > and asking users to either remove it, or explicitly mention "nbthread 1" > to get rid of the warning, and to report their use case. What about multi-threading performance across NUMA nodes? On discourse someone asked why nbproc and nbthread can't be combined: https://discourse.haproxy.org/t/cpu-affinity-nbproc-1-and-nbthread-1/5675 I'm not sure if this is a real use-case or simply a case of overengineering ... lukas
Re: source algorithm - question.
Hello, On Thu, 24 Sep 2020 at 11:40, Łukasz Tasz wrote: > > Hi all, > haproxy is gr8 - simply. > > Till today I was using roundobin algorithm, but after studying documentation > it popped up that source might be better. > I'm using haproxy in tcp mode, version 1.8, load from one client sometimes > requires more then few servers from the backend. > > but also initialization of handling requests takes some cost - I considered > picking a source as an algorithm - to avoid picking the next server according > to roundrobin. > But I realised that the behaviour of the queue has changed. With a source > algorithm for every source(host, client) there is a separate queue and one > server picked. > would it be possible that when one server reaches it's slots, the next one is > allocated (and next one, and next one)? where I can find detailed information > on how queues are managed depending on the algorithm which is used? It sounds like what you are looking for is "balance first". You can read more about this in the documentation about balance: https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4.2-balance Lukas
Re: [PATCH] BUILD: makefile: Update feature flags for Solaris / FreeBSD / NetBSD / OpenBSD
On Tue, 15 Sep 2020 at 09:05, Brad Smith wrote: > >> NetBSD 8.0 adds support for accept4() and closefrom(). Enable > >> getaddrinfo(). > > We just had to disable threading on OpenBSD 6.7 for the build to succeed: > > > > https://github.com/haproxy/haproxy/issues/725 > > > > Did you actually test all those combinations? Because otherwise it > > does not seem like a good idea to commit this. > > I know. I saw that. That was wrong. The other diff I submitted switching > the compiler default > as well as this came from that bogus diff. > > 2 of the 4 targets are (FreeBSD / OpenBSD). The other 2 are just based > on being aware of the API > and what has been implemented and when. I'll split up my diff and send > in the tested bits first. Ok thanks (I was concerned about disabling threading on OpenBSD as well). Please add the information that your previous patch switching gcc to cc is required for the OpenBSD change to work; it's obvious to me now, but when looking at the single change we are not always aware of the history (especially later during bisects). Thanks, Lukas
Re: [PATCH] BUILD: makefile: Update feature flags for Solaris / FreeBSD / NetBSD / OpenBSD
Hello Brad, On Sun, 13 Sep 2020 at 09:08, Brad Smith wrote: > > The following diff updates the feature flags for Solaris / FreeBSD / NetBSD / > OpenBSD. > > Bump the baseline Solaris to 9 which intruduced closefrom(). > > FreeBSD 10 is already EOL for support but its the new baseline. Introduces > accept4(). > Enable getaddrinfo(). The FreeBSD port enables these. > > OpenBSD 6.3 is pretty old but it brings a compiler that supports TLS for the > threads > support. 5.7 already supported closefrom(). Enable getaddrinfo(). > > NetBSD 8.0 adds support for accept4() and closefrom(). Enable getaddrinfo(). We just had to disable threading on OpenBSD 6.7 for the build to succeed: https://github.com/haproxy/haproxy/issues/725 Did you actually test all those combinations? Because otherwise it does not seem like a good idea to commit this. Lukas
Re: [RFC PATCH] MAJOR: ssl: Support for validating backend certificates with URI SANs (subjectAltName)
On Tue, 8 Sep 2020 at 12:39, Teo Klestrup Röijezon wrote: > > Hey Willy, sorry about the delay.. managed to get sick right after that stuff. > > > I don't understand what you mean here in that it does not make sense to > > you. Actually it's not even about overriding verifyhost, it's more that > > we match that the requested host (if any) is indeed supported by the > > presented certificate. The purpose is to make sure that the connection > > is not served by a server presenting a valid cert which doesn't match > > the authority we're asking for. And if we don't send any servername, > > then we can still enforce the check against a hard-coded servername > > presented in verifyhost. > > To my mind, `verifyhost` is more or less an acknowledgement that "no, this > isn't quite set up perfectly, but we can at least verify with some caveats". It's about proper certificate validation. > Otherwise, the host could just be taken from the address in the `connect` > keyword before SNI? No, because the address often is an actual IP address, not a hostname, and if it is a hostname, then it often does not match the actual certificate SAN. In a common setup we'd be serving a HTTPS site like www.example.org, so that is the certificate. However the backend servers that haproxy accesses is not www.example.org - because www.example.org would actually point to haproxy, not a backend server. Backend servers could be www1.example.org and www2.example.org, so there is a mismatch. So in the most common configuration, those hostnames do not match, which is why we need to specify it, with the intention to fully validate it. Lukas
[PATCH] DOC: overhauling github issue templates
as per the suggestions from Cyril and Willy on the mailing list: Message-ID: and with direct contributions from Tim, this changes large parts of the bug issue template. The Feature template is also updated as well as a new template for Code Reports is introduced. Co-authored-by: Tim Duesterhus --- .github/ISSUE_TEMPLATE/Bug.md | 67 --- .github/ISSUE_TEMPLATE/Code-Report.md | 55 .github/ISSUE_TEMPLATE/Feature.md | 30 3 files changed, 117 insertions(+), 35 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/Code-Report.md diff --git a/.github/ISSUE_TEMPLATE/Bug.md b/.github/ISSUE_TEMPLATE/Bug.md index 54f0475..038330a 100644 --- a/.github/ISSUE_TEMPLATE/Bug.md +++ b/.github/ISSUE_TEMPLATE/Bug.md @@ -25,27 +25,22 @@ Thanks for understanding, and for contributing to the project! --> -## Output of `haproxy -vv` and `uname -a` - +## Detailed description of the problem -``` -(paste your output here) -``` + -## What's the configuration? +## Expected behavior -``` -(paste your output here) -``` - ## Steps to reproduce the behavior -## Expected behavior +``` +(paste your output here) +``` + +## Output of `haproxy -vv` and `uname -a` + + + +``` +(paste your output here) +``` + +## If HAProxy crashed: Last outputs and backtraces -## Do you have any idea what may have caused this? +``` +(paste your output here) +``` -## Do you have an idea how to solve the issue? +## Additional information (if helpful) + diff --git a/.github/ISSUE_TEMPLATE/Code-Report.md b/.github/ISSUE_TEMPLATE/Code-Report.md new file mode 100644 index 000..d1bcd49 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/Code-Report.md @@ -0,0 +1,55 @@ +--- +name: Code Report +about: File a Code Report (for example from coverity or valgrind) +labels: 'type: code-report' +--- + + + +## Code Report + + + +Tool: (tool name goes here) + +``` +(paste your output here) +``` + + +## Output of `haproxy -vv` and `uname -a` + + + +``` +(paste your output here) +``` diff --git a/.github/ISSUE_TEMPLATE/Feature.md b/.github/ISSUE_TEMPLATE/Feature.md index 8f1a9df..eaaa7e5 100644 --- a/.github/ISSUE_TEMPLATE/Feature.md +++ b/.github/ISSUE_TEMPLATE/Feature.md @@ -25,27 +25,12 @@ Thanks for understanding, and for contributing to the project! --> -## Output of `haproxy -vv` and `uname -a` - - - -``` -(paste your output here) -``` - ## What should haproxy do differently? Which functionality do you think we should add? - ## What are you trying to do? +## Output of `haproxy -vv` and `uname -a` + + + +``` +(paste your output here) +``` -- 2.7.4
Re: github template
Hi, I prepared: - changes to Bug.md as per this discussion - changes to Features.md (just different sequence here) - added a new label "type: code-report" and a new issue template for those as well The changes can be seen here: https://github.com/lukastribus/hap-issue-trial/issues/new/choose If you agree, I will send the patches. Lukas
Re: Is the "source" keyword supported on FreeBSD?
On Wed, 12 Aug 2020 at 21:03, Jerome Magnin wrote: > > Hi Frank, > > On Wed, Aug 12, 2020 at 11:50:05AM +0200, Frank Wall wrote: > > Hi, > > > > this *feels* like a silly question and I may have missed something > > pretty obvious, but... I've tried to use the "source" keyword and > > it doesn't work. HAProxy does not use the specified IP address when > > connecting to the server. > > > > Is this keyword supposed to work on FreeBSD or are there any known caveats? > > Yes it is supposed to work on FreeBSD. The only caveat I know is that > you must use ipfw if you want to do "full transparent proxy" mode, which > mean using the client IP addresses to establish connections on the > backend side, because divert-reply is not available in FreeBSD's pf but > this is not what you are trying to do. Aren't those two different things? Just bind()ing to a source IP should not require any transport mode or ipfw settings. Of course, the IP needs to be actually configured on the system. Running haproxy through truss would show what happens to this bind() call. Lukas
Re: github template
Hello, On Mon, 20 Jul 2020 at 06:35, Willy Tarreau wrote: > > (Another case is when I try to follow the issue reports during vacation) > > > > I think it could be easier and quicker by only changing the sections order > > like this : > > 1. Expected behavior > > 2. Actual behavior > > 3. Steps to reproduce the behavior > > 4. Do you have any idea what may have caused this? > > 5. Do you have an idea how to solve the issue? > > 6. What's the configuration? > > 7. Output of haproxy -vv and uname -a > > > > What do you think about it ? > > Actually I'm wondering whether we should merge points 1 and 2 above into > "detailed description of the problem": sometimes it's easier to mention > "I'm seeing this which I find strange" without knowing exactly what the > expected behavior should be. We could indicate in the comments for this > section "please clearly describe the whole problem, preferably starting > with the expected behavior and followed by the actual behavior". > > And even then, I'm now thinking that most users would probably more > naturally first describe what they observed then explain what they > would have expected. This would flow more naturally: > >1) subject of the bug report Not sure whether you are just referring to the title of the issue (which actually is the subject already) or whether you are proposing to add a new section: I feel like that's redundant and would advise against it. >2) more detailed description matching the subject above: "when I > do this, haproxy does that" That's what "Actual behavior" is. Are you suggesting we rephrase? I agree it should be at the beginning, before "expected behavior". >3) expected behavior: explain why what's described above is > considered as wrong and what was expected instead (probably > a mismatch with the doc, can be skipped if it's about a crash) >4, 5, 6, 7) unchanged >8) haproxy's last output if it crashed, gdb output if a core was > produced >9) any additional info that may help (local patches, environment > specificities, unusual workload, observations, coincidences > with events on other components ...) Agreed. Features.md need's something similar (moving all the boring stuff below). Let me know about 1) and 2) above, and I will send a patch. Lukas
Re: Can I help with the 2.1 release?
Hello, On Thu, 30 Jul 2020 at 20:49, Valter Jansons wrote: > > On Thu, Jul 30, 2020 at 6:44 PM Harris Kaufmann > wrote: > > my company really needs the next 2.1 release but we want to avoid > > deploying a custom, self compiled version. > > > > Is there something I can do to help with the release? I guess there > > are no blocking issues left? > > For mission-critical setups you should be running the LTS release > lines. The 2.1 release line was more of a technical preview line for > the following 2.2 LTS release, to keep changes flowing, and you should > not expect regular new release tags on the 2.1 line considering the > 2.2 line has shipped. I am not involved in the release process but I > would assume the team will push a new 2.1 tag some day however I do > not see that being a high priority for them in any way. > > As a result, I would instead rephrase the question in the other > direction: Are there any blockers for you to upgrade to 2.2? I'm not sure I agree. I would be reluctant to suggest upgrading mission-critical setups to 2.2, it's not even a month old at this point. Unless you expect to run into bugs and have time and resources to troubleshoot it. 2.1 is not a technical preview, it's a proper release train with full support. Support for it will cease in 2021-Q1, but I don't think you can conclude that that means it's getting less love now. Lukas
Re: SLZ vs ZLIB
Hello, On Wed, 29 Jul 2020 at 19:19, Илья Шипицин wrote: > however, ZLIB is enabled by default in many distros and docker images. > any idea why ZLIB is chosen by default ? Because zlib is known, packaged and used everywhere and by everyone while slz is a niche library. It would need a marketing campaign I guess. lukas
Re: Several CVEs in Lua 5.4
Hello, On Wed, 29 Jul 2020 at 11:16, Froehlich, Dominik wrote: > > Hi Lukas, > > Thanks for the reply. > My query goes along the lines of which Lua version is compatible with HAproxy > and contains fixes to those CVEs. > I could not find a specific instruction as to which Lua version can be used > to build HAproxy / has been tested for production use. Currently LUA 5.3 is supported, patches will be committed soon for LUA 5.4 support: https://github.com/haproxy/haproxy/issues/730#issuecomment-664555213 But the way to fix this is not to rush to a new major LUA release, but to backport the fixes to LUA 5.3 instead. Lukas
Re: Several CVEs in Lua 5.4
Hello, On Wed, 29 Jul 2020 at 10:23, Froehlich, Dominik wrote: > > Hello everyone, > > Not sure if this is already addressed. Today I got a CVE report of several > issues with Lua 5.3.5 up to 5.4. > > I believe Lua 5.4 is currently recommended to build with HAproxy 2.x? > > Before I open an issue on github I would like to ask if these are already > known / addressed: I don't understand, specifically what are you asking us to do here? It's not like we ship LUA ... Lukas
Re: http-reuse and Proxy protocol
On Mon, 27 Jul 2020 at 13:14, Willy Tarreau wrote: > > However on a unix domain socket like this we never had this issue in > > the first place, as connection-reuse cannot be used on it by > > definition, correct? > > No, it doesn't change anything. We consider the connection, the protocol > family it uses is irrelevant. I don't know why, but I always wrongly assumed that a unix domain socket can only be datagram sockets, while really it's up to the application. And of course we use a stream sockets. Glad I could eliminate this wrong assumption :) Lukas
Re: http-reuse and Proxy protocol
Hello, On Thu, 23 Jul 2020 at 14:34, Willy Tarreau wrote: > > defaults > > http-reuse always > > > > backend abuse > > timeout server 60s > > balance roundrobin > > hash-balance-factor 0 > > server s_abuse u...@abuse.sock send-proxy-v2 maxconn 4 > > > > listen l_abuse > > bind u...@abuse.sock accept-proxy > > http-request set-var(req.delay) int(500) > > http-request lua.add_delay > > server 192.168.000.aaa:80 maxconn 1 > > server 192.168.000.bbb:80 maxconn 1 > > server z 192.168.000.ccc:80 maxconn 1 > > > > Is it OK ? Because i have no warning when verifying the configuration, or > > should i add a "http-reuse never" in "backend abuse" ? > > It is now properly dealt with, by marking the connection private, which > means it will not be shared at all. So what you'll see simply is that > there is no reuse for connections employing send-proxy. So your config > is safe, but you will just not benefit from the reuse. > > Anyway it's generally not a good idea to use proxy protocol over HTTP > from an HTTP-aware agent. Better use Forward/X-Forwarded-for that passes > the info per request and that nowadays everyone can consume. However on a unix domain socket like this we never had this issue in the first place, as connection-reuse cannot be used on it by definition, correct? Lukas
Re: github template
I will comment next week, but I generally agree that we should move the version output to the end, as I noticed the same issue. expected/actual behaviour sections are painful in the obvious cases (dont crash/crash), but oftentimes users just assume their itent is obvious when it's really not. lukas
[PATCH] MINOR: doc: ssl: req_ssl_sni needs implicit TLS
req_ssl_sni is not compatible with protocols negotiating TLS
explicitly, like SMTP on port 25 or 587 and IMAP on port 143.
Fix an example referring to 587 (SMTPS port with implicit TLS
is 465) and amend the req_ssl_sni documentation.
This doc fix should be backported to supported versions.
---
doc/configuration.txt | 23 +--
1 file changed, 13 insertions(+), 10 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index e2e9d88..93137ca 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -12259,15 +12259,16 @@ use-server unless
The "use-server" statement works both in HTTP and TCP mode. This makes it
suitable for use with content-based inspection. For instance, a server could
- be selected in a farm according to the TLS SNI field. And if these servers
- have their weight set to zero, they will not be used for other traffic.
+ be selected in a farm according to the TLS SNI field when using protocols
with
+ implicit TLS (also see "req_ssl_sni"). And if these servers have their weight
+ set to zero, they will not be used for other traffic.
Example :
# intercept incoming TLS requests based on the SNI field
use-server www if { req_ssl_sni -i www.example.com }
server www 192.168.0.1:443 weight 0
use-server mail if { req_ssl_sni -i mail.example.com }
- server mail 192.168.0.1:587 weight 0
+ server mail 192.168.0.1:465 weight 0
use-server imap if { req_ssl_sni -i imap.example.com }
server imap 192.168.0.1:993 weight 0
# all the rest is forwarded to this server
@@ -17670,13 +17671,15 @@ req_ssl_sni : string (deprecated)
contains data that parse as a complete SSL (v3 or superior) client hello
message. Note that this only applies to raw contents found in the request
buffer and not to contents deciphered via an SSL data layer, so this will not
- work with "bind" lines having the "ssl" option. SNI normally contains the
- name of the host the client tries to connect to (for recent browsers). SNI is
- useful for allowing or denying access to certain hosts when SSL/TLS is used
- by the client. This test was designed to be used with TCP request content
- inspection. If content switching is needed, it is recommended to first wait
- for a complete client hello (type 1), like in the example below. See also
- "ssl_fc_sni".
+ work with "bind" lines having the "ssl" option. This will only work for
actual
+ implicit TLS based protocols like HTTPS (443), IMAPS (993), SMTPS (465),
+ however it will not work for explicit TLS based protocols, like SMTP (25/587)
+ or IMAP (143). SNI normally contains the name of the host the client tries to
+ connect to (for recent browsers). SNI is useful for allowing or denying
access
+ to certain hosts when SSL/TLS is used by the client. This test was designed
to
+ be used with TCP request content inspection. If content switching is needed,
+ it is recommended to first wait for a complete client hello (type 1), like in
+ the example below. See also "ssl_fc_sni".
ACL derivatives :
req_ssl_sni : exact string match
--
2.7.4
Re: Documentation
Hello, On Sat, 11 Jul 2020 at 13:20, Jonathan Matthews wrote: > > On Sat, 11 Jul 2020 at 12:14, Tofflan wrote: >> >> Hello! >> >> Im trying to setup a setup HAProxy on my Pfsense router, the links under >> documentation dont work. example: >> https://cbonte.github.io/haproxy-dconv/2.3/intro.html and >> https://cbonte.github.io/haproxy-dconv/2.3/configuration.html >> Is there anyway to read or download them somewhere? > > > Hey there, > > I’m not sure if someone jumped the gun by updating the site’s doc links to > reference the unreleased 2.3 version, but you’ll have better luck changing > the “2.3” to either 2.2 or 2.0, depending on the version you’re trying to > install :-) Right, 2.3 is a development tree that you will not use in production anyway. Use the documentation that matches what you are actually using, and what you should be using is a stable release. If you are on a bleeding edge development tree, you should be looking at the files in the doc/ directory anyway, because that is what is as recent as the code itself. But yes, I'm sure Cyril will publish the 2.3-dev documentation shortly, then the links on haproxy.org will work. cheers, lukas
proposing a haproxy 2.0.16 release (was [BUG] haproxy retries dispatch to wrong server)
Hello,
On Fri, 10 Jul 2020 at 08:08, Christopher Faulet wrote:
> Hi,
>
> I finally pushed this fix in the 2.0. Note the same bug affected the HTTP
> proxy
> mode (using http_proxy option). In this case, the connection retries is now
> disabled (on the 2.0 only) because the destination address is definitely lost.
> It was the easiest way to work around the bug without backporting a bunch of
> sensitive patches from the 2.1.
Given the importance and impact of this bug you just fixed (at least 5
independent people already reported it on GH and ML) and the amount of
other important fixes already in the tree (including at least one
crash fix), I'm suggesting to release 2.0.16. Unless there are other
important open bugs with ongoing troubleshooting?
lukas@dev:~/haproxy-2.0$ git log --oneline v2.0.15..
d982a8e BUG/MEDIUM: stream-int: Disable connection retries on plain
HTTP proxy mode
e8d2423 BUG/MAJOR: stream: Mark the server address as unset on new
outgoing connection
b1e9407 MINOR: http: Add support for http 413 status
c3db7c1 BUG/MINOR: backend: Remove CO_FL_SESS_IDLE if a client remains
on the last server
0d881b2 BUG/MEDIUM: connection: Continue to recv data to a pipe when
the FD is not ready
847271d MINOR: connection: move the CO_FL_WAIT_ROOM cleanup to the reader only
39bb227 BUG/MEDIUM: mux-h1: Subscribe rather than waking up in h1_rcv_buf()
e0ca6ad BUG/MEDIUM: mux-h1: Disable splicing for the conn-stream if
read0 is received
0528ae2 BUG/MINOR: mux-h1: Disable splicing only if input data was processed
8e8168a BUG/MINOR: mux-h1: Don't read data from a pipe if the mux is
unable to receive
afadc9a BUG/MINOR: mux-h1: Fix the splicing in TUNNEL mode
8e4e357 BUG/MINOR: http_act: don't check capture id in backend (2)
c55e3e1 DOC: configuration: fix alphabetical ordering for
tune.pool-{high,low}-fd-ratio
a5e11c0 DOC: configuration: add missing index entries for
tune.pool-{low,high}-fd-ratio
ab06f88 BUG/MINOR: proxy: always initialize the trash in show servers state
ca212e5 BUG/MINOR: proxy: fix dump_server_state()'s misuse of the trash
135899e BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible
0b77c18 DOC: ssl: add "allow-0rtt" and "ciphersuites" in crt-list
4271c77 MINOR: cli: make "show sess" stop at the last known session
8ba978b BUG/MEDIUM: fetch: Fix hdr_ip misparsing IPv4 addresses due to
missing NUL
9bd736c REGTEST: ssl: add some ssl_c_* sample fetches test
15080cb REGTEST: ssl: tests the ssl_f_* sample fetches
d6cd2b3 MINOR: spoe: Don't systematically create new applets if
processing rate is low
1b4cc2e BUG/MINOR: http_ana: clarify connection pointer check on L7 retry
d995d5f BUG/MINOR: spoe: correction of setting bits for analyzer
26e1841 REGTEST: Add a simple script to tests errorfile directives in
proxy sections
8645299 BUG/MINOR: systemd: Wait for network to be online
b88a37c MEDIUM: map: make the "clear map" operation yield
c5034a3 REGTEST: http-rules: test spaces in ACLs with master CLI
4cdff8b REGTEST: http-rules: test spaces in ACLs
c3a2e35 BUG/MINOR: mworker/cli: fix semicolon escaping in master CLI
da9a2d1 BUG/MINOR: mworker/cli: fix the escaping in the master CLI
7ed43aa BUG/MINOR: cli: allow space escaping on the CLI
249346d BUG/MINOR: spoe: add missing key length check before checking key names
1b7f58f BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to compare
memory blocks
47a5600 BUG/MINOR: tcp-rules: tcp-response must check the buffer's fullness
9f3bda0 MINOR: http: Add 404 to http-request deny
c09f797 MINOR: http: Add 410 to http-request deny
lukas@dev:~/haproxy-2.0$
Thanks,
Lukas
Re: [BUG] haproxy retries dispatch to wrong server
Hello Michael, On Tue, 7 Jul 2020 at 15:16, Michael Wimmesberger wrote: > > Hi, > > I might have found a potentially critical bug in haproxy. It occurs when > haproxy is retrying to dispatch a request to a server. If haproxy fails > to dispatch a request to a server that is either up or has no health > checks enabled it dispatches the request to a random server on any > backend in any mode (tcp or http) as long as they are in the up state > (via tcp-connect or httpchk health checks). In addition haproxy logs the > correct server although it dispatches the request to a wrong server. > > I could not reproduce this issue on 2.0.14 or any 2.1.x version. This > happens in tcp and http mode and http requests might be dispatched to > tcp servers and vice versa. > > I have tried to narrow this problem down in source using git bisect, > which results in this commit marked as the first bad one: > 7b69c91e7d9ac6d7513002ecd3b06c1ac3cb8297. Makes sense that 2.1 is not affected because this commit was specifically written for 2.0 (it's not a backport). Exceptionally detailed and thorough reporting here, this will help a lot, thank you. A bug has been previously filed, but the details mentioned in this thread will help get things going: https://github.com/haproxy/haproxy/issues/717 Lukas
Re: [PATCH v2 0/2] Warnings for truncated lines
Hello, On Monday, 22 June 2020, Willy Tarreau wrote: > > > Configuration file is valid > > Looks good to me. > > > I guess a truncated last line cannot be differentiated from file that > > does not > > end with a new line, because fgets() consumes the full line (triggering > the > > eof), even if reading a NUL byte? > > Definitely! At least we're giving info with the warning and that's what > matters to me. > > Lukas is that also OK for you ? Yes, looks good to me. lukas
Re: [PATCH] BUG/MINOR: cfgparse: Support configurations without newline at EOF
On Mon, 22 Jun 2020 at 21:21, Willy Tarreau wrote:
>
> Hi guys,
>
> On Mon, Jun 22, 2020 at 07:49:34PM +0200, Lukas Tribus wrote:
> > Hello Tim,
> >
> > On Mon, 22 Jun 2020 at 18:56, Tim Düsterhus wrote:
> > >
> > > Lukas,
> > >
> > > Am 22.06.20 um 18:41 schrieb Lukas Tribus:
> > > > On Mon, 22 Jun 2020 at 18:16, Tim Duesterhus wrote:
> > > >>
> > > >> Fix parsing of configurations if the configuration file does not end
> > > >> with
> > > >> an LF.
> > > >
> > > > ... but it's also warning about it at the same time.
> > > >
> > > > So it's unclear to me:
> > > >
> > > > Do we support a configuration without trailing LF or not?
> > > >
> > > > If yes, there is no point in a warning (just as < 2.1).
> > > > If no, we should abort and error out.
> > > >
> > > > We can "just warn" if we plan to deprecate it in future release, and
> > > > at that point, explain that fact. But I find it strange to warn about
> > > > something, without a clear indication about *WHY* (unsupported,
> > > > deprecated, etc).
> > > >
> > > >
> > > > Thoughts?
> > > >
> > > I consider leaving out a trailing newline a bug for these reasons:
> > >
> > > [...]
> > > A non-terminated line thus is not a line and handling non-terminated
> > > lines is a bit wonky.
> >
> > What you are explaining is that the behavior is basically undefined,
> > so in my opinion we should just flat out reject this configuration.
> > s/ha_warning/ha_alert ?
> >
> > If we want to continue with a warning only (to not break older
> > configs), let's elaborate, because the warning just explains that the
> > line is not terminated, but not if and why that is actually a problem,
> > which I don't like to leave as an exercise for the reader (user).
> >
> > ha_warning("parsing [%s:%d]: line is not terminated by a newline (LF /
> > '\\n'), this may lead to undefined behavior.\n",
> >
> > ha_warning("parsing [%s:%d]: line should be terminated by a newline
> > (LF / '\\n'), otherwise undefined behavior may occur.\n",
> >
> > Something like that?
> >
> >
> > But really, I think we should just reject this instead (then the text
> > suffices, because it actually stops working).
>
> It used to work and certainly happens from time to time to people who
> generate their own configs. It's too easy to concatenate pieces of
> strings and not have the final condition ready to emit the last LF.
> For example if you emit you cookie options based on various tests,
> you may end up appending words without the trailing "\n" and decide
> to emit one at the beginning of each line instead. I've seen such
> configs from time to time so I know they do happen.
>
> I think I could be fine with dropping support for this (unless somebody
> objects here), but not in the LTS branch and not without a prior warning,
> so that users have time to fix their scripts.
>
> Also if we change this we have to update the doc to mention that all
> lines must be terminated.
>
> It's not uncommon to see files missing the last LF, and Git even has a
> warning for this. But the main issue to be reported there is that the
> file might have accidently been truncated (e.g. file-system full during
> the copy). However I agree with you Lukas that the warning should
> clearly indicate the impact and what needs to be fixed.
>
> But it gets tricky. There's a known flaw of fgets() making it return an
> incomplete line: if a \0 is present on the line before the \n. And given
> that fgets returns a pointer to a zero-terminated string instead of a
> length, the only way to read the string is to read it up to \0. So a
> line not ending with \n is not necessarily a truncated one but might be
> one with a \0 anywhere in the middle.
>
> So what we could do if we want to do something clean is the following:
> - detect \0 or truncation on a line and note its position ;
> - if we find another line once a truncated line has been found, we
> emit a warning about "stray NUL character at position X on line Y,
> this will not be supported in 2.3 anymore" and reset this position.
> - at the end of the loop, if the last NUL position is still set, we
> emit a warning about "missing LF on last line, file might have been
> truncated, this will not be supported in 2.3 anymore".
>
> And in 2.3 we turn that into errors.
>
> What do you guys think about it ?
I like it, yes.
Lukas
Re: [PATCH] BUG/MINOR: cfgparse: Support configurations without newline at EOF
Hello Tim,
On Mon, 22 Jun 2020 at 18:56, Tim Düsterhus wrote:
>
> Lukas,
>
> Am 22.06.20 um 18:41 schrieb Lukas Tribus:
> > On Mon, 22 Jun 2020 at 18:16, Tim Duesterhus wrote:
> >>
> >> Fix parsing of configurations if the configuration file does not end with
> >> an LF.
> >
> > ... but it's also warning about it at the same time.
> >
> > So it's unclear to me:
> >
> > Do we support a configuration without trailing LF or not?
> >
> > If yes, there is no point in a warning (just as < 2.1).
> > If no, we should abort and error out.
> >
> > We can "just warn" if we plan to deprecate it in future release, and
> > at that point, explain that fact. But I find it strange to warn about
> > something, without a clear indication about *WHY* (unsupported,
> > deprecated, etc).
> >
> >
> > Thoughts?
> >
> I consider leaving out a trailing newline a bug for these reasons:
>
> [...]
> A non-terminated line thus is not a line and handling non-terminated
> lines is a bit wonky.
What you are explaining is that the behavior is basically undefined,
so in my opinion we should just flat out reject this configuration.
s/ha_warning/ha_alert ?
If we want to continue with a warning only (to not break older
configs), let's elaborate, because the warning just explains that the
line is not terminated, but not if and why that is actually a problem,
which I don't like to leave as an exercise for the reader (user).
ha_warning("parsing [%s:%d]: line is not terminated by a newline (LF /
'\\n'), this may lead to undefined behavior.\n",
ha_warning("parsing [%s:%d]: line should be terminated by a newline
(LF / '\\n'), otherwise undefined behavior may occur.\n",
Something like that?
But really, I think we should just reject this instead (then the text
suffices, because it actually stops working).
--
lukas
Re: [PATCH] BUG/MINOR: cfgparse: Support configurations without newline at EOF
Hello, On Mon, 22 Jun 2020 at 18:16, Tim Duesterhus wrote: > > Fix parsing of configurations if the configuration file does not end with > an LF. ... but it's also warning about it at the same time. So it's unclear to me: Do we support a configuration without trailing LF or not? If yes, there is no point in a warning (just as < 2.1). If no, we should abort and error out. We can "just warn" if we plan to deprecate it in future release, and at that point, explain that fact. But I find it strange to warn about something, without a clear indication about *WHY* (unsupported, deprecated, etc). Thoughts? --lukas
Re: [PATCH] BUG/MINOR: systemd: Wait for network to be online
Hello Ryan, On Mon, 15 Jun 2020 at 20:01, Ryan O'Hara wrote: > > I posted this patch to start some discussion here. I'm not the first to notice > this problem but I was, until now, hesitant to change the systemd service > file until now. The reason for this was that waiting for network-online.target > could delay boot time. I agree with this change, I think the advantages outweigh the disadvantages. Acked-by: Lukas Tribus Lukas
Re: Ubuntu 20.04 + TLSv1
Hello Bjoern, On Fri, 12 Jun 2020 at 15:09, bjun...@gmail.com wrote: > > Hi, > > currently i'm testing Ubuntu 20.04 and HAProxy 2.0.14. > > I'm trying to get TLSv1 working (we need this for some legacy clients), so > far without success. > > I've read different things, on the one hand Ubuntu has removed TLSv1/TLSv1.1 > support completely, otherwise that it can be enabled: > http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.1.1f-1ubuntu2/changelog > > Is there anything that can be set in HAProxy? (apart from > "ssl-default-bind-options ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.2") > > Has anybody more information on this matter or has TLSv1 working in Ubuntu > 20.04 + HAProxy? Please try "force-tlsv10" *directly* on the bind line (not from ssl-default-bind-options). There are two issues: Bug 595 [1]: ssl-min-ver does not work from ssl-default-bind-options Bug 676 [2]: ssl-min-ver does not work properly depending on OS defaults If force-tlsv10 works directly on the bind line to enable TLSv1.0, then the next release 2.0.15 should work fine as it contains both fixes. Regards, Lukas [1] https://github.com/haproxy/haproxy/issues/595 [2] https://github.com/haproxy/haproxy/issues/676
Re: Fail to send unique-id by using proxy-v2-options
Hello, On Fri, 29 May 2020 at 04:39, lufeng0...@outlook.com wrote: > > Hi, > > > > I have compiled haproxy of version2.2-dev8 using Cygwin, in order to use it > as a load balancer in Windows 10. I want to send a unique ID generated using > the frontend's "unique-id-format" within the PROXYv2 header. However, it > reports an error: > > > > 0 [main] haproxy 1076 cygwin_exception::open_stackdumpfile: Dumping stack > trace to haproxy.exe.stackdump There are a least two related bugfixes in the tree after -dev8: https://github.com/haproxy/haproxy/commit/3ab504f5ff53968ae70d592cba4c1c7da6a0e7ff https://github.com/haproxy/haproxy/commit/68ad53cb3781010ccde7c781b6a3a1e926b5ed23 Can you try with a clone from the master branch directly or use today's snapshot: http://www.haproxy.org/download/2.2/src/snapshot/haproxy-ss-20200529.tar.gz Thanks, Lukas
Re: RFC: set minimum default TLS version to 1.2 for HAProxy 2.2
Hello, On Wed, 27 May 2020 at 13:33, Илья Шипицин wrote: > ср, 27 мая 2020 г. в 16:09, Tim Düsterhus : >> >> William, >> >> Am 27.05.20 um 12:40 schrieb William Lallemand: >> > Hello List, >> > >> > Since HAProxy 1.8, the minimum default TLS version for bind lines is >> > TLSv10. I was thinking to increase this minimum default to TLSv11 before >> > the 2.2 release. But when we discussed the other day about the DH >> > param set to 2048 by default, I read that RHEL 8 was also disabling >> > TLSv11 by default. TLSv12 now exists for 12 years, it is widely-spread >> > nowadays. >> > >> > So in my opinion we should do the same, and set the minimum version to >> > TLSv12 by default on bind lines. It's still configurable with >> > min-ssl-ver if you want the support for prior TLS versions. >> > >> > Does anybody have any objections? >> > >> >> As a data point: >> >> The OpenSSL shipped with Debian Buster does not support anything below >> TLS 1.2 by default [1]. The same is true starting with Ubuntu 20.04 LTS. > > > > I know several real-world cases when people had to build their own openssl on > Debian Buster in order get TLS1.0 back I'm certain that there is a ton of sites that still need TLSv1.0 going forward, however that doesn't mean we cannot change our *DEFAULTS* in a new *MAJOR* release, a default that is easily overwritten by a single configuration statement (and which is present in most configurations anyway). We are not talking about build options here. So in my opinion bumping the default minimum TLS version to 1.2 is a good thing and brings us inline with industry standard practices at this point. Therefor I don't have any objections. Lukas
Re: [tcp|http]-check expect status explained
On Wed, 6 May 2020 at 23:33, Aleksandar Lazic wrote: > > Hi. > > The doc for [tcp|http]-check expect have some *-status arguments like "L7OK", > "L7OKC","L6OK" and "L4OK" and so on. > > In the whole documentation are this states not explained. > I'm not sure in which chapter this states fit's, quick reminder HTTP,global, > logging, new chapter? > My suggestion is to add "1.2." HTTP in HAProxy and explain how the htx works > and what layer 4+6+7 means, opinions. It's not in the configuration documentation, it's in the management doc: https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.1 Lukas
Re: about Warning: Setting tune.ssl.default-dh-param to 1024
Hello, On Wed, 6 May 2020 at 20:25, William Lallemand wrote: > > As such I think it's about time we change the default value to 2048 and > > get rid of this annoying warning before 2.2 gets released (and at the > > same time 86% of the users will be able to remove one cryptic line in > > their config). This way those who don't know/need it will be more > > secure by default and those who need it will still be able to. > > > > Does anyone have any objection or alternate recommendation ? > > > > Thanks, > > Willy > > > I'm fine with that, most people use at least a value of 2048 because of > the warning, their modern distribution will probably deny a lower value, > and we add this warning a long time ago. I agree, we should default to 2048 and remove warnings. Lukas
Re: [PATCH] CI: special purpose build, testing compatibility against "no-deprecated" openssl
Hello Ilya , On Mon, 20 Apr 2020 at 16:12, Илья Шипицин wrote: >> I added weekly build for detection incompatibilities against "no-deprecated" >> openssl. >> >> (well, I first thought to add those option to travis, but it became >> over-engineered from my point of view) >> >> Lukas, if you have suggestions how to add to travis, I can try. I agree that it would be nice to have testing around the no-deprecated openssl option, it's just that I don't know anything about travis/github actions *at all*. I did see the make options in this patch and I don't see anything wrong with it. So from my point of view, you can proceed how you see fit. Thanks, Lukas
Re: HAproxy Error
On Fri, 17 Apr 2020 at 13:57, wrote:
> Even clean installation isn’t working because the default package available
> in RHEL from you is without openssl.
You are wrong.
1) we don't provide any packages. RHEL does.
2) a fresh RHEL 8.1 AMI on AWS works just fine and uses the provided
1.8.15 image with SSL support, as opposed to 2.1.2 which clearly you
manually compiled from source.
Here is the evidence from a freshly installed RHEL 8.1 AMI image on AWS:
[root@ip-172-31-42-121 ~]# uname -a
Linux ip-172-31-42-121.eu-central-1.compute.internal
4.18.0-147.el8.x86_64 #1 SMP Thu Sep 26 15:52:44 UTC 2019 x86_64
x86_64 x86_64 GNU/Linux
[root@ip-172-31-42-121 ~]# date
Fri Apr 17 12:22:38 UTC 2020
[ec2-user@ip-172-31-42-121 ~]$ haproxy -vv
-bash: haproxy: command not found
[root@ip-172-31-42-121 ~]# yum update
Red Hat Update Infrastructure 3 Client Configuration Server 8
7.6 kB/s | 2.1 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream from RHUI (RPMs)
11 kB/s | 2.8 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS from RHUI (RPMs)
10 kB/s | 2.4 kB 00:00
Dependencies resolved.
Nothing to do.
Complete!
[root@ip-172-31-42-121 ~]# yum install haproxy
Last metadata expiration check: 0:00:07 ago on Fri 17 Apr 2020 12:22:43 PM UTC.
Dependencies resolved.
=
Package
Architecture
Version
Repository
Size
=
Installing:
haproxy
x86_64
1.8.15-6.el8_1.1
rhel-8-appstream-rhui-rpms
1.3 M
Transaction Summary
=
Install 1 Package
Total download size: 1.3 M
Installed size: 4.4 M
Is this ok [y/N]: y
Downloading Packages:
haproxy-1.8.15-6.el8_1.1.x86_64.rpm
4.9 MB/s | 1.3 MB 00:00
-
Total
3.5 MB/s | 1.3 MB 00:00
Running transaction check
Waiting for process with pid 1068 to finish.
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing:
1/1
Running scriptlet: haproxy-1.8.15-6.el8_1.1.x86_64
1/1
Installing : haproxy-1.8.15-6.el8_1.1.x86_64
1/1
Running scriptlet: haproxy-1.8.15-6.el8_1.1.x86_64
1/1
Verifying: haproxy-1.8.15-6.el8_1.1.x86_64
1/1
Installed:
haproxy-1.8.15-6.el8_1.1.x86_64
Complete!
[root@ip-172-31-42-121 ~]# haproxy -vv
HA-Proxy version 1.8.15 2018/12/13
Copyright 2000-2018 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-format-truncation -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1
USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1
USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.1c FIPS 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c FIPS 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"),
Re: HAproxy Error
Hello, On Thu, 16 Apr 2020 at 13:51, wrote: > # which haproxy > /usr/ local/sbin/haproxy > > > > Attached output for command “haproxy –vv” > > > > Also I’m using a AWS RHEL 8.1 version AMI. > > Let us know what else is required. Also let me know how to enable Openssl. > Provide me the rpm link which has openssl enabled by default. As I expected, you are not using the RedHat provided package, but a local build of 2.1.2, without SSL support. I suggest you uninstall/remove your local build, and go back using the RedHat provided package. Lukas Lukas
Re: HAproxy Error
Hello, On Thu, 16 Apr 2020 at 06:04, wrote: > > Hi Team > > Let us know your availability to work on this. As Aleks already said: This haproxy executable has been build without OpenSSL support, which is required for your configuration. Provide the output of "which haproxy" and "haproxy -vv", I doubt you are actually running the Redhat package you indicated, more likely you built Haproxy manually from source on top of it. Lukas
Re: Disclaimer in emails (was: Re: HAproxy Error)
Hello Tim, Aleks, I fully agree with everything Tim just said. Let's keep the list about haproxy. Lukas
Re: List of ports opened for Listening by HAProxy
Hello, On Wed, 8 Apr 2020 at 13:59, kkazmierc...@wp.pl wrote: > > Hello, > We need to know which ports on the server need to be reopened in order to > appropriate work of HAProxy. Haproxy does not listen to any ports by default. It listens only to those ports that you configured haproxy to listen on. Some packages may provide a default configuration, with a bind statement. In that case, read the configuration, consult the package documentation or contact the maintainer. Lukas
Re: Any chance of PPA packages updates for that security fix?
Hello Sean, On Mon, 6 Apr 2020 at 18:12, Sean Reifschneider wrote: > > Been kind of watching for the haproxy versions to update in the PPAs for > Ubuntu. Considering the security nature of them, I'm kind of chomping at the > bit... :-) Any chance of those getting updated soonish? I can build my own > if there's a delay, but I should be able to build them here. If you are referring to Vincent Bernat's PPA, the patch was backported and the packages rebuilt on the same day. Lukas
Re: [PATCH] MINOR: config: make strict limits enabled by default
Hello, On Sat, 28 Mar 2020 at 19:19, William Dauchy wrote: > > as agreed a few months ago, enable strict-limits for v2.3 master is still for 2.2 which is in development. If you want to target v2.3, you have to wait until 2.2 is released. Lukas
Re: TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)
Hello, On Wed, 11 Mar 2020 at 08:32, Илья Шипицин wrote: >> On 09.03.20 20:37, Lukas Tribus wrote: >> >> I think the wording from the patch is still quite relaxed :). One of the >> >> best >> >> summaries describing the session ticket flaws, which I recommend is this: >> >> https://blog.filippo.io/we-need-to-talk-about-session-tickets/ >> > Nothing about this is a MITM attack. The point in the article is that >> > when the ticket-key is compromised, captured traffic can be passively >> > decrypted (which is what broken PFS means, as explained by the Apache >> > docs). >> >> take also this article, which clearly states that session tickets are >> vulnerable to replay attacks (which are a kind of MITM): >> https://eprint.iacr.org/2019/228.pdf > > > > major players of a big picture are 0RTT and session tickets. > indeed, if you wish to fight replay attack, you cannot use 0RTT (also, you > are supposed to maintain keys rotation). > > as for keys rotation, for unfamiliar people it is not clear why haproxy > itself does not provide such rotation. > at least, it should be better documented. Sure. But we are not gonna use the documentation to spread wrong information and FUD, based on partial, incorrect and out of context quotes. We already explain the forward secrecy issue with TLS ticket resumption, just as we explain replay-attacks for TLSv1.3 0RTT. If anyone thinks this still needs improvement, feel free to send RFC patches based on FACTUAL information. But let's stop making up things and then go on a fishing expeditions to justify it. As for automatic key rotation features, I'm not aware of anyone doing this by default, expect some niche projects (Caddy I believe does this). Not nginx, not Apache. These are features that someone has to actually develop. -lukas
Re: TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)
Hello, On Tue, 10 Mar 2020 at 07:36, Илья Шипицин wrote: >> > if you specify, your security team will tell you that "it is not secure". >> > if you do not specify, keys are generated on startup and it lead to huge >> > CPU spike on app reload (if you apply new config, app is reloaded and keys >> > are generated from scratch) >> >> You rotate ticket keys with "tls-ticket-keys" and reload or you can >> even add new ticket keys online via the "set ssl tls-key" unix socket >> command. But you have to take care of it if you want ticket key >> rotation. > > it is clear for educated people. > but documentation is written for non educated, right :) ? So you are saying the documentation needs further clarification? Lukas
Re: TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)
Hello, On Mon, 9 Mar 2020 at 20:39, Илья Шипицин wrote: >> I would disable session tickets by default in haproxy. Given that most >> clients support TLS 1.3 already this change would not even slow down many >> clients. > > > TLS tickets really require more love :) > > actually, there are two bad choices here > > 1) to specify TLS ticket key > 2) not to specify > > if you specify, your security team will tell you that "it is not secure". > if you do not specify, keys are generated on startup and it lead to huge CPU > spike on app reload (if you apply new config, app is reloaded and keys are > generated from scratch) You rotate ticket keys with "tls-ticket-keys" and reload or you can even add new ticket keys online via the "set ssl tls-key" unix socket command. But you have to take care of it if you want ticket key rotation. A performant, scalable and fully secure TLS setup is a non-trivial configuration to make. Probably not possible without compromises in a production environment with real world clients, but that is just how it is. Regarding security aspects the actual attack surface has to be considered. This attack requires the intruder to read the TLS ticket key from memory, this kills the concept of Forward Secrecy, but it does not kill anything else (that would not already be compromised anyway when an attacker can read from memory). Lukas
[PATCH] DOC: ssl: clarify security implications of TLS tickets
Clarifies security implications of TLS ticket usage when not
rotating TLS ticket keys, after commit 7b5e136458 ("DOC:
improve description of no-tls-tickets").
---
doc/configuration.txt | 17 +
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 0083549..33425a6 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -11677,10 +11677,9 @@ no-tls-tickets
extension) and force to use stateful session resumption. Stateless
session resumption is more expensive in CPU usage. This option is also
available on global statement "ssl-default-bind-options".
- The TLS ticket mechanism is only used up to TLS 1.2 and it is prone to
- man-in-the-middle attacks. You should consider to disable them for
- security reasons. TLS 1.3 implements more secure methods for session
- resumption.
+ The TLS ticket mechanism is only used up to TLS 1.2.
+ Forward Secrecy is compromised with TLS tickets, unless ticket keys
+ are periodically rotated (via reload or by using "tls-ticket-keys").
no-tlsv10
This setting is only available when support for OpenSSL was built in. It
@@ -12380,10 +12379,9 @@ no-tls-tickets
extension) and force to use stateful session resumption. Stateless
session resumption is more expensive in CPU usage for servers. This option
is also available on global statement "ssl-default-server-options".
- The TLS ticket mechanism is only used up to TLS 1.2 and it is prone to
- man-in-the-middle attacks. You should consider to disable them for
- security reasons. TLS 1.3 implements more secure methods for session
- resumption.
+ The TLS ticket mechanism is only used up to TLS 1.2.
+ Forward Secrecy is compromised with TLS tickets, unless ticket keys
+ are periodically rotated (via reload or by using "tls-ticket-keys").
See also "tls-tickets".
no-tlsv10
@@ -12813,6 +12811,9 @@ tls-tickets
This option may be used as "server" setting to reset any "no-tls-tickets"
setting which would have been inherited from "default-server" directive as
default value.
+ The TLS ticket mechanism is only used up to TLS 1.2.
+ Forward Secrecy is compromised with TLS tickets, unless ticket keys
+ are periodically rotated (via reload or by using "tls-ticket-keys").
It may also be used as "default-server" setting to reset any previous
"default-server" "no-tls-tickets" setting.
--
2.7.4
Re: TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)
On Mon, 9 Mar 2020 at 19:18, Björn Jacke wrote: > > On 2020-03-09 at 17:44 +0100 Lukas Tribus sent off: > > Perhaps we can relax the wording a bit here and describe the actual > > technical issue along with some recommendations. Apache for example > > documents [1]: > > I think the wording from the patch is still quite relaxed :). One of the best > summaries describing the session ticket flaws, which I recommend is this: > https://blog.filippo.io/we-need-to-talk-about-session-tickets/ Nothing about this is a MITM attack. The point in the article is that when the ticket-key is compromised, captured traffic can be passively decrypted (which is what broken PFS means, as explained by the Apache docs). I agree we should explain the operational issues, just as Apache does, but your patch does not do that. I will send a patch. > I would disable session tickets by default in haproxy. > Given that most clients support TLS 1.3 already this change > would not even slow down many clients. I disagree, the clients are not our only concern. Servers are. And most deployments on the server side are far away from TLS 1.3 support. Lukas
TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)
Hello, On Mon, 9 Mar 2020 at 11:23, PR Bot wrote: > > Dear list! > > Author: Björn Jacke > Number of patches: 2 > > This is an automated relay of the Github pull request: >Docs tls tickets > > Patch title(s): >BUG/MINOR: fix typo of tls-tickets >DOC: improve description of no-tls-tickets [...] > + The TLS ticket mechanism is only used up to TLS 1.2 and it is prone to > + man-in-the-middle attacks. You should consider to disable them for > + security reasons. TLS 1.3 implements more secure methods for session > + resumption. I know that this is already merged, but I disagree with the statements regarding TLS-tickets, the reality is a bit more nuanced. What is true is that *PFS* could be compromised if the TLS ticket key is not periodically rotated. That doesn't mean this configuration is "prone to MITM attacks", that the actual SSL session is compromised or that TLS tickets need to be disabled in every case. Perhaps we can relax the wording a bit here and describe the actual technical issue along with some recommendations. Apache for example documents [1]: > TLS session tickets are enabled by default. Using them without restarting the > web server with an appropriate frequency (e.g. daily) compromises perfect > forward secrecy. and for a static TLS ticket file: > The ticket key file contains sensitive keying material and should be > protected with file permissions similar to those used for > SSLCertificateKeyFile. Thanks, Lukas [1] https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslsessionticketkeyfile
Re: option forwardfor with IPv6
Hello, On Tue, 3 Mar 2020 at 19:06, Ionel GARDAIS wrote: > > Hi, > > What is the expected behavior of "option forwardfor" with an IPv6 connection ? > Frontend listen on IPv4 and IPv6. The expected behavior is to insert the IPv6 address into the X-F-F header, and this is exactly what happens in my repro here. > For IPv4 incoming connections, the server correctly displays the original IP > address, wether the haproxy-to-server is made with IPv4 or IPv6. > For IPv6 incoming connections, the server displays the IP of haproxy, wether > the haproxy-to-server is made with IPv4 or IPv6. Ok, but "server displays" is not equivalent with "haproxy sends in the X-Forwarded-For header". Does your server actually support IPv6 addresses in this header? If yes, what do you see in your logs/on your servers, when you make a call directly to it without haproxy in the question? curl -H "X-Forwarded-For: 2001:0db8:85a3:::8a2e:0370:7334" http://direct-backend-server.example.org/testurl Lukas
Re: Let's Encrypt ca-file for check-ssl on server line
Hello Aleks,
On Mon, 2 Mar 2020 at 22:21, Aleksandar Lazic wrote:
> check-ssl check-sni str("storage.sbg.cloud.ovh.net")
For the health check it's:
check-sni storage.sbg.cloud.ovh.net
(not a expression as per the doc: check-sni )
and for the traffic:
sni str(storage.sbg.cloud.ovh.net)
(as per the doc which says: sni )
You need both.
Lukas
[PATCH] BUG/MINOR: dns: ignore trailing dot
As per issue #435 a hostname with a trailing dot confuses our DNS code,
as for a zero length DNS label we emit a null-byte. This change makes us
ignore the zero length label instead.
Must be backported to 1.8.
---
As discussed in issue #435
---
src/dns.c | 6 ++
1 file changed, 6 insertions(+)
diff --git a/src/dns.c b/src/dns.c
index c131f08..e2fa387 100644
--- a/src/dns.c
+++ b/src/dns.c
@@ -1208,6 +1208,12 @@ int dns_str_to_dn_label(const char *str, int str_len,
char *dn, int dn_len)
if (i == offset)
return -1;
+ /* ignore trailing dot */
+ if (i + 2 == str_len) {
+ i++;
+ break;
+ }
+
dn[offset] = (i - offset);
offset = i+1;
continue;
--
2.7.4
Re: [PATCH v2] BUG/MINOR: connection: fix ip6 dst_port copy in make_proxy_line_v2
Hello, On Sun, 26 Jan 2020 at 20:11, William Dauchy wrote: > > > The explanation of the user-visible impact and the need for > > backporting to stable branches or not are MANDATORY. > > Yes; I was simply challenging that, as it is also open to mistakes to > write in commit message to which version it should be backported. My two cents: Just because commit 456 fixes something that has been introduced with commit 123 DOES NOT mean we backport 456 to all the branches that 123 was committed to - instead we backport 456 to a certain branch if it *actually* makes sense to do so. Here are some examples as to why we would not backport a specific fix: - the change may not be needed (no user impact) - the change may be to complex or unfeasible for the specific branch - there is a good chance that the change will have branch specific side-effects (for example in a subsystem that is significantly different or fragile in that branch) - the issue may be so minor that it doesn't make sense for the specific branch (think for example Critical fixes only branches) This is not something a VCS will figure out for us, but is a human decision. The author should be suggesting it's intention regarding the backport in the commit message, because it's the one person that spend the most time with the issue and probably has the best understanding of its impacts as well as the complexity and necessity of a backport. That does not mean the author needs to tests the fix with those branches, send branch-specific diff's, or do other research regarding the backport, but *the intention needs to be stated* (based on the research already conducted for the matter at hand). If the committer (or reviewer) disagrees, it will be clarified. If during the backport issues come up, those issues will be dealt with then (this happens latter, not when the fix is committed to master). Otherwise committers need to repeat the research the author already did, to figure out whether or not a backport is needed. Your patch here is a simple one-line change, and everything is probably immediately obvious (I did not read the code change myself), but that's not very common, often a lot of complexity is involved (even with one-line code changes) and committers or stable branch maintainers must not be spending their time researching issues while backporting (otherwise nothing will get backported). If you think commit 456 should be backported to all the branches that have 123, please research what branches have 123 and spell that out in the commit message, referring to that branch, like we usually do: [Must|Should] be backported to 1.8. Backports are always cherry-picked. Commits that are not cherry-picked have been committed to the particular branch when it was master (or a patch has been specifically written for a particular stable branch, but that's very rare). > it is also open to mistakes to write in commit message > to which version it should be backported. Of course it is open to mistakes, everything is. We try to minimize it, but we don't claim or expect perfection from anybody. Like I said, those are my two cents, Lukas
Re: Haproxy loadbalancing out going mail to Antispam servers
Hello, On Wed, 22 Jan 2020 at 16:18, Brent Clark wrote: > > Good day Guys > > We have a project where we are trying to load balance to our outbound > Spamexperts Antispam relays / servers. > > We hit a snag where our clients servers are getting 'Too many concurrent > SMTP connections from this IP address'. As a result the mail queue is > building up on the servers. > > After reverting our change, the problem went away. > > Our setup is: > (CLIENT SERVERS INDC) ---> 587 (HAPROXY) ---> (ANTISPAM) ---> (INTERNET) > > While I am performance tuning and repoking under the hood etc, could I > ask if someone could please peer review my config / setup. > > https://pastebin.com/raw/3D8frtzw > > If someone from the community can help, it would be appreciated. Your antispam relays see thousands of connections from a single IP address - the haproxy server, which is why it blocks the IP address of haproxy. You should implement the PROXY protocol to let your antispam relays know what the real client IP is, otherwise you will be in a world of hurt *especially* since AntiSPAM is so reliant on correct source IP addresses for IP reputation and blacklists, etc. Lukas
Re: SameSite attribute for persistent session cookie
Hello, On Tue, 21 Jan 2020 at 13:09, Tim Düsterhus wrote: > I don't need it myself, but I want to mention that it should be > backported, because the current situation can be "considered a bug" (a > feature now longer works due to changes in the ecosystem). I guess the > patch is fairly low risk, so backport until 1.8 or so? I agree we probably have to backport this (now) necessary *feature*, but I don't believe there is a need to masquerade it as a bug. Lukas
Re: [PATCH] CLEANUP: server: remove unused err section in server_finalize_init
Hello, On Thu, 9 Jan 2020 at 06:08, Илья Шипицин wrote: > > btw, if you add "Fixes: #438", the issue will be closed automatically > > https://help.github.com/en/github/managing-your-work-on-github/closing-issues-using-keywords which we are asking people *NOT* to do, because we are also tracking backports in the github issues. Issues will be closed manually on Github. Thanks, Lukas
Re: Re: Help, URL does not work with CHINESE charactor?
On Tue, 24 Dec 2019 at 11:46, JWD wrote: > > I have tried version 1.7,1.8,2.0,2.1, all the same. > > Config: > frontend www > acl acl-app hdr(host) -i sharepoint.domain.com > use_backend app if acl-app > backend > cookie HA-Server insert indirect nocache > server app 192.168.129.66:80 cookie app check inter 30s > > Log: > Dec 24 18:37:01 localhost haproxy[20108]: 192.168.134.81 - - > [24/Dec/2019:10:37:01 +] "" 400 0 "" "" 2423 066 "www" "www" > "" -1 -1 -1 -1 0 CR-- 2 2 0 0 0 0 0 "" "" "" "" > > # echo "show errors" | socat unix-connect:/etc/haproxy/hastats stdio > Total events captured on [24/Dec/2019:10:13:18.909] : 3 > > [24/Dec/2019:10:07:53.573] frontend www (#2): invalid request > backend (#-1), server (#-1), event #2 > src 192.168.134.81:3400, session #103, session flags 0x0080 > HTTP msg state MSG_RQURI(4), msg flags 0x, tx flags 0x > HTTP chunk len 0 bytes, HTTP body len 0 bytes > buffer flags 0x20808002, out 0 bytes, total 566 bytes > pending 566 bytes, wrapping at 16384, error at position 109: > > 0 GET > /CorWork/_layouts/15/TD.ECM.DoucmentDepartment/DepartmentFileDefau > 00070+ lt.aspx?destLink=/CorWork/ProjectShare/\xB8\xC4\xBD\xF8\xCF\xEE\xC4 > 00116+ \xBF/ECM\xD0\xC2\xB9\xA6\xC4\xDC HTTP/1.1\r\n Those are invalid requests, the URL must be encoded. Does IE really still sends this crap after all those years? You can try ignoring this with: option accept-invalid-http-request But it does not ignore everything. See: https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4.2-option%20accept-invalid-http-request Lukas
Re: PATCH: partially fix build if OpenSSL is built with no-deprecated option
Hello, > Guys, I must confess I'm completely lost in your discussions. I intend > to produce another round of 2.1 and 2.0 tomorrow as time permits, so if > you want me to get anything merged into it, please let me know. Lukas, > I'll count on you to summarize and suggest what's expected from me to > do at this point, I guess it will be easier :-) You can merge the patch I posted today, as there is consensus for this particular fix: https://www.mail-archive.com/haproxy@formilux.org/msg35760.html It should be backported to 2.0 (or even 1.9 - I forgot to mention that in the commit message). Further discussion is needed for some remaining points, which I doubt can be fully clarified for the release tomorrow. On Fri, 20 Dec 2019 at 19:53, Илья Шипицин wrote: >> On Wed, 27 Nov 2019 at 07:11, Илья Шипицин wrote: >> > >> > -#if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL) >> > +#if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL) || >> > defined(OPENSSL_NO_DEPRECATED) >> > [...] >> > -#if defined(USE_THREAD) && (HA_OPENSSL_VERSION_NUMBER < 0x1010L) >> > +#if defined(USE_THREAD) && (HA_OPENSSL_VERSION_NUMBER < 0x1010L) && >> > !defined(OPENSSL_NO_DEPRECATED) >> > [...] >> > -#if defined(USE_THREAD) && (HA_OPENSSL_VERSION_NUMBER < 0x1010L) >> > +#if defined(USE_THREAD) && (HA_OPENSSL_VERSION_NUMBER < 0x1010L) && >> > !defined(OPENSSL_NO_DEPRECATED) >> > [...] >> >> I'm confused. This is not required in my environment for the build to >> succeed and I don't see any reason why HA_OPENSSL_VERSION_NUMBER would >> be smaller here? Can you elaborate why the HA_OPENSSL_VERSION_NUMBER >> comparison would fail to do its job in those comparisons? > > > what is the lowest openssl we support ? > > those callbacks are required if threads are used for non-deprecated builds > and for early openssl versions like 1.0.0 Are you saying this is for openssl 1.0.2 with no-deprecated, is that what fails without this? I have troubles understanding what it is *exactly* that requires this change. If your question is, which openssl releases should we support with no-deprecated, I'd say 1.1.0 and newer. I don't believe there is any reason to support openssl 1.0.2 with no-deprecated. Sure we support old releases. But I would not waste any time/changes and especially complexity on adding support for "no-deprecated" feature for obsolete openssl releases (1.0.2 is EOL in a few days, nobody turns on no-deprecated on 1.0.2 now). What I am saying is that this change is not needed in my environment for the build to succeed, which is against openssl 1.1.1d with no-deprecated option. Maybe there is something that I don't see from my build machine here, that's why I need more information. If you see a build failure after the SSL_CTX_set_ecdh_auto() fix on top of current master, please elaborate how that build failure looks like and how openssl was compiled. Thanks, Lukas
[RFC PATCH] BUILD: ssl: improve SSL_CTX_set_ecdh_auto compatibility
SSL_CTX_set_ecdh_auto() is not defined when OpenSSL 1.1.1 is compiled
with the no-deprecated option. Remove existing, incomplete guards and
add a compatibility macro in openssl-compat.h, just as OpenSSL does:
https://github.com/openssl/openssl/blob/bf4006a6f9be691ba6eef0e8629e63369a033ccf/include/openssl/ssl.h#L1486
---
Please wait for Ilya's comments before committing this patch.
thanks,
-l
---
include/common/openssl-compat.h | 4
src/ssl_sock.c | 2 --
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/include/common/openssl-compat.h b/include/common/openssl-compat.h
index 31971bd..72b4e2f 100644
--- a/include/common/openssl-compat.h
+++ b/include/common/openssl-compat.h
@@ -374,5 +374,9 @@ static inline void EVP_PKEY_up_ref(EVP_PKEY *pkey)
#define BIO_meth_set_destroy(m, f) do { (m)->destroy = (f); } while (0)
#endif
+#ifndef SSL_CTX_set_ecdh_auto
+#define SSL_CTX_set_ecdh_auto(dummy, onoff) ((onoff) != 0)
+#endif
+
#endif /* USE_OPENSSL */
#endif /* _COMMON_OPENSSL_COMPAT_H */
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 00258b1..e4dd913 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -5178,9 +5178,7 @@ int ssl_sock_prepare_ctx(struct bind_conf *bind_conf,
struct ssl_bind_conf *ssl_
err && *err ? *err : "", curproxy->id,
conf_curves, bind_conf->arg, bind_conf->file, bind_conf->line);
cfgerr |= ERR_ALERT | ERR_FATAL;
}
-#if defined(SSL_CTX_set_ecdh_auto)
(void)SSL_CTX_set_ecdh_auto(ctx, 1);
-#endif
}
#endif
#if defined(SSL_CTX_set_tmp_ecdh) && !defined(OPENSSL_NO_ECDH)
--
2.7.4
