Re: [PATCH] FEATURE: Adding MPTCP with option to disable it and fall-back to TCP
Hi Willy, On 30/05/2024 16:08, Willy Tarreau wrote: > Hi Matthieu, > > finally a bit more available again... > > On Fri, Apr 26, 2024 at 06:34:02PM +0200, Matthieu Baerts wrote: >>> I *am* interested in the feature, which has been >>> floating around for a few years already. However I tend to agree with >>> Nicolas that, at least for the principle of least surprise, I'd rather >>> not have this turned on by default. There might even be security >>> implications that some are not willing to take yet, and forcing them >>> to guess that they need to opt out of something is not great in general >>> (it might change in the future though, once everyone turns it on by >>> default, like we did for keep-alive, threads, and h2 for example). >> >> It makes sense as well! I initially suggested to Dorian to first send a >> patch where MPTCP is enabled by default because of the low impact (and >> also after having seen an old comment on that topic [1]). But indeed, >> doing that in steps sounds safer. >> >> [1] https://github.com/haproxy/haproxy/issues/1028#issuecomment-754560146 > > Yeah I understand, it would essentially depend on how all of this evolves > over time and how adoption grows for mptcp. Maybe one day most users will > expect it to work by default. I understand your position. >>> I'm also concerned by the change at the socket layer that will make all >>> new sockets cause two syscalls on systems where this is not enabled, >> >> I thought the listening sockets were created only once at startup and >> having two syscalls would have been fine. I guess it should be easy to >> remember the failure the first time, to avoid the extra syscalls for the >> next listening sockets. > > What you say is true for the listeners, but the socket() call is also > used when connecting to a backend server (though maybe I missed something > during the review). Oh sorry, I understood from Dorian that the modification he did was only impacting the listener sockets, it was not supposed to affect the others. Otherwise, we have the same behaviour as when we launch HAProxy with: mptcpize run haproxy > >>> and I'd really really prefer that we use the extended syntax for >>> addresses that offers a lot of flexibility. We can definitely have >>> "mptcp@address" and probably mptcp as a variant of tcp. >> >> >From what I understood, Dorian is now looking at that. It is not clear >> if he should create a new proto (struct protocol) or modifying it after >> having called protocol_lookup() in str2sa_range(). > > I *tend* to think that a new struct protocol is easier to add and > maintain. It could just share most (if not all) of the functions > with tcp, and probably be declared in the same file for ease of > sharing code. > > I'm seeing that in the comment you linked above I also proposed a > keyword for "bind" and "server" lines, like we have "tfo" to enable > TCP fast-open. It tends to be slightly easier to implement but then > requires more care everywhere because such options do no apply to > all protocols, so if you have "mptcp on" on a "bind quic4@:443" line, > that's quite confusing so it deserves extra checks to make sure it > is not silently ignored. Also I do have some doubts about how to > retrieve the source address, maybe we'll find that in fact it should > be seen as a new address family and not a new transport layer. But I > think not at this point. And BTW Björn had apparently implemented a > solution based on mptcp@ as well so it's likely workable. Yes, it looks better with a new 'struct protocol'. I worked on a first draft a few weeks ago: https://github.com/matttbe/haproxy/commit/f48f36191 As I mentioned in a previous email, I can wait for you to be back to send a new version on the mailing list. >> Can MPTCP be used as a variant of "rhttp" as well? > > I don't see how it should be related. The rhttp part is always tricky, > but the concept is that we accept a regular connection on a regular > frontend, then based on an explicit rule, return it and assign it as > an idle conn to a server. And for the server setting up pre-connected > idle conns to the gateway, it's just a regular address as well and the > protocol should not be involved there. Thank you, so "rhttp" is not a priority here for MPTCP. >>> Regarding how >>> we'd deal with the fallback, we'll see, but overall, thinking that >>> would explicitly configure mptcp and not even get an error if >>> it cannot bind is problematic to me, because among the most common >>> causes of tro
Re: [PATCH v2] FEATURE: add opt-in MPTCP support
Hi Willy, On 30/05/2024 15:48, Willy Tarreau wrote: > Hi Dorian, > > I'm now done with the release and having more time to read your > work. First, thanks for this update. I understand that you're almost > running out of time on this topic which must be completed before > June so I'm not going to make you waste your time. Some comments > below. Thank you for your reply! I hope you don't mind if I reply instead of Dorian, as he is no longer available to work on that. > On Thu, May 16, 2024 at 03:53:40PM +0200, Dorian Craps wrote: >> From: Dorian Craps >> >> Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension >> that enables a TCP connection to use different paths. >> >> Multipath TCP has been used for several use cases. On smartphones, MPTCP >> enables seamless handovers between cellular and Wi-Fi networks while >> preserving established connections. This use-case is what pushed Apple >> to use MPTCP since 2013 in multiple applications [2]. On dual-stack >> hosts, Multipath TCP enables the TCP connection to automatically use the >> best performing path, either IPv4 or IPv6. If one path fails, MPTCP >> automatically uses the other path. >> >> To benefit from MPTCP, both the client and the server have to support >> it. Multipath TCP is a backward-compatible TCP extension that is enabled >> by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...). >> Multipath TCP is included in the Linux kernel since version 5.6 [3]. To >> use it on Linux, an application must explicitly enable it when creating >> the socket. No need to change anything else in the application. >> >> This attached patch adds the "mptcp" global option in the config, which >> allows the creation of an MPTCP socket instead of TCP on Linux. If >> Multipath TCP is not supported on the system, an error will be reported, >> and the application will stop. >> >> A test has been added, it is a copy of "default_rules.vtc" in tcp-rules >> with the addition of "mptcp" in the config. I'm not sure what else needs >> to be tested for the moment, with this global MPTCP option. >> >> Note: another patch is coming to support enabling MPTCP per address, but >> I prefer to already send this patch, just in case, as I will soon have >> less time to dedicate to this. > > Thanks for this. As I previously mentioned, I'm not going to merge a > patch with a global option that does all or nothing, however having a > working patch like this can definitely serve as a proof-of-concept and > a reference when testing a more granular approach, and in this regard > your patch is much welcome! Thank you, that's clear. I started to look for a solution where MPTCP can be enabled per address [1]. I will try to find more time to drop the global option. [1] https://github.com/matttbe/haproxy/commit/f48f36191 >> Due to the limited impact within a data center environment, >> we have decided not to implement MPTCP between the proxy and the servers. >> The high-speed, low-latency nature of data center networks reduces >> the benefits of MPTCP, making the complexity of its implementation >> unnecessary in this context. > > I definitely understand. However let's keep in mind that whenever we > envision a use case, someone will come with a different one and suggest > that we should support it. Here I can imagine that some CDN operators > might be interested in using MPTCP to the origin server as well by > connecting via multiple paths. This opens a big can of worms with > questions about source address binding, interface binding etc, which > cannot be addressed easily as they'll all have impacts on the way > servers are configured. So I'm totally fine with having only a frontend > support in a first time. I just mentioned this to give you more context > and so that you know it's not only the DC that's on the backend, even > if it's by far the most common, and what some of the complex aspects > can be. Thank you for the explanation! The priority has been put on the frontend side mainly because it looked easier -- thanks to 'sock_prot' from the 'struct protocol' -- and it was addressing the main use-case. That was a good starting point, then the rest would have seen later if there was an interest (and time). > I'll be in vacation for two weeks at the end of this week, will you need > some review of a possible next patch, or did you give up due to some > difficulties you faced while exploring the per-listener configuration ? Dorian faced some difficulties, but he also had to stop to prepare his future exams. On my side, I quickly looked for a solution (see [1] above), but I was waiting for the v3.0.0 release before pushing anything. I can wait for you to be back before sending a version to the mailing list. > I'll give you a few comments on the patch below: Thank you, much appreciated! > >> diff --git a/src/cfgparse-global.c b/src/cfgparse-global.c >> index b173511c9..0feccd4b2 100644 >> --- a/src/cfgparse-global.c >> +++
Re: [PATCH] FEATURE: Adding MPTCP with option to disable it and fall-back to TCP
Hi Willy, Thank you for the feedback! On 25/04/2024 00:12, Willy Tarreau wrote: > Hi! > > On Wed, Apr 24, 2024 at 05:45:03PM +0200, Nicolas CARPi wrote: >> Hello, >> >> On 24 Apr, Dorian Craps wrote: >>> This attached patch uses MPTCP by default instead of TCP on Linux. >> The backward compatibility of MPTCP is indeed a good point toward >> enabling it by default. Nonetheless, I feel your message should include >> a discussion on the security implications of this change. >> >> As you know, v0 had security issues. v1 addresses them, and we can >> likely consider that new attacks targeting this protocol will pop up as >> it becomes widespread. >> >> In fact, that's already the case: >> >> See: CVE-2024-26708: mptcp: really cope with fastopen race >> or CVE-2024-26826: mptcp: fix data re-injection from stale subflow >> or CVE-2024-26782 kernel: mptcp: fix double-free on socket dismantle >> >> The three CVEs above are all from April 2024. >> >> Given that MPTCP v1 is relatively new (2020), and that we do not have >> real assurances that it is at least as secure as plain TCP, I would >> humbly suggest inverting the logic, and making it an opt-in option. >> >> This way, a vulnerability impacting MPTCP would only impact users that >> enabled it, instead of 100% of HAProxy users. In a few years, making it >> the default could be reconsidered. >> >> Please note that I'm simply voicing my concern as a user, and the core >> dev team might have a very different view about these aspects. >> >>> It sounds good to have MPTCP enabled by default >> Except when looking at it through the prism of the increased attack >> surface! ;) >> >>> IPPROTO_MPTCP is defined just in case old libC are being used and >>> don't have the ref. >> Shouldn't it be defined with a value, as per >> https://www.mptcp.dev/faq.html#why-is-ipproto_mptcp-not-defined ? >> (sorry if it's a dumb remark, I'm not a C dev) > > Without going into all the details and comments above, I just want to > say that I'll study this after 3.0 is released, since there's still a > massive amount of stuff to adjust for the release and we're way past > a feature freeze. It makes sense, take your time! > I *am* interested in the feature, which has been > floating around for a few years already. However I tend to agree with > Nicolas that, at least for the principle of least surprise, I'd rather > not have this turned on by default. There might even be security > implications that some are not willing to take yet, and forcing them > to guess that they need to opt out of something is not great in general > (it might change in the future though, once everyone turns it on by > default, like we did for keep-alive, threads, and h2 for example). It makes sense as well! I initially suggested to Dorian to first send a patch where MPTCP is enabled by default because of the low impact (and also after having seen an old comment on that topic [1]). But indeed, doing that in steps sounds safer. [1] https://github.com/haproxy/haproxy/issues/1028#issuecomment-754560146 > I'm also concerned by the change at the socket layer that will make all > new sockets cause two syscalls on systems where this is not enabled, I thought the listening sockets were created only once at startup and having two syscalls would have been fine. I guess it should be easy to remember the failure the first time, to avoid the extra syscalls for the next listening sockets. > and I'd really really prefer that we use the extended syntax for > addresses that offers a lot of flexibility. We can definitely have > "mptcp@address" and probably mptcp as a variant of tcp. >From what I understood, Dorian is now looking at that. It is not clear if he should create a new proto (struct protocol) or modifying it after having called protocol_lookup() in str2sa_range(). Can MPTCP be used as a variant of "rhttp" as well? > Regarding how > we'd deal with the fallback, we'll see, but overall, thinking that> someone > would explicitly configure mptcp and not even get an error if > it cannot bind is problematic to me, because among the most common > causes of trouble are things that everyone ignores (module not loaded, > missing secondary IP, firewall rules etc) that usually cause problems. > Those we can discover at boot time should definitely be reported. With the v1 Dorian suggested where MPTCP is enabled by default, a warning is reported if it is not possible to create an MPTCP socket, and a "plain" TCP one has been created instead. If MPTCP is explicitly asked, I guess it would make sense to fail if it is not possible to create an MPTCP socket, no? > But > let's discuss this in early June instead (I mean, feel free to discuss > the points together, but I'm not going to invest more time on this at > this moment). Thank you! Please note that Dorian is doing this as part of a work with his uni (useful contributions to Open-Source), and I think he would prefer sending the v2 before June, if that's OK. I
Re: [PATCH] FEATURE: Adding MPTCP with option to disable it and fall-back to TCP
Hello Nicolas, Thank you for the feedback! On 24/04/2024 17:45, Nicolas CARPi wrote: > Hello, > > On 24 Apr, Dorian Craps wrote: >> This attached patch uses MPTCP by default instead of TCP on Linux. > The backward compatibility of MPTCP is indeed a good point toward > enabling it by default. Nonetheless, I feel your message should include > a discussion on the security implications of this change. > > As you know, v0 had security issues. v1 addresses them, and we can > likely consider that new attacks targeting this protocol will pop up as > it becomes widespread. Indeed, any new features (or code in general) can bring security issues. Same for the protocol. Here it looks like MPTCPv1 addressed all reported issues [1]. > In fact, that's already the case: > > See: CVE-2024-26708: mptcp: really cope with fastopen race > or CVE-2024-26826: mptcp: fix data re-injection from stale subflow > or CVE-2024-26782 kernel: mptcp: fix double-free on socket dismantle > > The three CVEs above are all from April 2024. :-) There were no CVEs linked to MPTCP before Linux being a CNA earlier this year [2]. Now any fixes could automatically get a CVE even if there is no proven security issues [3]. For example, I don't think CVE-2024-26826 can cause security issues, and TCP also got some CVEs recently [4] :) But anyway, here the CVEs are linked to the implementation in the Linux kernel, not the protocol (MPTCPv1). > Given that MPTCP v1 is relatively new (2020), and that we do not have > real assurances that it is at least as secure as plain TCP, I would > humbly suggest inverting the logic, and making it an opt-in option. > > This way, a vulnerability impacting MPTCP would only impact users that > enabled it, instead of 100% of HAProxy users. In a few years, making it > the default could be reconsidered. I understand that it is safer, first, to have an option to enable MPTCP support, then enabling it by default later. But I admit that "In a few years" sounds like "a very long time"! MPTCP has been introduced in 2020, 4 years ago, after more than two year of development. So it had time to mature a bit in 6+ years :) Still under, TCP is being used, it is not a full network implementation. When MPTCP is enabled on the server side, it will only be used when clients request it, without impacting "plain" TCP connections. So enabling it will only impact connections where the client asked to use MPTCP. I agree that it increases the attack surface, but like any new features, no? :) > Please note that I'm simply voicing my concern as a user, and the core > dev team might have a very different view about these aspects. > >> It sounds good to have MPTCP enabled by default > Except when looking at it through the prism of the increased attack > surface! ;) > >> IPPROTO_MPTCP is defined just in case old libC are being used and >> don't have the ref. > Shouldn't it be defined with a value, as per > https://www.mptcp.dev/faq.html#why-is-ipproto_mptcp-not-defined ? > (sorry if it's a dumb remark, I'm not a C dev) IPPROTO_MPTCP is a constant, it is 262. It is just that at build time, some old libC might not have IPPROTO_MPTCP. To avoid compilation errors, IPPROTO_MPTCP is defined if it was not before. MPTCP support is checked at run time. [1] https://datatracker.ietf.org/doc/html/rfc8684#name-security-considerations [2] http://www.kroah.com/log/blog/2024/02/13/linux-is-a-cna/ [3] https://docs.kernel.org/process/cve.html [4] https://lore.kernel.org/linux-cve-announce/?q=s%3Atcp Cheers, Matt -- Sponsored by the NGI0 Core fund.
