Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
On 2024-02-02 16:38, Lukas Tribus wrote: WolfSSL support in HAProxy is experimental to the point that not only does it require compiling library and application from source, it also requires tinkering with LD paths to be able to even start the binary, so it's not like the INSTALL instructions are "ready to roll" in production. I'll fix this, I thought I put the same command line that I'm using in my script, but it lacks ADDLIB=-Wl,-rpath=/opt/wolfssl/lib/. -- William Lallemand
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
On Fri, 2 Feb 2024 at 08:43, Willy Tarreau wrote: > > Hi Lukas! > > On Thu, Feb 01, 2024 at 02:52:10PM +, Lukas Tribus wrote: > > On Thu, 1 Feb 2024 at 12:08, William Lallemand > > wrote: > > > > > > That's interesting, however I'm surprised the init does not work before > > > the chroot, > > > we are doing a RAND_bytes() with OpenSSL before the chroot to achieve > > > this. > > > > This approach can actually hide chroot issues leading to nasty > > operational issues like "Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops > > working after 1 hour" (see [1] and [2]). It's also not unrealistic to > > cause issue with process management, like FD leaks [3]. > > > > Stable OpenSSL on stable OS release branches today use getrandom() and > > not /dev/urandom. > > > > I think using the filesystems for CRNG is a footgun. At least let us > > fail fast and immediately if there is an issue with CRNG seeding from > > chroot. > > > > I consider getrandom() a modern and simple solution to all those problems. > > It's not that black and white actually. I pretty well remember that we > had to patch some code (I think it was openssl) in the past to force > to switch back from getrandom() to /dev/urandom, precisely because it > could block (particularly during early boot), while /dev/urandom would > never. It's a bit old, it was around kernel 4.4 I think, and since then > there has been endless discussions on the usual topics of "should a > program work but be less safe or should it fail to protect users against > themselves" and "how to generate a random MAC address on a headless > system when the only entropy source is the network", leading to a debate > around the addition of a GRND_INSECURE flag, but I don't know what the > status is nowadays, especially in LTS distros. I just found an abstract > of this thread here: > > https://lwn.net/Articles/800509/ > > It might be one reason why it's not enabled by default, though I can't > say, really. Right, to behave exactly like /dev/urandom we need GRND_INSECURE. GRND_INSECURE is in linux 5.6: https://github.com/torvalds/linux/commit/75551dbf112c992bc6c99a972990b3f272247e23 defined in glibc-2.32: https://github.com/bminor/glibc/commit/319d2a7b60cc0d06bb5c29684c23475d41a7f8b7 As such it is in RHEL 9, Ubuntu 22.04 LTS and Debian Bookworm (Stable). There was actually an attempt at making /dev/urandom secure/blocking in v5.18-rc1, but it didn't last for more than a few days: https://github.com/torvalds/linux/commit/6f98a4bfee72c22f50aedb39fb761567969865fe https://github.com/torvalds/linux/commit/0313bc278dac7cd9ce83a8d384581dc043156965 https://github.com/torvalds/linux/commit/48bff1053c172e6c7f340e506027d118147c8b7f > Maybe we should *recommend* enabling getrandom, explaining however that > its reliability may vary depending on the OS version and the amount of > entropy sources. For the record I think everything in INSTALL is a recommendation, without any one size fits all. WolfSSL support in HAProxy is experimental to the point that not only does it require compiling library and application from source, it also requires tinkering with LD paths to be able to even start the binary, so it's not like the INSTALL instructions are "ready to roll" in production. I will make a V2 of the patch indicating to either enable getrandom, mount /dev/[u]random paths into the chroot or disable chroot. cheers, Lukas
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
Hi Lukas! On Thu, Feb 01, 2024 at 02:52:10PM +, Lukas Tribus wrote: > On Thu, 1 Feb 2024 at 12:08, William Lallemand wrote: > > > > That's interesting, however I'm surprised the init does not work before the > > chroot, > > we are doing a RAND_bytes() with OpenSSL before the chroot to achieve this. > > This approach can actually hide chroot issues leading to nasty > operational issues like "Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops > working after 1 hour" (see [1] and [2]). It's also not unrealistic to > cause issue with process management, like FD leaks [3]. > > Stable OpenSSL on stable OS release branches today use getrandom() and > not /dev/urandom. > > I think using the filesystems for CRNG is a footgun. At least let us > fail fast and immediately if there is an issue with CRNG seeding from > chroot. > > I consider getrandom() a modern and simple solution to all those problems. It's not that black and white actually. I pretty well remember that we had to patch some code (I think it was openssl) in the past to force to switch back from getrandom() to /dev/urandom, precisely because it could block (particularly during early boot), while /dev/urandom would never. It's a bit old, it was around kernel 4.4 I think, and since then there has been endless discussions on the usual topics of "should a program work but be less safe or should it fail to protect users against themselves" and "how to generate a random MAC address on a headless system when the only entropy source is the network", leading to a debate around the addition of a GRND_INSECURE flag, but I don't know what the status is nowadays, especially in LTS distros. I just found an abstract of this thread here: https://lwn.net/Articles/800509/ It might be one reason why it's not enabled by default, though I can't say, really. Maybe we should *recommend* enabling getrandom, explaining however that its reliability may vary depending on the OS version and the amount of entropy sources. Willy
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
Hello William, On Thu, 1 Feb 2024 at 17:52, William Lallemand wrote: > > I consider getrandom() a modern and simple solution to all those problems. > > Unfortunately this is still a fallback solution if getrandom() is not > accessible or if the support is not built, as this is a fallback in > openssl too.I don't want HAProxy to require getrandom() to work, even if > this is not an ideal solution, there is no reason it shouldn't work > without it, at least for the sake of portability. Although freebsd has getrandom(), openbsd uses a different API and I'm sure there are lots of other operating systems that do not provide this API. So yes, a single code path would have been nice, but you are right, it isn't realistic. > > > I'll check if we can do something like this instead of needing a explicit > > > option, but > > > if that's not possible we will require GETRANDOM in the --enable-haproxy > > > build option. > > > > Actually I think wolfssl should add feature detection just like it > > does with other optional syscalls. But that is not what the suggested > > wolfssl 5.6.6 release does. > > It does not seem to be the wolfSSL philosophy :/ Everything needs to be > compiled manually and there is a lot of options, it's quite complicated > to obtain an optimized built... We recently saw that they've done > something like this for AES-NI, so maybe we could try to push them to a > more dynamic build system. Wolfssl sure can be hard to love sometimes. > I opened a wolfSSL issue https://github.com/wolfSSL/wolfssl/issues/7197 > feel free to participate, we could try to push them to the detection of > getrandom() during the build of their library, and fix their urandom > Implementation. Yes, thank you, I will subscribe and participate. > We could also try a call to RAND_bytes() after the chroot and exit with > an error saying that the library is not compatible with chroot. I definitively prefer failing fast/early, to avoid getting blindsided. Let's see what can be improved at wolfssl. Thank you, Lukas
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
On 2024-02-01 15:52, Lukas Tribus wrote: On Thu, 1 Feb 2024 at 12:08, William Lallemand wrote: > > That's interesting, however I'm surprised the init does not work before the chroot, > we are doing a RAND_bytes() with OpenSSL before the chroot to achieve this. This approach can actually hide chroot issues leading to nasty operational issues like "Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour" (see [1] and [2]). It's also not unrealistic to cause issue with process management, like FD leaks [3]. Sure, these were fixable bugs due to changes in HAProxy or in OpenSSL. But once a FD is kept and the library deinit correctly that shouldn't be a major problem.I agree we should be attentive to these points. Stable OpenSSL on stable OS release branches today use getrandom() and not /dev/urandom. I think using the filesystems for CRNG is a footgun. At least let us fail fast and immediately if there is an issue with CRNG seeding from chroot. I consider getrandom() a modern and simple solution to all those problems. Unfortunately this is still a fallback solution if getrandom() is not accessible or if the support is not built, as this is a fallback in openssl too.I don't want HAProxy to require getrandom() to work, even if this is not an ideal solution, there is no reason it shouldn't work without it, at least for the sake of portability. > I'll check if we can do something like this instead of needing a explicit option, but > if that's not possible we will require GETRANDOM in the --enable-haproxy build option. Actually I think wolfssl should add feature detection just like it does with other optional syscalls. But that is not what the suggested wolfssl 5.6.6 release does. It does not seem to be the wolfSSL philosophy :/ Everything needs to be compiled manually and there is a lot of options, it's quite complicated to obtain an optimized built... We recently saw that they've done something like this for AES-NI, so maybe we could try to push them to a more dynamic build system. I opened a wolfSSL issue https://github.com/wolfSSL/wolfssl/issues/7197 feel free to participate, we could try to push them to the detection of getrandom() during the build of their library, and fix their urandom Implementation. We could also try a call to RAND_bytes() after the chroot and exit with an error saying that the library is not compatible with chroot. Regards, William Lallemand
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
On Thu, 1 Feb 2024 at 12:08, William Lallemand wrote: > > That's interesting, however I'm surprised the init does not work before the > chroot, > we are doing a RAND_bytes() with OpenSSL before the chroot to achieve this. This approach can actually hide chroot issues leading to nasty operational issues like "Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour" (see [1] and [2]). It's also not unrealistic to cause issue with process management, like FD leaks [3]. Stable OpenSSL on stable OS release branches today use getrandom() and not /dev/urandom. I think using the filesystems for CRNG is a footgun. At least let us fail fast and immediately if there is an issue with CRNG seeding from chroot. I consider getrandom() a modern and simple solution to all those problems. > I'll check if we can do something like this instead of needing a explicit > option, but > if that's not possible we will require GETRANDOM in the --enable-haproxy > build option. Actually I think wolfssl should add feature detection just like it does with other optional syscalls. But that is not what the suggested wolfssl 5.6.6 release does. Regards, Lukas [1] https://www.mail-archive.com/haproxy@formilux.org/msg29592.html [2] https://github.com/openssl/openssl/issues/5330 [3] https://github.com/haproxy/haproxy/issues/314
Re: [PATCH] DOC: install: enable WOLFSSL_GETRANDOM
On 2024-01-30 20:45, Lukas Tribus wrote: Suggest enabling getrandom() syscall in wolfssl to avoid chroot problems when using wolfssl. --- Also see: https://discourse.haproxy.org/t/haproxy-no-responses-when-built-with-wolfssl-while-working-with-openssl/9320/15 --- INSTALL | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/INSTALL b/INSTALL index 18eb67f311..284b9825ba 100644 --- a/INSTALL +++ b/INSTALL @@ -285,7 +285,8 @@ least WolfSSL 5.6.6 is needed, but a development version might be needed for some of the features: $ cd ~/build/wolfssl - $ ./configure --enable-haproxy --enable-quic --prefix=/opt/wolfssl-5.6.6/ + $ ./configure --enable-haproxy --enable-quic \ + --prefix=/opt/wolfssl-5.6.6/ EXTRA_CFLAGS=-DWOLFSSL_GETRANDOM=1 $ make -j $(nproc) $ make install That's interesting, however I'm surprised the init does not work before the chroot, we are doing a RAND_bytes() with OpenSSL before the chroot to achieve this. I'll check if we can do something like this instead of needing a explicit option, but if that's not possible we will require GETRANDOM in the --enable-haproxy build option. -- William Lallemand
