Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Petr Štetiar [2019-03-15 13:28:39]: > I've found out, that it's all happening because of uclient-fetch being > blocked in getrandom syscall, leading for example to following: > > root@OpenWrt:~# time uclient-fetch > ^CCommand terminated by signal 2 > real 8m 31.08s Hi, one month has already passed, from my point of view it should be enough time to either get NAK or ACK. I find this part of system very sensitive, so I would really like to get at least one `Acked-by` before pushing it. Thanks! -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Hi Petr, Le ven. 15 mars 2019 à 13:01, Petr Štetiar a écrit : > > Etienne Champetier [2019-03-15 10:46:09]: > > Hi, > > > Just a side note, on first boot we save a random seed using getrandom() > > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/etc/init.d/urandom_seed > > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/sbin/urandom_seed > > > > And we restore it in preinit > > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/lib/preinit/81_urandom_seed > > Well, it seems like we're writing the /etc/urandom.seed to the /dev/urandom > which doesn't help with initialization of CRNG, just seeding the /dev/urandom. > > > So even if kernel PRNG is considered not initialized, in reality it > > is, so starting from second boot we are ~ok > > Nope, see bellow. > > 1st boot: > > [3.944674] urandom-seed: Seed file not found (/etc/urandom.seed) > [ 75.120166] random: fast init done > [ 140.917418] random: crng init done > > 2nd boot: > > [3.938414] urandom-seed: Seeding with /etc/urandom.seed > [ 22.440981] random: fast init done > [ 135.737309] random: crng init done I was not precise enough, writing to /dev/urandom do add entropy to the internal CSPRNG state, but the kernel count it as 0 because he doesn't trust it So after urandom-seed you are ~ok from a security stand point, but yes getrandom() still blocks > > -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Petr Štetiar [2019-03-15 15:09:23]: > I see it more as a problem of the implementation of getrandom syscall in > Linux kernel I've just found following interesting upstream commits in v4.18: commit 39a8883a2b989d1d21bd8dd99f5557f0c5e89694 Author: Theodore Ts'o Date: Tue Jul 17 18:24:27 2018 -0400 random: add a config option to trust the CPU's hwrng This gives the user building their own kernel (or a Linux distribution) the option of deciding whether or not to trust the CPU's hardware random number generator (e.g., RDRAND for x86 CPU's) as being correctly implemented and not having a back door introduced (perhaps courtesy of a Nation State's law enforcement or intelligence agencies). This will prevent getrandom(2) from blocking, if there is a willingness to trust the CPU manufacturer. commit 9b25436662d5fb4c66eb527ead53cab15f596ee0 Author: Kees Cook Date: Mon Aug 27 14:51:54 2018 -0700 random: make CPU trust a boot parameter Instead of forcing a distro or other system builder to choose at build time whether the CPU is trusted for CRNG seeding via CONFIG_RANDOM_TRUST_CPU, provide a boot-time parameter for end users to control the choice. The CONFIG will set the default state instead. So this actually might be a better direction for exploration. -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Etienne Champetier [2019-03-15 10:46:09]: Hi, > Just a side note, on first boot we save a random seed using getrandom() > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/etc/init.d/urandom_seed > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/sbin/urandom_seed > > And we restore it in preinit > https://github.com/openwrt/openwrt/blob/master/package/base-files/files/lib/preinit/81_urandom_seed Well, it seems like we're writing the /etc/urandom.seed to the /dev/urandom which doesn't help with initialization of CRNG, just seeding the /dev/urandom. > So even if kernel PRNG is considered not initialized, in reality it > is, so starting from second boot we are ~ok Nope, see bellow. 1st boot: [3.944674] urandom-seed: Seed file not found (/etc/urandom.seed) [ 75.120166] random: fast init done [ 140.917418] random: crng init done 2nd boot: [3.938414] urandom-seed: Seeding with /etc/urandom.seed [ 22.440981] random: fast init done [ 135.737309] random: crng init done -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Kristian Evensen [2019-03-15 13:57:41]: Hi, > I had a similar problem on some x86-devices. The problem is that > OpenWRT-devices are so "quiet" that it takes a while before a sufficient > amount of entropy is generated. I don't see it as problem of devices, I see it more as a problem of the implementation of getrandom syscall in Linux kernel, musl libc (blocking getentropy) and OpenSSL (blocking getrandom). I find it quite amusing, that following: $ cat /etc/rc.local cat /dev/random & getrandom 1 | hexdump > /tmp/getrandom.log exit 0 would never finish booting on my QEMU machine. > Instead of disabling the blocking getrandom()-call, what I did to "solve" > the issue was to install the haveged-packet on devices where I could not > find a driver for the hardware generator. Or we can switch to systemd :-) Putting jokes aside, I'm not sure if we want to add another dependency just because we've bumped OpenSSL. -- ynezz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Hi All, Le ven. 15 mars 2019 à 09:29, Petr Štetiar a écrit : > > While testing simple firmware image for x86/64 in QEMU I've discovered > some weird behavior today. This image contains simple package with > simple init script to bootstrap the device UCI configuration from > network server. This init script uses uclient-fetch and libustream-openssl. > > This image was booting fine until today, usually finished booting under > 10s, but today it was booting much slowly, boot times were in range from > 60s to a few minutes. I was also unable to power off the QEMU with > poweroff command. > > I've found out, that it's all happening because of uclient-fetch being > blocked in getrandom syscall, leading for example to following: > > root@OpenWrt:~# time uclient-fetch > ^CCommand terminated by signal 2 > real 8m 31.08s > > The problem passes away after `random: crng init done` hits > the system log, but this step can take ages in some cases (usually when there > are more processes calling getrandom in parallel), but I couldn't get it > under 60s on my QEMU machine. I've similar weird reports from users on > MIPS devices as well. > > [ 13.786576] random: fast init done > ... > [ 653.153740] random: crng init done > > I've bisected the problem down to the following commit (reverting it > fixed the problem): > > # first bad commit: [d872d00b2f] openssl: update to version 1.1.1a > > So this patch tries to fix this issue by making getrandom syscall > nonblocking, and also removes possible usage of getentropy libc call, > which in case of musl libc results again in use of getrandom syscall in > blocking mode. > > I've also added new config option just in case someone would prefer to > have probably safer but much slower boot times on some devices. Just a side note, on first boot we save a random seed using getrandom() https://github.com/openwrt/openwrt/blob/master/package/base-files/files/etc/init.d/urandom_seed https://github.com/openwrt/openwrt/blob/master/package/base-files/files/sbin/urandom_seed And we restore it in preinit https://github.com/openwrt/openwrt/blob/master/package/base-files/files/lib/preinit/81_urandom_seed So even if kernel PRNG is considered not initialized, in reality it is, so starting from second boot we are ~ok I'm not sure if we block on getrandom to generate ssh keys (and any other keys) on first boot though Regards Etienne > > Fixes: d872d00b2f ("openssl: update to version 1.1.1a") > Reviewed-by: Eneas U de Queiroz > Signed-off-by: Petr Štetiar > --- > package/libs/openssl/Config.in | 12 ++ > package/libs/openssl/Makefile | 7 +++- > .../openssl/patches/150-unblock-getrandom.patch| 45 > ++ > 3 files changed, 63 insertions(+), 1 deletion(-) > create mode 100644 package/libs/openssl/patches/150-unblock-getrandom.patch > > diff --git a/package/libs/openssl/Config.in b/package/libs/openssl/Config.in > index ecb9eea..0809afa 100644 > --- a/package/libs/openssl/Config.in > +++ b/package/libs/openssl/Config.in > @@ -70,6 +70,18 @@ config OPENSSL_WITH_ERROR_MESSAGES > This option aids debugging, but increases package size and > memory usage. > > +config OPENSSL_BLOCKING_GETRANDOM > + bool > + prompt "Enable back getrandom in blocking mode" > + help > + Enable back the default (upstream) blocking behavior. By > default, when > + reading from the random source, getrandom() blocks if no > random bytes are > + available, and when reading from the urandom source, it > blocks if the entropy > + pool has not yet been initialized. > + > + Please note, that turning this option on may affect the boot > time, which can > + in some cases take minutes. > + > comment "Protocol Support" > > config OPENSSL_WITH_TLS13 > diff --git a/package/libs/openssl/Makefile b/package/libs/openssl/Makefile > index 56e95af..6e7a603 100644 > --- a/package/libs/openssl/Makefile > +++ b/package/libs/openssl/Makefile > @@ -11,7 +11,7 @@ PKG_NAME:=openssl > PKG_BASE:=1.1.1 > PKG_BUGFIX:=b > PKG_VERSION:=$(PKG_BASE)$(PKG_BUGFIX) > -PKG_RELEASE:=3 > +PKG_RELEASE:=4 > PKG_USE_MIPS16:=0 > ENGINES_DIR=engines-1.1 > > @@ -30,6 +30,7 @@ PKG_LICENSE:=OpenSSL > PKG_LICENSE_FILES:=LICENSE > PKG_CPE_ID:=cpe:/a:openssl:openssl > PKG_CONFIG_DEPENDS:= \ > + CONFIG_OPENSSL_BLOCKING_GETRANDOM \ > CONFIG_OPENSSL_ENGINE \ > CONFIG_OPENSSL_ENGINE_BUILTIN \ > CONFIG_OPENSSL_ENGINE_BUILTIN_AFALG \ > @@ -327,6 +328,10 @@ ifdef CONFIG_i386 >endif > endif > > +ifdef CONFIG_OPENSSL_BLOCKING_GETRANDOM > + OPENSSL_OPTIONS += -DOPENSSL_BLOCKING_GETRANDOM > +endif > + > OPENSSL_TARGET:=linux-$(call qstrip,$(CONFIG_ARCH))-openwrt > > STAMP_CONFIGURED := $(STAMP_CONFIGURED)_$(shell echo $(OPENSSL_OPTIONS) | > mkhash md5) > diff --git
Re: [OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
Hi, On Fri, Mar 15, 2019 at 1:29 PM Petr Štetiar wrote: > > While testing simple firmware image for x86/64 in QEMU I've discovered > some weird behavior today. This image contains simple package with > simple init script to bootstrap the device UCI configuration from > network server. This init script uses uclient-fetch and libustream-openssl. > > This image was booting fine until today, usually finished booting under > 10s, but today it was booting much slowly, boot times were in range from > 60s to a few minutes. I was also unable to power off the QEMU with > poweroff command. > > I've found out, that it's all happening because of uclient-fetch being > blocked in getrandom syscall, leading for example to following: > > root@OpenWrt:~# time uclient-fetch > ^CCommand terminated by signal 2 > real 8m 31.08s > > The problem passes away after `random: crng init done` hits > the system log, but this step can take ages in some cases (usually when there > are more processes calling getrandom in parallel), but I couldn't get it > under 60s on my QEMU machine. I've similar weird reports from users on > MIPS devices as well. > > [ 13.786576] random: fast init done > ... > [ 653.153740] random: crng init done > > I've bisected the problem down to the following commit (reverting it > fixed the problem): > > # first bad commit: [d872d00b2f] openssl: update to version 1.1.1a > > So this patch tries to fix this issue by making getrandom syscall > nonblocking, and also removes possible usage of getentropy libc call, > which in case of musl libc results again in use of getrandom syscall in > blocking mode. > > I've also added new config option just in case someone would prefer to > have probably safer but much slower boot times on some devices. I had a similar problem on some x86-devices. The problem is that OpenWRT-devices are so "quiet" that it takes a while before a sufficient amount of entropy is generated. Instead of disabling the blocking getrandom()-call, what I did to "solve" the issue was to install the haveged-packet on devices where I could not find a driver for the hardware generator. haveged attempts to provide an unpredictable random number generator, and I was able to get the crng init down to a couple of sections. haveged apparently has/had some issues with VMs (https://wiki.archlinux.org/index.php/Haveged), but most of them seems to have been resolved. BR, Kristian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] [PATCH] openssl: Fix longer booting times by unblocking getrandom
While testing simple firmware image for x86/64 in QEMU I've discovered some weird behavior today. This image contains simple package with simple init script to bootstrap the device UCI configuration from network server. This init script uses uclient-fetch and libustream-openssl. This image was booting fine until today, usually finished booting under 10s, but today it was booting much slowly, boot times were in range from 60s to a few minutes. I was also unable to power off the QEMU with poweroff command. I've found out, that it's all happening because of uclient-fetch being blocked in getrandom syscall, leading for example to following: root@OpenWrt:~# time uclient-fetch ^CCommand terminated by signal 2 real 8m 31.08s The problem passes away after `random: crng init done` hits the system log, but this step can take ages in some cases (usually when there are more processes calling getrandom in parallel), but I couldn't get it under 60s on my QEMU machine. I've similar weird reports from users on MIPS devices as well. [ 13.786576] random: fast init done ... [ 653.153740] random: crng init done I've bisected the problem down to the following commit (reverting it fixed the problem): # first bad commit: [d872d00b2f] openssl: update to version 1.1.1a So this patch tries to fix this issue by making getrandom syscall nonblocking, and also removes possible usage of getentropy libc call, which in case of musl libc results again in use of getrandom syscall in blocking mode. I've also added new config option just in case someone would prefer to have probably safer but much slower boot times on some devices. Fixes: d872d00b2f ("openssl: update to version 1.1.1a") Reviewed-by: Eneas U de Queiroz Signed-off-by: Petr Štetiar --- package/libs/openssl/Config.in | 12 ++ package/libs/openssl/Makefile | 7 +++- .../openssl/patches/150-unblock-getrandom.patch| 45 ++ 3 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 package/libs/openssl/patches/150-unblock-getrandom.patch diff --git a/package/libs/openssl/Config.in b/package/libs/openssl/Config.in index ecb9eea..0809afa 100644 --- a/package/libs/openssl/Config.in +++ b/package/libs/openssl/Config.in @@ -70,6 +70,18 @@ config OPENSSL_WITH_ERROR_MESSAGES This option aids debugging, but increases package size and memory usage. +config OPENSSL_BLOCKING_GETRANDOM + bool + prompt "Enable back getrandom in blocking mode" + help + Enable back the default (upstream) blocking behavior. By default, when + reading from the random source, getrandom() blocks if no random bytes are + available, and when reading from the urandom source, it blocks if the entropy + pool has not yet been initialized. + + Please note, that turning this option on may affect the boot time, which can + in some cases take minutes. + comment "Protocol Support" config OPENSSL_WITH_TLS13 diff --git a/package/libs/openssl/Makefile b/package/libs/openssl/Makefile index 56e95af..6e7a603 100644 --- a/package/libs/openssl/Makefile +++ b/package/libs/openssl/Makefile @@ -11,7 +11,7 @@ PKG_NAME:=openssl PKG_BASE:=1.1.1 PKG_BUGFIX:=b PKG_VERSION:=$(PKG_BASE)$(PKG_BUGFIX) -PKG_RELEASE:=3 +PKG_RELEASE:=4 PKG_USE_MIPS16:=0 ENGINES_DIR=engines-1.1 @@ -30,6 +30,7 @@ PKG_LICENSE:=OpenSSL PKG_LICENSE_FILES:=LICENSE PKG_CPE_ID:=cpe:/a:openssl:openssl PKG_CONFIG_DEPENDS:= \ + CONFIG_OPENSSL_BLOCKING_GETRANDOM \ CONFIG_OPENSSL_ENGINE \ CONFIG_OPENSSL_ENGINE_BUILTIN \ CONFIG_OPENSSL_ENGINE_BUILTIN_AFALG \ @@ -327,6 +328,10 @@ ifdef CONFIG_i386 endif endif +ifdef CONFIG_OPENSSL_BLOCKING_GETRANDOM + OPENSSL_OPTIONS += -DOPENSSL_BLOCKING_GETRANDOM +endif + OPENSSL_TARGET:=linux-$(call qstrip,$(CONFIG_ARCH))-openwrt STAMP_CONFIGURED := $(STAMP_CONFIGURED)_$(shell echo $(OPENSSL_OPTIONS) | mkhash md5) diff --git a/package/libs/openssl/patches/150-unblock-getrandom.patch b/package/libs/openssl/patches/150-unblock-getrandom.patch new file mode 100644 index 000..f74abaa --- /dev/null +++ b/package/libs/openssl/patches/150-unblock-getrandom.patch @@ -0,0 +1,45 @@ +--- a/crypto/rand/rand_unix.c b/crypto/rand/rand_unix.c +@@ -20,6 +20,7 @@ + #include "internal/dso.h" + #if defined(__linux) + # include ++# include + #endif + #if defined(__FreeBSD__) + # include +@@ -292,7 +293,8 @@ static ssize_t syscall_random(void *buf, + */ + + /* +- * Do runtime detection to find getentropy(). ++ * Do runtime detection to find getentropy(). Please note, that at least ++ * on musl libc (version 1.2.21) getentropy() uses getrandom() in blocking mode. + * + * Known OSs that should support this: + * - Darwin since 16 (OSX 10.12, IOS 10.0). +@@ -301,6 +303,7 @@ static ssize_t syscall_random(void *buf, + *