The patchset in comment #2 [Focal/FIPS][PATCH 0/4] Fix kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates)
is now superseded by patch: [Focal/FIPS][PATCH] UBUNTU: SAUCE: random: fallback to CRNG while DRBG is not yet initialized -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1981487 Title: kernel BUG/Oops errors from modprobe while the DRBG has not yet initialized (focal/fips-updates) Status in linux package in Ubuntu: Invalid Status in linux source package in Bionic: Invalid Status in linux source package in Focal: In Progress Status in linux source package in Jammy: Invalid Bug description: [Impact] * The Focal FIPS kernel in fips-updates hits kernel BUG/Oops errors during boot with the FIPS OpenSSL library installed (but those don't cause issues), when it runs modprobe with request_module() when looking up crypto algorithms/modules. * The modprobe command happens to call the OpenSSL library, and the FIPS version of OpenSSL calls getrandom(), and the FIPS kernel calls the DRBG for that, BUT it's _not yet_ initialized that early during boot when the kernel can run modprobe via request_module(). (e.g., IPv6 initialization time.) * The issue impacts the kernels in fips-updates only, per: "UBUNTU: SAUCE: random: Use Crypto API DRBG for urandom in FIPS mode" which exists in Focal, but not in Xenial/Bionic/Jammy. * The issue only happens with the crypto algorithms, even if they're built-in (i.e., modprobe is not needed). [Fix] * Fall back to CRNG while the DRBG is not yet initialized. (Marcelo Cerri confirmed it's OK per other discussions.) * The fix doesn't change the list and details of algorithms as in /proc/crypto (e.g., name, driver, module, priority) by the time the DRBG is initialized / initramfs started, so even though behavior changes, the net effect doesn't. * (Note: it's not possible to just use an initcall level earlier than rootfs_initcall() so modprobe isn't there, because fips_drbg_init() must run _after_ module_init() level so that crypto_rng_reset() works, even though its required module is built-in too. [Test Steps] * Install the kernel and openssl from fips-updates, boot with fips=1, check dmesg for BUG/Oops errors: $ sudo apt install linux-image-fips libssl1.1 # fips-updates $ sudo vim /etc/default/grub # append fips=1 boot option $ sudo update-grub && sudo reboot $ sudo dmesg | grep BUG: * Check/store the /proc/crypto file for comparisons. You can boot with break=top as well, to check that as early as possible, and copy into /run/initramfs/ then exit, to get it later in the rootfs. [Regression Potential] * The fix falls back to the regular CRNG for a while in early boot. The CRNG is used permanently in the non-FIPS kernels (and in FIPS kernels w/out fips=1), so the code path is exercised/tested frequently. * Regressions would most likely occur in calls to getrandom() before the DRBG is initialized, but that currently hits a BUG/Oops anyway. [Original Bug Description] $ sudo apt install --yes linux-image-fips # fips-updates $ sudo vim /etc/default/grub # fips=1 $ sudo update-grub && sudo reboot $ uname -r 5.4.0-1056-fips $ cat /proc/cmdline ... fips=1 No errors with the original/non-FIPS openssl, because it does NOT call getrandom(): $ dmesg | grep -c BUG: 0 $ dpkg -s libssl1.1 | grep ^Version: Version: 1.1.1f-1ubuntu2.15 $ strace -e getrandom modprobe --version kmod version 27 +ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL +++ exited with 0 +++ But if you install the FIPS openssl, it calls getrandom(), then BUG/Oops happen: $ sudo apt install libssl1.1 # updates initramfs $ dpkg -s libssl1.1 | grep ^Version: Version: 1.1.1f-1ubuntu2.fips.13.1 $ strace -e getrandom modprobe --version getrandom("\xc4\x84\x26\x25\x6f\xd4\xed\x38\xdf\xa9\x67\xee\x15\x1c\xe3\x98", 16, GRND_NONBLOCK) = 16 getrandom("\xa1\xd6\x67\x3e\xe4\x90\xb3\x8b\xdf\xe6\x34\x2a\xa7\x50\xbc\x2f", 16, GRND_NONBLOCK) = 16 getrandom("\xf1\x3e\xe4\x27\x9d\x47\x8c\x4b\x8a\x39\x8c\xe1\x2e\xee\xfa\x45", 16, GRND_NONBLOCK) = 16 getrandom("\xfb\x34\x18\x44\xd8\x23\x4c\x87\x13\x2e\x6b\x03\x79\xa7\x99\xf8", 16, 0) = 16 getrandom("\xdd\x83\xa7\x02\x10\x51\x2b\x4f\x21\x6b\xc1\xf1\x0d\xe7\x44\xb7", 16, 0) = 16 kmod version 27 +ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL +++ exited with 0 +++ $ sudo reboot $ dmesg | grep -c BUG: 22 $ dmesg ... [ 1.595759] NET: Registered protocol family 10 [ 1.600256] BUG: kernel NULL pointer dereference, address: 0000000000000038 ... [ 1.603829] CPU: 2 PID: 137 Comm: modprobe Not tainted 5.4.0-1056-fips #64-Ubuntu [ 1.603829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 [ 1.603829] RIP: 0010:urandom_read+0x268/0x480 ... [ 1.603829] Call Trace: [ 1.603829] __x64_sys_getrandom+0x7f/0x130 [ 1.603829] do_syscall_64+0x57/0x190 [ 1.603829] entry_SYSCALL_64_after_hwframe+0x44/0xa9 ... All BUG/Oops errors are the same: $ dmesg | grep BUG: | sed 's/^.*BUG:/BUG:/' | uniq -c 22 BUG: kernel NULL pointer dereference, address: 0000000000000038 And they stop after the DRBG is initialized: [ 3.651566] random: DRBG (drbg_nopr_ctr_aes256) initialized! [Fix Impact Analysis] The patch adds a dynamic debug message that can be enabled in the kernel cmdline, for comparisons (dmesg); we can also compare /proc/crypto for no changes. dyndbg="func urandom_read +p" Also add break=top, so we can copy /proc/crypto and dmesg right after DRBG is initialized (when initramfs is started). @ break=top time suffix=original # or modified cat /proc/crypto > /run/initramfs/proc-crypto.$suffix dmesg > /run/initramfs/dmesg.$suffix exit @ login time sudo -s cp /run/initramfs/{proc-crypto.*,dmesg.*} . reboot # next test There's no difference in the list/details of loaded crypto algorithms at all, with any combination: # md5sum proc-crypto.* 0b91bd619078fa342c6b4da039cb1582 proc-crypto.modified 0b91bd619078fa342c6b4da039cb1582 proc-crypto.original The kernel with the fix does not hit BUG/Oops errors: # grep ^ -m1 dmesg.* dmesg.modified:[ 0.000000] Linux version 5.4.0-1060-fips ... #68+fipsdrbgnullcheck ... dmesg.original:[ 0.000000] Linux version 5.4.0-1060-fips ... #68-Ubuntu ... # grep -c BUG: dmesg.* dmesg.modified:0 dmesg.original:22 # grep -c 'random: DRBG uninitialized! crng fallback' dmesg.* dmesg.modified:110 dmesg.original:0 The 110 number is 5 * 22 calls as modprobe calls getrandom() 5 times (see strace above). The original kernel has 1 BUG/Oops only because that kills the modprobe task. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981487/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
