For reference purposes, I'm attaching a dmesg log with debugging boot
options to:
- print initcalls entry/return
- print calls to __request_module() while the FIPS DRBG is uninitialized
Method / boot options:
fips=1
random.fips_urandom_drbg_crypto_noload=0
random.fips_urandom_drbg_modprobe_panic=0
"dyndbg=func __request_module +p"
initcall_debug
break=top
List of modules/aliases requested to be loaded (regardless of
CONFIG=y|m) while the FIPS DRBG is uninitialized, per conversation with
Marcelo Cerri.
$ cat dmesg.initramfs-break-top.log | grep '__request_module()' | grep -v --
'crypto-.*-all$' | cut -d: -f4- | sort | uniq -c
1 crypto-cbc(aes)
1 crypto-cryptd(__cbc-aes-aesni)
1 crypto-cryptd(__ctr-aes-aesni)
1 crypto-cryptd(__ecb-aes-aesni)
1 crypto-cryptd(__generic-gcm-aesni)
1 crypto-cryptd(__rfc4106-gcm-aesni)
1 crypto-cryptd(__xts-aes-aesni)
2 crypto-ctr(aes)
1 crypto-gcm(aes)
1 crypto-hmac(sha1)
2 crypto-hmac(sha256)
1 crypto-pkcs1pad(rsa,sha512)
This is the gist of it:
@ dmesg.initramfs-break-top.log
[ 0.000000] Command line: <...> fips=1
random.fips_urandom_drbg_crypto_noload=0
random.fips_urandom_drbg_modprobe_panic=0 "dyndbg=func __request_module +p"
initcall_debug break=top
...
[ 1.244744] calling populate_rootfs+0x0/0x110 @ 1
[ 1.245245] Trying to unpack rootfs image as initramfs...
[ 1.893124] Freeing initrd memory: 82992K
[ 1.893689] initcall populate_rootfs+0x0/0x110 returned 0 after 633268 usecs
...
[ 2.672049] calling inet6_init+0x0/0x39a @ 1
[ 2.673434] FIPS DRBG uninitialized: __request_module(): module:
crypto-hmac(sha1)
[ 2.677101] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.711252] FIPS DRBG uninitialized: __request_module(): module:
crypto-hmac(sha1)-all
[ 2.715082] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.756484] FIPS DRBG uninitialized: __request_module(): module:
crypto-hmac(sha256)
[ 2.759045] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.793932] FIPS DRBG uninitialized: __request_module(): module:
crypto-hmac(sha256)-all
[ 2.797236] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.833370] initcall inet6_init+0x0/0x39a returned 0 after 157136 usecs
...
[ 2.864104] calling aesni_init+0x0/0x135 @ 1
[ 2.866739] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__ecb-aes-aesni)
[ 2.870655] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.909789] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__ecb-aes-aesni)-all
[ 2.912948] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.950626] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__cbc-aes-aesni)
[ 2.954469] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2.989342] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__cbc-aes-aesni)-all
[ 2.992103] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.021153] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__ctr-aes-aesni)
[ 3.023818] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.051813] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__ctr-aes-aesni)-all
[ 3.054825] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.083289] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__xts-aes-aesni)
[ 3.085977] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.114266] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__xts-aes-aesni)-all
[ 3.117076] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.146238] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__rfc4106-gcm-aesni)
[ 3.149379] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.187352] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__rfc4106-gcm-aesni)-all
[ 3.190197] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.228841] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__generic-gcm-aesni)
[ 3.231672] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.267896] FIPS DRBG uninitialized: __request_module(): module:
crypto-cryptd(__generic-gcm-aesni)-all
[ 3.270738] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.308554] initcall aesni_init+0x0/0x135 returned 0 after 433683 usecs
...
[ 3.357564] calling big_key_init+0x0/0xc1 @ 1
[ 3.357922] FIPS DRBG uninitialized: __request_module(): module:
crypto-gcm(aes)
[ 3.360823] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.400423] FIPS DRBG uninitialized: __request_module(): module:
crypto-gcm(aes)-all
[ 3.403084] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.438526] FIPS DRBG uninitialized: __request_module(): module:
crypto-ctr(aes)
[ 3.441590] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.480282] FIPS DRBG uninitialized: __request_module(): module:
crypto-ctr(aes)-all
[ 3.483628] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.522754] initcall big_key_init+0x0/0xc1 returned 0 after 160969 usecs
[ 3.523397] calling init_trusted+0x0/0x148 @ 1
[ 3.523813] initcall init_trusted+0x0/0x148 returned 0 after 0 usecs
[ 3.524364] calling init_encrypted+0x0/0xda @ 1
[ 3.524809] FIPS DRBG uninitialized: __request_module(): module:
crypto-cbc(aes)
[ 3.527702] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.565467] FIPS DRBG uninitialized: __request_module(): module:
crypto-cbc(aes)-all
[ 3.568124] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 3.606040] initcall init_encrypted+0x0/0xda returned 0 after 79329 usecs
...
[ 3.663363] calling fips_drbg_init+0x0/0x105 @ 1
[ 3.669039] random: DRBG (drbg_nopr_ctr_aes256) initialized!
[ 3.670370] initcall fips_drbg_init+0x0/0x105 returned 0 after 6450 usecs
** Attachment added: "dmesg.initramfs-break-top.log"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981487/+attachment/5603167/+files/dmesg.initramfs-break-top.log
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1981487
Title:
kernel BUG/Oops errors from modprobe while the DRBG has not yet
initialized (focal/fips-updates)
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Bionic:
Invalid
Status in linux source package in Focal:
In Progress
Status in linux source package in Jammy:
Invalid
Bug description:
[Impact]
* The Focal FIPS kernel in fips-updates hits kernel BUG/Oops
errors during boot with the FIPS OpenSSL library installed
(but those don't cause issues), when it runs modprobe with
request_module() when looking up crypto algorithms/modules.
* The modprobe command happens to call the OpenSSL library,
and the FIPS version of OpenSSL calls getrandom(),
and the FIPS kernel calls the DRBG for that,
BUT it's _not yet_ initialized that early during boot
when the kernel can run modprobe via request_module().
(e.g., IPv6 initialization time.)
* The issue impacts the kernels in fips-updates only, per:
"UBUNTU: SAUCE: random: Use Crypto API DRBG for urandom in FIPS mode"
which exists in Focal, but not in Xenial/Bionic/Jammy.
[Fix]
* The issue only happens with the crypto algorithms, even
if they're built-in (i.e., modprobe is not needed), so
the fix approach is to set CRYPTO_NOLOAD while the DRBG
isn't yet initialized; this skips request_module() calls.
* The fix doesn't change the list and details of algorithms
as in /proc/crypto (e.g., name, driver, module, priority)
by the time the DRBG is initialized / initramfs started,
so even though behavior changes, the net effect doesn't.
* (Note: it's not possible to just use an initcall level
earlier than rootfs_initcall() so modprobe isn't there,
because fips_drbg_init() must run _after_ module_init()
level so that crypto_rng_reset() works, even though its
required module is built-in too.
[Test Steps]
* Install the kernel and openssl from fips-updates,
boot with fips=1, check dmesg for BUG/Oops errors:
$ sudo apt install linux-image-fips libssl1.1 # fips-updates
$ sudo vim /etc/default/grub # append fips=1 boot option
$ sudo update-grub && sudo reboot
$ sudo dmesg | grep BUG:
* Check/store the /proc/crypto file for comparisons.
You can boot with break=top as well, to check that
as early as possible, and copy into /run/initramfs/
then exit, to get it later in the rootfs.
[Regression Potential]
* The fix skips calls to request_module()/modprobe on
early boot until the FIPS DRBG is initialized (only
in the crypto subsystem) thus potential regressions
would show up as early boot kernel errors messages
or crypto algorithms not up after system booted.
(These should have no impact, theoretically, as it
currently doesn't run anyway, because of BUG/Oops.)
* The fix also adds a panic() call in request_module()
if other/non-crypto kernel code attempts to load a
module while the DRBG isn't ready yet (so we catch
that earlier). It's possible to use a boot option
to disable that behavior, but it disables FIPS mode
for strictness.
[Original Bug Description]
$ sudo apt install --yes linux-image-fips # fips-updates
$ sudo vim /etc/default/grub # fips=1
$ sudo update-grub && sudo reboot
$ uname -r
5.4.0-1056-fips
$ cat /proc/cmdline
... fips=1
No errors with the original/non-FIPS openssl, because it does NOT call
getrandom():
$ dmesg | grep -c BUG:
0
$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-1ubuntu2.15
$ strace -e getrandom modprobe --version
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++
But if you install the FIPS openssl, it calls getrandom(), then
BUG/Oops happen:
$ sudo apt install libssl1.1 # updates initramfs
$ dpkg -s libssl1.1 | grep ^Version:
Version: 1.1.1f-1ubuntu2.fips.13.1
$ strace -e getrandom modprobe --version
getrandom("\xc4\x84\x26\x25\x6f\xd4\xed\x38\xdf\xa9\x67\xee\x15\x1c\xe3\x98",
16, GRND_NONBLOCK) = 16
getrandom("\xa1\xd6\x67\x3e\xe4\x90\xb3\x8b\xdf\xe6\x34\x2a\xa7\x50\xbc\x2f",
16, GRND_NONBLOCK) = 16
getrandom("\xf1\x3e\xe4\x27\x9d\x47\x8c\x4b\x8a\x39\x8c\xe1\x2e\xee\xfa\x45",
16, GRND_NONBLOCK) = 16
getrandom("\xfb\x34\x18\x44\xd8\x23\x4c\x87\x13\x2e\x6b\x03\x79\xa7\x99\xf8",
16, 0) = 16
getrandom("\xdd\x83\xa7\x02\x10\x51\x2b\x4f\x21\x6b\xc1\xf1\x0d\xe7\x44\xb7",
16, 0) = 16
kmod version 27
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL
+++ exited with 0 +++
$ sudo reboot
$ dmesg | grep -c BUG:
22
$ dmesg
...
[ 1.595759] NET: Registered protocol family 10
[ 1.600256] BUG: kernel NULL pointer dereference, address: 0000000000000038
...
[ 1.603829] CPU: 2 PID: 137 Comm: modprobe Not tainted 5.4.0-1056-fips
#64-Ubuntu
[ 1.603829] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
[ 1.603829] RIP: 0010:urandom_read+0x268/0x480
...
[ 1.603829] Call Trace:
[ 1.603829] __x64_sys_getrandom+0x7f/0x130
[ 1.603829] do_syscall_64+0x57/0x190
[ 1.603829] entry_SYSCALL_64_after_hwframe+0x44/0xa9
...
All BUG/Oops errors are the same:
$ dmesg | grep BUG: | sed 's/^.*BUG:/BUG:/' | uniq -c
22 BUG: kernel NULL pointer dereference, address: 0000000000000038
And they stop after the DRBG is initialized:
[ 3.651566] random: DRBG (drbg_nopr_ctr_aes256) initialized!
[Fix Impact Analysis]
The patches add 3 module options to configure the behavior
of the components involved in the issue/fix. They can be
seen with dynamic debug in the kernel cmdline:
dyndbg="func crypto_alg_mod_lookup +p; func crypto_lookup_template +p;
func __request_module +p"
Also add break=top, so we can copy /proc/crypto and dmesg
right after DRBG is initialized (when initramfs is started).
Let's compare the 4 possible combination (first is 'no functional
change'):
random.fips_urandom_drbg_enable=0
random.fips_urandom_drbg_crypto_noload=0
random.fips_urandom_drbg_modprobe_panic=0
random.fips_urandom_drbg_enable=1
random.fips_urandom_drbg_crypto_noload=0
random.fips_urandom_drbg_modprobe_panic=0
random.fips_urandom_drbg_enable=1
random.fips_urandom_drbg_crypto_noload=1
random.fips_urandom_drbg_modprobe_panic=0
random.fips_urandom_drbg_enable=1
random.fips_urandom_drbg_crypto_noload=1
random.fips_urandom_drbg_modprobe_panic=1
@ break=top time
enable=$(cat /sys/module/random/parameters/fips_urandom_drbg_enable)
noload=$(cat /sys/module/random/parameters/fips_urandom_drbg_crypto_noload)
panic=$(cat /sys/module/random/parameters/fips_urandom_drbg_modprobe_panic)
suffix="enable${enable}_noload${noload}_panic${panic}"
cat /proc/crypto > /run/initramfs/proc-crypto.$suffix
dmesg > /run/initramfs/dmesg.$suffix
exit
@ login time
sudo -s
cp /run/initramfs/{proc-crypto.*,dmesg.*} .
reboot # next possible combination
There's no difference in the list/details of loaded crypto algorithms
at all, with any combination:
# md5sum proc-crypto.enable*_noload*_panic*
0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableN_noloadN_panicN
0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadN_panicN
0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadY_panicN
0b91bd619078fa342c6b4da039cb1582 proc-crypto.enableY_noloadY_panicY
The only option that hits BUG/Oops errors is when you disable the fix
(ie, don't set CRYPTO_NOLOAD):
# grep -c BUG: dmesg.enable*_noload*_panic*
dmesg.enableN_noloadN_panicN:0
dmesg.enableY_noloadN_panicN:22
dmesg.enableY_noloadY_panicN:0
dmesg.enableY_noloadY_panicY:0
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1981487/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
