The bug reporter has tested that the 6.8.0-62.65 kernel in -proposed
solves the problem[0]:
ubuntu@ip-172-31-0-29:~/firecracker$ uname -r
6.8.0-62-generic
ubuntu@ip-172-31-0-29:~/firecracker$ sudo apt list --installed | grep
linux-generic
linux-generic/noble-proposed,now 6.8.0-62.65 amd64 [installed]
ubuntu@ip-172-31-0-29:~/firecracker$ ./tools/devtool test -- -n16
integration_tests/functional/test_snapshot_basic.py::test_cycled_snapshot_restore
[Firecracker devtool 2025-05-21T17:40:40+00:00] Fetching CI artifacts from S3
[...]
=========================================== 84 passed in 133.64s (0:02:13)
===========================================
[Firecracker devtool 2025-05-21T17:43:17+00:00] Finished test run ...
[0]:
https://github.com/firecracker-microvm/firecracker/issues/5179#issuecomment-2898767675
** Tags removed: verification-needed-noble-linux
** Tags added: verification-done-noble-linux
** Bug watch added: github.com/firecracker-microvm/firecracker/issues #5179
https://github.com/firecracker-microvm/firecracker/issues/5179
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2109859
Title:
KVM bug causes Firecracker crash when it runs the vCPU for the first
time
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Noble:
Fix Committed
Bug description:
[Impact]
Firecracker process crashes with an "out of memory" error when it attempts to
run the vCPU for the first time, even if the system has enough available
memory:
```
2025-05-02T16:31:21.850912998 [daf77128-f177-4a01-9b97-a88dd9faa78f:fc_vcpu
0] Failure during vcpu run: Out of memory (os error 12)
```
The issue is triggered by a race condition caused by the VMM thread sending a
SIGRTMIN to the vCPU thread, while it is starting the
nx_huge_page_recovery_thread. This makes the thread creation fail, but due to
a
bug in the kernel, it is classified as a ENOMEM, instead of a ERESTARTNOINTR,
which should be retried.
This only affects 6.8 kernels, since the bug is introduced by the following
commits, backported to the noble:linux 6.8.0-58.60 kernel as part of the
upstream
stable updates (LP: #2101915):
- 43fb96ae7855 ("KVM: x86/mmu: Ensure NX huge page recovery thread is alive
before waking")
- 931656b9e2ff ("kvm: defer huge page recovery vhost task to later")
- d96c77bd4eeb ("KVM: x86: switch hugepage recovery thread to vhost_task")
[Fix]
Cherry-pick cb380909ae3b ("vhost: return task creation error instead of NULL")
and 916b7f42b3b3 ("kvm: retry nx_huge_page_recovery_thread creation").
[Test Case]
1) Launch a Noble c5.metal instance on AWS
2) Install and boot into the linux-generic 6.8 kernel
3) Install docker and aws-cli
4) git clone https://github.com/firecracker-microvm/firecracker.git
5) Go to the firecracker directory and run `./tools/devtool test -- -n16
integration_tests/functional/test_snapshot_basic.py::test_cycled_snapshot_restore`
6) With this patchset, observe that all tests pass. Without it, a couple
of tests will fail accusing out of memory.
[Where problems could occur]
This patchset touches vhost_task_create(), making it return specific error
pointers instead of just NULL. Problems could occur if its callers
mishandle the return value.
More broadly, it also touches code responsible for MM of KVM VMs, and issues
could appear as these VMs failing to initialize.
[Other info]
SF #00410184
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2109859/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
