[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
** Description changed: # Overview Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics. Launching the image with secure boot, but without confidential compute works as expected. # Expected result The system is able to reboot after the upgrade. # Actual result Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/ # Steps to reproduce Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM: $ gcloud beta compute instances create $USER-confidential-testing --zone =us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104 -hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE The last known good working image is daily- ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2 # Logs & notes * 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/ * 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/ * diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/ * serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/ # Cause: shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData. Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed Note that this affects all 5.11 kernels not just gcp. It is possible that gcp is the only cloud that uses sev currently (for "Confidential Computing"). # Fix: Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b + + # Test + + Without the fix applied, confirmed that I was able to reproduce the issue described here (complete failure to boot, kernel panic) + With fix, confirmed no issues booting + # Regression potential The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug: general protection fault, probably for non-canonical address 0x314836c31124d346: [#1] SMP NOPTI ** Summary changed: - Google Confidential Compute fails to boot with 1.47 + Google Confidential Compute fails to boot with shim version 1.47 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with shim version 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
** Description changed: # Overview Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics. Launching the image with secure boot, but without confidential compute works as expected. # Expected result The system is able to reboot after the upgrade. # Actual result Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/ # Steps to reproduce Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM: $ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily- ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE The last known good working image is daily- ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2 # Logs & notes * 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/ * 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/ * diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/ * serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/ + + + # Cause: + + shim changed the memory type for pages reserved for EFI runtime + services, from EfiRuntimeServicesData to EfiBootServicesData. + + Memory reserved for EFI runtime/boot services must be remapped as + encrypted in the kernel (during boot) if SEV (secure encrypted + virtualization) is enabled. The original kernel implementation of + ioremap only correctly mapped the region as encrypted for + EfiRuntimeServicesData regions, so when shim changed the type to + EfiBootServicesData the kernel bug was exposed + + # Fix: + + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b + + + # Regression potential + + The fix could potentially cause boot failures, if a memory region is + marked encrypted when it isn't. I assume it would cause a panic similar + to the one seen here for this bug: + + general protection fault, probably for non-canonical address + 0x314836c31124d346: [#1] SMP NOPTI ** Description changed: # Overview Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics. Launching the image with secure boot, but without confidential compute works as expected. # Expected result The system is able to reboot after the upgrade. # Actual result Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/ # Steps to reproduce Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM: $ gcloud beta compute instances create $USER-confidential-testing --zone =us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104 -hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE The last known good working image is daily- ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2 # Logs & notes * 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/ * 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/ * diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/ * serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/ - # Cause: shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData. Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed # Fix: + Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as + encrypted if SEV is active, as per: + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b - # Regression potential
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
** Also affects: linux-gcp (Ubuntu Hirsute) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-gcp/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
@Peter Gonda -- You seem to be 100% correct. Thank you for the helpful pointer! I have confirmed the fix and shim-side of the changes that exposed the issue. I will SRU the fix asap. ** No longer affects: shim-signed (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-gcp/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
Per the last comment, if the kernel is panicking, we should look at this from the kernel side. ** Also affects: linux-gcp (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-gcp/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
I think that this kernel needs a new fix that has been posted to KVM mailing list. The stack trace seems to match the report and the fix. https://www.spinics.net/lists/kernel/msg396.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-gcp/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1931254] Re: Google Confidential Compute fails to boot with 1.47
** Summary changed: - Google Confidnetial Compute fails to boot with 1.47 + Google Confidential Compute fails to boot with 1.47 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931254 Title: Google Confidential Compute fails to boot with 1.47 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/shim-signed/+bug/1931254/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
