[Kernel-packages] [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud
Upstream has been working with me to try to determine what is going on here. The conclusion is that we believe that firmware is piggy-backing on the ARM_SMCCC_ARCH_WORKAROUND calls, and is clobbering some of the registers in the x4-x17 range. The patch series mentioned in Comment #9 happens to avoid consuming those registers, hiding the issue. Apparently clobbering those registers was previously OK, but the SMCCCv1.1 update mandates that x4-x17 be preserved, which the firmware authors may have missed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Invalid Bug description: This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack. When we boot bionic arm64 Ubuntu cloud images in "bos01" and they land on the eMAG systems, they always fail like this: ... [1.585611] Key type dns_resolver registered [1.587408] registered taskstats version 1 [1.588913] Loading compiled-in X.509 certificates [1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4' [1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969' [1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19' [1.610861] Couldn't get size: 0x800e [1.613413] MODSIGN: Couldn't get UEFI db list [1.615920] Couldn't get size: 0x800e [1.618256] MODSIGN: Couldn't get UEFI MokListRT [1.620315] Couldn't get size: 0x800e [1.622317] MODSIGN: Couldn't get UEFI dbx list [1.624446] zswap: loaded using pool lzo/zbud [1.628185] Key type big_key registered [1.629937] Key type trusted registered [1.632012] Key type encrypted registered [1.633668] AppArmor: AppArmor sha1 policy hashing enabled [1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19) [1.638009] ima: Allocated hash algorithm: sha1 [1.639272] evm: HMAC attrs: 0x1 [1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323) [1.646247] Freeing unused kernel memory: 5824K [1.657870] Checked W+X mappings: passed, no W+X pages found [1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.660250] [1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu [1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [1.668204] Call trace: [1.668981] dump_backtrace+0x0/0x198 [1.670212] show_stack+0x24/0x30 [1.671282] dump_stack+0x98/0xc8 [1.672433] panic+0x128/0x2b0 [1.673502] do_exit+0x75c/0xa80 [1.674749] do_group_exit+0x40/0xb0 [1.676191] get_signal+0x114/0x6e8 [1.677867] do_signal+0x18c/0x240 [1.679498] do_notify_resume+0xd0/0x328 [1.681302] work_pending+0x8/0x10 [1.682771] SMP: stopping secondary CPUs [1.684624] Kernel Offset: disabled [1.686119] CPU features: 0x04802008 [1.687353] Memory Limit: none [1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.688453] Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest- bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible. The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5) The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11) The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh. That's about all we have right now. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931728/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud
Note that bare metal boots also fail the same way w/ the MDS workaround enabled (failing VM boots all had a newer kernel running on the host): EFI stub: Booting Linux Kernel... EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied EFI stub: Using DTB from configuration table EFI stub: Exiting boot services and installing virtual address map... [0.00] Booting Linux on physical CPU 0x00 [0x503f0002] [0.00] Linux version 4.15.0-147-generic (buildd@bos02-arm64-076) (gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #151-Ubuntu SMP Fri Jun 18 19:18:37 UTC 2021 (Ubuntu 4.15.0-147.151-generic 4.15.18) [0.00] efi: Getting EFI parameters from FDT: [0.00] efi: EFI v2.70 by American Megatrends [0.00] efi: ACPI 2.0=0xbff596 SMBIOS 3.0=0xbff686fd98 ESRT=0xbff1a3a018 [0.00] esrt: Reserving ESRT space from 0x00bff1a3a018 to 0x00bff1a3a078. [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x00BFF596 24 (v02 ALASKA) [0.00] ACPI: XSDT 0x00BFF5960028 94 (v01 ALASKA A M I 01072009 AMI 00010013) [0.00] ACPI: FACP 0x00BFF59600C0 000114 (v06 Ampere eMAG 0003 INTL 20190509) [0.00] ACPI: DSDT 0x00BFF59601D8 0077CD (v05 ALASKA A M I 0001 INTL 20190509) [0.00] ACPI: FIDT 0x00BFF59679A8 9C (v01 ALASKA A M I 01072009 AMI 00010013) [0.00] ACPI: DBG2 0x00BFF5967A48 61 (v00 Ampere eMAG INTL 20190509) [0.00] ACPI: GTDT 0x00BFF5967AB0 000108 (v02 Ampere eMAG 0001 INTL 20190509) [0.00] ACPI: IORT 0x00BFF5967BB8 000BCC (v00 Ampere eMAG INTL 20190509) [0.00] ACPI: MCFG 0x00BFF5968788 AC (v01 Ampere eMAG 0001 INTL 20190509) [0.00] ACPI: SSDT 0x00BFF5968838 2D (v02 Ampere eMAG 0001 INTL 20190509) [0.00] ACPI: SPMI 0x00BFF5968868 41 (v05 ALASKA A M I AMI. ) [0.00] ACPI: APIC 0x00BFF59688B0 000A68 (v04 Ampere eMAG AMP. 0113) [0.00] ACPI: PCCT 0x00BFF5969318 0005D0 (v01 Ampere eMAG 0003 0113) [0.00] ACPI: BERT 0x00BFF59698E8 30 (v01 Ampere eMAG 0003 INTL 20190509) [0.00] ACPI: HEST 0x00BFF5969918 000328 (v01 Ampere eMAG 0003 INTL 20190509) [0.00] ACPI: SPCR 0x00BFF5969C40 50 (v02 A M I APTIO V 01072009 AMI. 0005000D) [0.00] ACPI: PPTT 0x00BFF5969C90 000CB8 (v01 Ampere eMAG 0003 0113) [0.00] ACPI: SPCR: console: pl011,mmio32,0x1260,115200 [0.00] ACPI: NUMA: Failed to initialise from firmware [0.00] NUMA: Faking a node at [mem 0x9000-0x00bf] [0.00] NUMA: NODE_DATA [mem 0xbe7d00-0xbeafff] [0.00] Zone ranges: [0.00] DMA [mem 0x9000-0x] [0.00] Normal [mem 0x0001-0x00bf] [0.00] Movable zone start for each node [0.00] Early memory node ranges [0.00] node 0: [mem 0x9000-0x91ff] [0.00] node 0: [mem 0x9200-0x928f] [0.00] node 0: [mem 0x9290-0xfffb] [0.00] node 0: [mem 0xfffc-0x] [0.00] node 0: [mem 0x00088000-0x000f] [0.00] node 0: [mem 0x0088-0x00bff5913fff] [0.00] node 0: [mem 0x00bff5914000-0x00bff595] [0.00] node 0: [mem 0x00bff596-0x00bff59d] [0.00] node 0: [mem 0x00bff59e-0x00bff7de] [0.00] node 0: [mem 0x00bff7df-0x00bff7e5] [0.00] node 0: [mem 0x00bff7e6-0x00bff7ff] [0.00] node 0: [mem 0x00bff800-0x00bf] [0.00] Initmem setup node 0 [mem 0x9000-0x00bf] [0.00] psci: probing for conduit method from ACPI. [0.00] psci: PSCIv1.1 detected in firmware. [0.00] psci: Using standard PSCI v0.2 function IDs [0.00] psci: MIGRATE_INFO_TYPE not supported. [0.00] psci: SMC Calling Convention v1.1 [0.00] random: get_random_bytes called from start_kernel+0xa8/0x478 with crng_init=0 [0.00] percpu: Embedded 25 pages/cpu s62232 r8192 d31976 u102400 [0.00] Detected PIPT I-cache on CPU0 [0.00] ARM_SMCCC_ARCH_WORKAROUND_1 missing from firmware [0.00] CPU features: enabling workaround for Speculative Store Bypass Disable [0.00] CPU features: detected: Kernel page table isolation (KPTI) [0.00] Built 1 zonelists, mobility grouping on. Total pages: 65995776 [0.00] Policy zone: Normal [0.00] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-147-generic
[Kernel-packages] [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud
** Changed in: linux (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Invalid Bug description: This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack. When we boot bionic arm64 Ubuntu cloud images in "bos01" and they land on the eMAG systems, they always fail like this: ... [1.585611] Key type dns_resolver registered [1.587408] registered taskstats version 1 [1.588913] Loading compiled-in X.509 certificates [1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4' [1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969' [1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19' [1.610861] Couldn't get size: 0x800e [1.613413] MODSIGN: Couldn't get UEFI db list [1.615920] Couldn't get size: 0x800e [1.618256] MODSIGN: Couldn't get UEFI MokListRT [1.620315] Couldn't get size: 0x800e [1.622317] MODSIGN: Couldn't get UEFI dbx list [1.624446] zswap: loaded using pool lzo/zbud [1.628185] Key type big_key registered [1.629937] Key type trusted registered [1.632012] Key type encrypted registered [1.633668] AppArmor: AppArmor sha1 policy hashing enabled [1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19) [1.638009] ima: Allocated hash algorithm: sha1 [1.639272] evm: HMAC attrs: 0x1 [1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323) [1.646247] Freeing unused kernel memory: 5824K [1.657870] Checked W+X mappings: passed, no W+X pages found [1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.660250] [1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu [1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [1.668204] Call trace: [1.668981] dump_backtrace+0x0/0x198 [1.670212] show_stack+0x24/0x30 [1.671282] dump_stack+0x98/0xc8 [1.672433] panic+0x128/0x2b0 [1.673502] do_exit+0x75c/0xa80 [1.674749] do_group_exit+0x40/0xb0 [1.676191] get_signal+0x114/0x6e8 [1.677867] do_signal+0x18c/0x240 [1.679498] do_notify_resume+0xd0/0x328 [1.681302] work_pending+0x8/0x10 [1.682771] SMP: stopping secondary CPUs [1.684624] Kernel Offset: disabled [1.686119] CPU features: 0x04802008 [1.687353] Memory Limit: none [1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.688453] Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest- bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible. The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5) The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11) The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh. That's about all we have right now. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931728/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud
The changeset referenced in Comment #8 was part of a 21-part changeset shown here: https://patches.linaro.org/cover/141739/ I was able to backport these changes to our 4.15 kernel and get it booting w/ the MDS workaround enabled. One option is to SRU those (and a number of follow-up Fixes: changes for them). Presumably that's of some security benefit, though I'm not sure how significant of one. It also isn't clear exactly why those changes are needed for compat with the firmware MDS workaround. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Invalid Bug description: This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack. When we boot bionic arm64 Ubuntu cloud images in "bos01" and they land on the eMAG systems, they always fail like this: ... [1.585611] Key type dns_resolver registered [1.587408] registered taskstats version 1 [1.588913] Loading compiled-in X.509 certificates [1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4' [1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969' [1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19' [1.610861] Couldn't get size: 0x800e [1.613413] MODSIGN: Couldn't get UEFI db list [1.615920] Couldn't get size: 0x800e [1.618256] MODSIGN: Couldn't get UEFI MokListRT [1.620315] Couldn't get size: 0x800e [1.622317] MODSIGN: Couldn't get UEFI dbx list [1.624446] zswap: loaded using pool lzo/zbud [1.628185] Key type big_key registered [1.629937] Key type trusted registered [1.632012] Key type encrypted registered [1.633668] AppArmor: AppArmor sha1 policy hashing enabled [1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19) [1.638009] ima: Allocated hash algorithm: sha1 [1.639272] evm: HMAC attrs: 0x1 [1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323) [1.646247] Freeing unused kernel memory: 5824K [1.657870] Checked W+X mappings: passed, no W+X pages found [1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.660250] [1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu [1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [1.668204] Call trace: [1.668981] dump_backtrace+0x0/0x198 [1.670212] show_stack+0x24/0x30 [1.671282] dump_stack+0x98/0xc8 [1.672433] panic+0x128/0x2b0 [1.673502] do_exit+0x75c/0xa80 [1.674749] do_group_exit+0x40/0xb0 [1.676191] get_signal+0x114/0x6e8 [1.677867] do_signal+0x18c/0x240 [1.679498] do_notify_resume+0xd0/0x328 [1.681302] work_pending+0x8/0x10 [1.682771] SMP: stopping secondary CPUs [1.684624] Kernel Offset: disabled [1.686119] CPU features: 0x04802008 [1.687353] Memory Limit: none [1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.688453] Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest- bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible. The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5) The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11) The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh. That's about all we have right now. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931728/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud
** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Invalid Bug description: This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack. When we boot bionic arm64 Ubuntu cloud images in "bos01" and they land on the eMAG systems, they always fail like this: ... [1.585611] Key type dns_resolver registered [1.587408] registered taskstats version 1 [1.588913] Loading compiled-in X.509 certificates [1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4' [1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969' [1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19' [1.610861] Couldn't get size: 0x800e [1.613413] MODSIGN: Couldn't get UEFI db list [1.615920] Couldn't get size: 0x800e [1.618256] MODSIGN: Couldn't get UEFI MokListRT [1.620315] Couldn't get size: 0x800e [1.622317] MODSIGN: Couldn't get UEFI dbx list [1.624446] zswap: loaded using pool lzo/zbud [1.628185] Key type big_key registered [1.629937] Key type trusted registered [1.632012] Key type encrypted registered [1.633668] AppArmor: AppArmor sha1 policy hashing enabled [1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19) [1.638009] ima: Allocated hash algorithm: sha1 [1.639272] evm: HMAC attrs: 0x1 [1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323) [1.646247] Freeing unused kernel memory: 5824K [1.657870] Checked W+X mappings: passed, no W+X pages found [1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.660250] [1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu [1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [1.668204] Call trace: [1.668981] dump_backtrace+0x0/0x198 [1.670212] show_stack+0x24/0x30 [1.671282] dump_stack+0x98/0xc8 [1.672433] panic+0x128/0x2b0 [1.673502] do_exit+0x75c/0xa80 [1.674749] do_group_exit+0x40/0xb0 [1.676191] get_signal+0x114/0x6e8 [1.677867] do_signal+0x18c/0x240 [1.679498] do_notify_resume+0xd0/0x328 [1.681302] work_pending+0x8/0x10 [1.682771] SMP: stopping secondary CPUs [1.684624] Kernel Offset: disabled [1.686119] CPU features: 0x04802008 [1.687353] Memory Limit: none [1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0005 [1.688453] Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest- bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible. The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5) The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11) The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh. That's about all we have right now. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1931728/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
