[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-08-16 Thread dann frazier
Upstream has been working with me to try to determine what is going on here. The conclusion is that we believe that firmware is piggy-backing on the ARM_SMCCC_ARCH_WORKAROUND calls, and is clobbering some of the registers in the x4-x17 range. The patch series mentioned in Comment #9 happens to

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-07-19 Thread dann frazier
Note that bare metal boots also fail the same way w/ the MDS workaround enabled (failing VM boots all had a newer kernel running on the host): EFI stub: Booting Linux Kernel... EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied EFI stub: Using DTB from configuration table EFI stub:

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-25 Thread dann frazier
** Changed in: linux (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-24 Thread dann frazier
The changeset referenced in Comment #8 was part of a 21-part changeset shown here: https://patches.linaro.org/cover/141739/ I was able to backport these changes to our 4.15 kernel and get it booting w/ the MDS workaround enabled. One option is to SRU those (and a number of follow-up Fixes:

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-24 Thread dann frazier
** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title:

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-23 Thread dann frazier
I found that booting an upstream 4.18 kernel in the guest trips this problem, while an upstream 5.4 does not. I bisected and found that this commit seems to be the relevant change: commit 3b7142752e4bee153df6db4a76ca104ef0d7c0b4 (refs/bisect/bad) Author: Mark Rutland Date: Wed Jul 11 14:56:45

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-22 Thread dann frazier
I suppose the next step is to see if we can figure out why this only seems to impact bionic guests. I'll see if I can get access to the system again and try and identify the relevant difference. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-17 Thread dann frazier
I've confirmed that enabling MDS mitigation causes the problem to occur - it does not occur with it disabled. Firmware version 11.05.116 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1931728 Title:

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-16 Thread dann frazier
I've got access to an eMAG system now, so I'm going to try to reproduce there. I see the MDS mitigation setting, so I can try flipping that as well. I don't have any theory as to why that would help/hurt, but no reason not to try it. -- You received this bug notification because you are a member

Re: [Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-15 Thread Iain Lane
On Mon, Jun 14, 2021 at 11:23:06PM -, dann frazier wrote: > btw, this bug says bionic guests fails to boot - do we know if other > Ubuntu guests versions are OK? Thanks for your efforts looking into this so far. Yeah, I've only seen it in bionic. I tried spawning instances for all current

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-14 Thread dann frazier
IS gave me a copy of the libvirt xml for the guest, and I used it to build a VM as similar as possible (minus some details about storage/NIC). However I was still unable to reproduce on a ThunderX2 host. I asked a contact over at Ampere Computing to see if this symptom was something they'd seen

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-11 Thread dann frazier
I don't have access to an eMAG system at the moment, but I tried and failed to reproduce on ThunderX 2 box by running xenial + hwe kernel + cloud-archive:queens. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1931728] Re: [scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud

2021-06-11 Thread dann frazier
This is very strange. It looks like the guest has managed to start /init in the initramfs. /init is a simple shell script that just creates a few directories/mounts sysfs/proc, before printing "Loading, please wait" - which we do not see. I've tried to simulate this by removing /init from the