[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
This bug was fixed in the package linux - 5.13.0-19.19 --- linux (5.13.0-19.19) impish; urgency=medium * impish/linux: 5.13.0-19.19 -proposed tracker (LP: #1946337) * impish:linux-aws 5.13 panic during systemd autotest (LP: #1946001) - [Config] disable KFENCE -- Andrea Righi Thu, 07 Oct 2021 11:09:51 +0200 ** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Also good in focal: ubuntu@doerfel:~$ cat /proc/version Linux version 5.13.0-17-generic (buildd@bos02-arm64-034) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #17~20.04.1-Ubuntu SMP Tue Sep 28 14:05:10 UTC 2021 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Verified: ubuntu@doerfel:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.13.0-18-generic root=UUID=36113164-ebf4-4fed-9d98-4b0b859bf98e ro acpi=force ubuntu@doerfel:~$ cat /proc/version Linux version 5.13.0-18-generic (buildd@bos02-arm64-027) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #18-Ubuntu SMP Mon Oct 4 14:52:32 UTC 2021 ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Changed in: linux (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
This bug is awaiting verification that the linux- hwe-5.13/5.13.0-17.17~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
The patch was merged into 5.13.0-17.17, which is currently in impish-proposed. At some point it should promulgate to the release pocket. You can view the status here: https://launchpad.net/ubuntu/+source/linux I don't know how long after that it will before it appears in images. As a workaround, you can test with kpti=off on the kernel command line. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Thanks a lot dann. I dont know the release process of the kernel, and when our public images are generated by CPC. I have to test it when is uploaded here: https://images.maas.io/ephemeral-v3/candidate/impish/arm64/ Do you have any tip about how to be aware when this should happen? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
A patch has been merged into the arm64 tree and is tagged for stable 5.13+ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Patch submitted: https://www.spinics.net/lists/arm-kernel/msg922152.html ** Changed in: linux (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Root caused and reported upstream: https://www.spinics.net/lists/arm-kernel/msg921821.html ** Changed in: linux (Ubuntu) Assignee: (unassigned) => dann frazier (dannf) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Update: The issue does not follow toolchain, which is good. Rather I did find that upstream v5.13-rc1 is stable - with the patch in comment #15 reverted - while upstream v5.13 is not. I bisected v5.13-rc1..v5.13, reverting the comment #15 patch at each test. A "bad" kernel wouldn't always fail the same way, but would always fail before completing boot. The bisect hit this commit: commit 0c6c2d3615efb7c292573f2e6c886929a2b2da6c (HEAD, refs/bisect/bad) Author: Mark Brown Date: Wed Apr 28 13:12:31 2021 +0100 arm64: Generate cpucaps.h While this looks innocuous, it is messing with the code that chooses which "features" a CPU has, which includes erratum that may need kernel workarounds. So I went back and compared the CPU features messages between a "good" kernel and a "bad" one. Noticeably missing from a "bad" one was this message: [0.00] CPU features: kernel page table isolation forced OFF by ARM64_WORKAROUND_CAVIUM_27456 I went back and tested Ubuntu's 5.13.0-16 w/ kpti=off, and it booted fine. As does upstream v5.15-rc2, where previously I was also seeing stack overflows/corruption. So, seems like the above change is likely the problem, next step is to figure out why. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
hm.. you're right. I was tracking down a different issue which had the following backtrace: [ 10.701967] Call trace: [ 10.704415] ata_host_activate+0x160/0x170 [ 10.708518] ahci_host_activate+0x170/0x1e0 [ 10.712711] ahci_init_one+0x898/0xd74 [ahci] [ 10.717116] local_pci_probe+0x4c/0xc0 [ 10.720883] work_for_cpu_fn+0x28/0x40 [ 10.724635] process_one_work+0x20c/0x4d0 [ 10.728647] worker_thread+0x250/0x564 [ 10.732393] kthread+0x134/0x140 [ 10.735616] ret_from_fork+0x10/0x18 Reverting the bisect-id'd commit from upstream 5.13-rc1 does fix this issue for me, but perhaps it was already fixed in v5.13 final. I *can* reproduce the backtrace here w/ the archive 5.13.0-16, but I can not with a locally built 5.13-rc1 (w/ the above patch reverted). The fact that you can reproduce with 5.13-rc1 makes me wonder if the issue maybe toolchain related. I'm building in a hirsute environment. I'll try building 5.13.0-16 in a hirsute environment to test that theory. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
I think it's a red herring: even after reversing this commit on top of 5.13-rc1 (the first tag showing the issue) or Impish/master-next HEAD, the kernel still Oops and hang there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
First bad commit: commit 9ec37efb87832b578d7972fc80b04d94f5d2bbe3 (HEAD, refs/bisect/bad) Author: Marc Zyngier Date: Tue Mar 30 16:11:42 2021 +0100 PCI/MSI: Make pci_host_common_probe() declare its reliance on MSI domains The generic PCI host driver relies on MSI domains for MSIs to be provided to its end-points. Make this dependency explicit. This cures the warnings occuring on arm/arm64 VMs when booted with PCI virtio devices and no MSI controller (no GICv3 ITS, for example). It is likely that other drivers will need to express the same dependency. Link: https://lore.kernel.org/r/20210330151145.997953-12-...@kernel.org Signed-off-by: Marc Zyngier Signed-off-by: Lorenzo Pieralisi Acked-by: Bjorn Helgaas -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
For the CRB1S systems, the following is the firmware string that matters, and it shows that phanpy is running the latest (and very likely final) version: BIOS Date: 06/14/2018 14:42:48 Ver: 0ACGA022 Note that this is also reproducible on our Gigabyte R120 systems which use acpi=force by default, which suggests this is not a device tree problem. I can reproduce with upstream v5.13, and I've a bisect between that and v5.11 in progress. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Feel free to bisect, i'm checking our SAUCE patches/configs. BTW: Firmware Version: 2017-10-12 12:34:31 it's the last FW version available, isn't it? I'm wondering about the device tree. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
I can attempt a bisect - let me know if someone else is already doing that :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Attachment added: "System.map-5.13.0-16-generic" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524362/+files/System.map-5.13.0-16-generic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Attachment added: "System.map-5.14.0-9-generic" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524363/+files/System.map-5.14.0-9-generic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Attachment added: "System.map-5.13.0-14-generic" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524361/+files/System.map-5.13.0-14-generic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
5.14.0-9.9 exhibits kernel stack corruption on boot. ** Attachment added: "phanpy.bootlog" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524292/+files/phanpy.bootlog -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
With 5.13.0-16.16, the issue is way more evident. ** Attachment added: "phanpy-fullbootlog.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524116/+files/phanpy-fullbootlog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Attachment added: "Phanpy full bootlog attached" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5524091/+files/phanpy-fullbootlog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
I can reproduce the issue on phanpy with 5.13.0-14-generic: ... [ 169.882563] random: fast init done [ 169.926830] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... Failed to send exit request: Connection refused done. [ 171.269768] systemd[1]: Inserted module 'autofs4' [ 171.421523] systemd[1]: systemd 248.3-1ubuntu3 running in system mode. (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid) [ 171.463836] systemd[1]: Detected architecture arm64. Welcome to Ubuntu Impish Indri (development branch)! [ 171.523576] systemd[1]: Hostname set to . [ 173.864657] random: crng init done ... I've noticed some stack traces during boot, i'll try to extract them. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
Im not able to run: apport-collect ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942633] Re: Can not boot impish in Cavium ThunderX
** Attachment added: "debug_impish_arm_machine_20210830.log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+attachment/5522867/+files/debug_impish_arm_machine_20210830.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942633 Title: Can not boot impish in Cavium ThunderX To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942633/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
