On Thu, Sep 03, 2026 at 03:28:26PM +0200, Michal Suchánek wrote: > On Thu, Sep 03, 2026 at 06:06:16PM +0530, Harsh Prateek Bora wrote: > > + Amit, Gautam - since they recently came across a similar regression which > > is seen with non KVM workload as well. > > + Venkat - for CI improvement suggestion below. > > > > Hi Michal, > > > > On 03/09/26 5:55 pm, Michal Suchánek wrote: > > > > > > > There is another problem that Linux 7.2.2 often locks up on boot inside > > > the guest, and as a result a different kernel is needed in the guest to > > > reproduce the host crash. This is more recent, linux 6.12 does not have > > > this problem. Will try some different kernel combinations to see if I > > > can narrow this down. For this the repro is merely booting the kernel in > > > the VM, it typically fails early before userspace produces any messages. > > This one regressed between 6.19.12 > https://github.com/openSUSE/kernel-source/blob/9f6830f/config/ppc64le/default > and 7.0.12 > https://github.com/openSUSE/kernel-source/blob/2ebf0bc/config/ppc64le/default
Which is when preempt was enabled. Very likely preexisting problem that was exposed by enabling preemption again. @@ -121,9 +122,7 @@ CONFIG_BPF_LSM=y CONFIG_PREEMPT_BUILD=y CONFIG_ARCH_HAS_PREEMPT_LAZY=y -# CONFIG_PREEMPT_NONE is not set -CONFIG_PREEMPT_VOLUNTARY=y -# CONFIG_PREEMPT is not set +CONFIG_PREEMPT=y # CONFIG_PREEMPT_LAZY is not set CONFIG_PREEMPT_COUNT=y CONFIG_PREEMPTION=y Thanks Michal > > Thanks > > Michal > > SLOF ********************************************************************** > QEMU Starting > Build Date = Aug 13 2026 14:47:36 > FW Version = abuild@OBS release 20230918 > Press "s" to enter Open Firmware. > > Populating /vdevice methods > Populating /vdevice/vty@30000000 > Populating /vdevice/nvram@71000000 > Populating /pci@800000020000000 > Loading Linux 7.0.12-1.g2ebf0bc-default ... > Loading initial ramdisk ... > OF stdout device is: /vdevice/vty@30000000 > Preparing to boot Linux version 7.0.12-1.g2ebf0bc-default (geeko@buildhost) > (gcc (SUSE Linux) 15.3.0, GNU ld (GNU Binutils; openSUSE Tumbleweed) > 2.45.0.20251103-4) #1 SMP PREEMPT_DYNAMIC Mon Jun 15 08:39:32 UTC 2026 > (2ebf0bc) > Detected machine type: 0000000000000101 > command line: BOOT_IMAGE=/boot/vmlinux-7.0.12-1.g2ebf0bc-default > root=UUID=304c7f07-efbb-1070-2f27-accd935ec088 rw quiet systemd.show_status=1 > security=selinux selinux=1 > Max number of cores passed to firmware: 8192 (NR_CPUS = 8192) > Calling ibm,client-architecture-support... done > memory layout at init: > memory_limit : 0000000000000000 (16 MB aligned) > alloc_bottom : 00000000066b0000 > alloc_top : 0000000030000000 > alloc_top_hi : 0000003e00000000 > rmo_top : 0000000030000000 > ram_top : 0000003e00000000 > instantiating rtas at 0x000000002fff0000... done > prom_hold_cpus: skipped > copying OF device tree... > Building dt strings... > Building dt structure... > Device tree strings 0x00000000066c0000 -> 0x00000000066c0bec > Device tree struct 0x00000000066d0000 -> 0x00000000066f0000 > Quiescing Open Firmware ... > Booting Linux via __start() @ 0x0000000000250000 ... > [ 0.000000][ T0] ERROR: Failed to allocate trace buffer > [ 0.000000][ T0] ERROR: tracer: failed to allocate ring buffer! > Linux ppc64le > #1 SMP PREEMPT_D[ 0.285758][ T673] BUG: Kernel NULL pointer dereference > on read at 0x00000010 > [ 0.285877][ T673] Faulting instruction address: 0xc000000000485bd0 > [ 0.285903][ T1] VFS: Dquot-cache hash table entries: 8192 (order 0, > 65536 bytes) > [ 0.285954][ T673] Oops: Kernel access of bad area, sig: 7 [#1] > [ 0.286133][ T673] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=8192 NUMA > pSeries > [ 0.286240][ T673] Modules linked in: > [ 0.286288][ T673] CPU: 15 UID: 0 PID: 673 Comm: kworker/u531:0 Not > tainted 7.0.12-1.g2ebf0bc-default #1 PREEMPT(full) openSUSE Tumbleweed > (unreleased) cf0846124d7853fab338aeae87203b36cac18419 > [ 0.286527][ T673] Hardware name: IBM pSeries (emulated by qemu) Power11 > (architected) 0x820200 0xf000007 of:SLOF,HEAD hv:linux,kvm pSeries > [ 0.286739][ T673] Workqueue: trace_init_wq tracer_init_tracefs_work_func > [ 0.286818][ T673] NIP: c000000000485bd0 LR: c000000000448864 CTR: > c0000000005a3420 > [ 0.286906][ T673] REGS: c00000000a7a7960 TRAP: 0300 Not tainted > (7.0.12-1.g2ebf0bc-default) > [ 0.287014][ T673] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: > 44088404 XER: 00000000 > [ 0.287122][ T673] CFAR: c000000000448860 DAR: 0000000000000010 DSISR: > 00080000 IRQMASK: 0 > [ 0.287122][ T673] GPR00: c000000000448864 c00000000a7a7c00 > c000000001f38100 c000000002a5c098 > [ 0.287122][ T673] GPR04: c0000000017dc5f0 c0000000017dc5e8 > 000000000000001f 0000000000000064 > [ 0.287122][ T673] GPR08: c0000000017dc628 00000000000005f0 > 00000000000005e8 0000000084000404 > [ 0.287122][ T673] GPR12: c0000000005a3420 c000003dfff43d80 > c0000000018e5600 c0000000018e54f0 > [ 0.287122][ T673] GPR16: 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > [ 0.287122][ T673] GPR20: c0000000013ec418 c0000000013ef6d8 > c0000000017dc5a0 c0000000017dc590 > [ 0.287122][ T673] GPR24: c000000001825330 c0000000017dc628 > c000000014b52a05 0000000000000000 > [ 0.287122][ T673] GPR28: c0000000017dc5e8 0000000000000000 > c0000000017dc5f0 c000000002a5e008 > [ 0.287970][ T673] NIP [c000000000485bd0] __find_event_file+0x70/0x3c0 > [ 0.288046][ T673] LR [c000000000448864] init_tracer_tracefs+0x274/0xc80 > [ 0.288122][ T673] Call Trace: > [ 0.288167][ T673] [c00000000a7a7c00] [c00000000a7a7c60] > 0xc00000000a7a7c60 (unreliable) > [ 0.288258][ T673] [c00000000a7a7c60] [c000000000448864] > init_tracer_tracefs+0x274/0xc80 > [ 0.288348][ T673] [c00000000a7a7dc0] [c00000000203fcf0] > tracer_init_tracefs_work_func+0x50/0x320 > [ 0.288452][ T673] [c00000000a7a7e50] [c0000000002620e8] > process_one_work+0x1e8/0x5c0 > [ 0.288541][ T673] [c00000000a7a7f10] [c00000000026309c] > worker_thread+0x1dc/0x3d0 > [ 0.288630][ T673] [c00000000a7a7f90] [c00000000026fa34] > kthread+0x194/0x1b0 > [ 0.288721][ T673] [c00000000a7a7fe0] [c00000000000de58] > start_kernel_thread+0x14/0x18 > [ 0.288810][ T673] Code: fb410030 fb810040 fba10048 7cbc2b78 3ba00000 > fbc10050 7d194378 7c9e2378 2e2a0fc0 2da90fc0 f8010070 60420000 <e93b0010> > 81490058 e8890018 714a0208 > [ 0.289001][ T673] ---[ end trace 0000000000000000 ]--- > [ 0.290718][ T673] pstore: backend (nvram) writing error (-1) > [ 0.290805][ T673] > [ 0.290839][ T673] note: kworker/u531:0[673] exited with irqs disabled > [ 0.304261][ T1] NET: Registered PF_INET protocol family > [ 0.304602][ T1] IP idents hash table entries: 262144 (order: 5, > 2097152 bytes, linear) > [ 0.315892][ T1] tcp_listen_portaddr_hash hash table entries: 65536 > (order: 4, 1048576 bytes, linear) > [ 0.316104][ T1] Table-perturb hash table entries: 65536 (order: 2, > 262144 bytes, linear) > [ 0.316224][ T1] TCP established hash table entries: 524288 (order: 6, > 4194304 bytes, linear) > [ 0.316873][ T1] TCP bind hash table entries: 65536 (order: 5, 2097152 > bytes, linear) > [ 0.317199][ T1] TCP: Hash tables configured (established 524288 bind > 65536) > [ 0.317748][ T1] MPTCP token hash table entries: 65536 (order: 5, > 1572864 bytes, linear) > [ 0.317970][ T1] UDP hash table entries: 65536 (order: 6, 4194304 > bytes, linear) > [ 0.318402][ T1] UDP-Lite hash table entries: 65536 (order: 6, 4194304 > bytes, linear) > [ 0.319148][ T1] NET: Registered PF_UNIX/PF_LOCAL protocol family > [ 0.319249][ T1] NET: Registered PF_XDP protocol family > [ 0.320019][ T1] PCI: CLS 0 bytes, default 128 > [ 0.320281][ T1] rtas_flash: no firmware flash support > [ 0.320967][ T770] Trying to unpack rootfs image as initramfs... > [ 0.331385][ T1] Initialise system trusted keyrings > [ 0.331739][ T1] Key type blacklist registered > [ 0.332191][ T1] workingset: timestamp_bits=38 max_order=22 > bucket_order=0 > [ 0.333804][ T1] integrity: Platform Keyring initialized > [ 0.333875][ T1] integrity: Machine keyring initialized > [ 0.333935][ T1] Allocating IMA blacklist keyring. > [ 0.343039][ T1] Key type asymmetric registered > [ 0.343109][ T1] Asymmetric key parser 'x509' registered > [ 0.344215][ T1] Block layer SCSI generic (bsg) driver version 0.4 > loaded (major 246) > [ 0.344623][ T1] io scheduler mq-deadline registered > [ 0.344688][ T1] io scheduler kyber registered > [ 0.344800][ T1] io scheduler bfq registered > [ 0.380554][ T1] ledtrig-cpu: registered to indicate activity on CPUs > [ 0.380794][ T1] virtio-pci 0000:00:01.0: enabling device (0100 -> 0103) > [ 0.382384][ T1] virtio-pci 0000:00:01.0: > ibm,query-pe-dma-windows(2026) 800 8000000 20000000 returned 0, lb=2000000000 > ps=107 wn=1 > [ 0.383175][ T1] virtio-pci 0000:00:01.0: > ibm,create-pe-dma-window(2027) 800 8000000 20000000 18 26 returned 0 (liobn = > 0x80000001 starting addr = 8000000 0) > [ 0.386395][ T1] virtio-pci 0000:00:01.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.386509][ T1] virtio-pci 0000:00:01.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.386621][ T1] virtio-pci 0000:00:01.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.386723][ T1] virtio-pci 0000:00:01.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.388040][ T1] virtio-pci 0000:00:03.0: enabling device (0100 -> 0103) > [ 0.390005][ T1] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.390111][ T1] virtio-pci 0000:00:03.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.390216][ T1] virtio-pci 0000:00:03.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.390316][ T1] virtio-pci 0000:00:03.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.392116][ T1] virtio-pci 0000:00:04.0: enabling device (0100 -> 0103) > [ 0.394076][ T1] virtio-pci 0000:00:04.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.394184][ T1] virtio-pci 0000:00:04.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.394288][ T1] virtio-pci 0000:00:04.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.394389][ T1] virtio-pci 0000:00:04.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.395729][ T1] virtio-pci 0000:00:05.0: enabling device (0100 -> 0103) > [ 0.397705][ T1] virtio-pci 0000:00:05.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.397814][ T1] virtio-pci 0000:00:05.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.397919][ T1] virtio-pci 0000:00:05.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.398024][ T1] virtio-pci 0000:00:05.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.399426][ T1] virtio-pci 0000:00:06.0: enabling device (0100 -> 0103) > [ 0.401288][ T1] virtio-pci 0000:00:06.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.401396][ T1] virtio-pci 0000:00:06.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.401502][ T1] virtio-pci 0000:00:06.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.401604][ T1] virtio-pci 0000:00:06.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.403644][ T1] virtio-pci 0000:00:07.0: enabling device (0100 -> 0103) > [ 0.405723][ T1] virtio-pci 0000:00:07.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.405830][ T1] virtio-pci 0000:00:07.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.405936][ T1] virtio-pci 0000:00:07.0: lsa_required: 0, lsa_enabled: > 0, direct mapping: 1 > [ 0.406044][ T1] virtio-pci 0000:00:07.0: iommu: 64-bit OK but direct > DMA is limited by 800004000000000 > [ 0.409460][ T1] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled > [ 0.427929][ T770] Freeing initrd memory: 21120K > [ 0.431965][ T1] Non-volatile memory driver v1.3 > [ 0.432069][ T1] pseries_rng: Registering IBM pSeries RNG driver > [ 0.433884][ T1] mousedev: PS/2 mouse device common for all mice > [ 0.435769][ T1] pseries_idle_driver registered > [ 0.435912][ T1] hid: raw HID events driver (C) Jiri Kosina > [ 0.436062][ T1] drop_monitor: Initializing network drop monitor service > [ 0.436412][ T1] NET: Registered PF_INET6 protocol family > [ 0.437731][ T1] Segment Routing with IPv6 > [ 0.437819][ T1] RPL Segment Routing with IPv6 > [ 0.437897][ T1] In-situ OAM (IOAM) with IPv6 > [ 0.438002][ T1] PFKEY is deprecated and scheduled to be removed in > 2027, please contact the netdev mailing list > [ 0.438125][ T1] NET: Registered PF_KEY protocol family > [ 0.438508][ T1] secvar-sysfs: Failed to retrieve secvar operations > [ 0.441215][ T1] registered taskstats version 1 > [ 0.464617][ T1] Loading compiled-in X.509 certificates > [ 0.481670][ T1] Loaded X.509 cert 'home:tiwai OBS Project: > 466e10c6490242bdd3a9985914c1906197b53a81' > [ 0.487739][ T1] Demotion targets for Node 0: null > [ 0.487834][ T1] page_owner is disabled > [ 0.518088][ T1] Key type .fscrypt registered > [ 0.518163][ T1] Key type fscrypt-provisioning registered > [ 0.518473][ T1] Key type big_key registered > [ 0.536738][ T1] Key type encrypted registered > [ 0.536916][ T1] Secure boot mode disabled > [ 0.536994][ T1] ima: No TPM chip found, activating TPM-bypass! > [ 0.537070][ T1] Loading compiled-in module X.509 certificates > [ 0.537347][ T1] Loaded X.509 cert 'home:tiwai OBS Project: > 466e10c6490242bdd3a9985914c1906197b53a81' > [ 0.537451][ T1] ima: Allocated hash algorithm: sha256 > [ 0.538132][ T1] Secure boot mode disabled > [ 0.538236][ T1] Trusted boot mode disabled > [ 0.538303][ T1] ima: No architecture policies found > [ 0.538448][ T1] evm: Initialising EVM extended attributes: > [ 0.538523][ T1] evm: security.selinux > [ 0.538569][ T1] evm: security.SMACK64 (disabled) > [ 0.538629][ T1] evm: security.SMACK64EXEC (disabled) > [ 0.538689][ T1] evm: security.SMACK64TRANSMUTE (disabled) > [ 0.538762][ T1] evm: security.SMACK64MMAP (disabled) > [ 0.538822][ T1] evm: security.apparmor > [ 0.538868][ T1] evm: security.ima > [ 0.538915][ T1] evm: security.capability > [ 0.538983][ T1] evm: HMAC attrs: 0x1 > [ 0.553344][ T1] SED: plpks not available > > > >
