Bug#1029270: linux-image-6.1.0-2-686-pae fails to boot in qemu
Hi, On Wed, 1 Feb 2023 14:59:06 +0100 Salvatore Bonaccorso wrote: > On Tue, Jan 31, 2023 at 05:05:53PM +0100, Salvatore Bonaccorso wrote: > > Hi, > > > > On Fri, Jan 20, 2023 at 04:42:27PM +0100, Helmut Grohne wrote: > > > Control: severity -1 normal > > > > > > On Fri, Jan 20, 2023 at 03:55:53PM +0100, Helmut Grohne wrote: > > > > A CI job of debvm started failing. debvm creates a minimalistic virtual > > > > machine based on Debian unstable i386 and tries to run it in qemu. With > > > > the previous kernel package that worked. Once updating to 6.1.7-1, it > > > > fails to boot: > > > > > > I have one more data point. If you pass -enable-kvm to qemu, it actually > > > boots. It only fails to boot when disabling kvm. That shouldn't affect > > > that many users. It's still unclear what causes the issue. > > > > > > I'm also pulling qemu maintainer mjt into the discussion for possible > > > input. > > > > I cannot now confirm the issue anymore with 6.1.8-1 as uploaded to > > unstable. So think we can close the bug accordingly with 6.1.8-1 > > though it would be good to know/pin point the issue. > > Doing for now. Still think would be nice to know the root cause. since my house was cold this morning, I bisected the kernel again between 6.1.7 and 6.1.8. The commit that made the kernel panic go away is this one: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6c27fc15747967e350fa9f253a034ff03e943ccf Thanks! cheers, josch P.S.: if future-me wants to do this again, this is the command I ran on every bisect step: git reset --hard \ && git clean -fdx \ && cp /boot/config-6.1.0-2-686-pae .config \ && scripts/config --disable MODULE_SIG \ && scripts/config --disable DEBUG_INFO \ && QUILT_PATCHES=~/linux-6.1.7/debian/patches quilt push -a \ && time make -j16 bzImage \ && env --chdir=~/debvm bin/debvm-run -i ~/debvm/rootfs.ext4 -- -kernel ~/linux/arch/x86/boot/bzImage signature.asc Description: signature
Bug#1029270: linux-image-6.1.0-2-686-pae fails to boot in qemu
Hi, On Fri, Jan 20, 2023 at 04:42:27PM +0100, Helmut Grohne wrote: > Control: severity -1 normal > > On Fri, Jan 20, 2023 at 03:55:53PM +0100, Helmut Grohne wrote: > > A CI job of debvm started failing. debvm creates a minimalistic virtual > > machine based on Debian unstable i386 and tries to run it in qemu. With > > the previous kernel package that worked. Once updating to 6.1.7-1, it > > fails to boot: > > I have one more data point. If you pass -enable-kvm to qemu, it actually > boots. It only fails to boot when disabling kvm. That shouldn't affect > that many users. It's still unclear what causes the issue. > > I'm also pulling qemu maintainer mjt into the discussion for possible > input. I cannot now confirm the issue anymore with 6.1.8-1 as uploaded to unstable. So think we can close the bug accordingly with 6.1.8-1 though it would be good to know/pin point the issue. Regards, Salvatore
Bug#1029270: linux-image-6.1.0-2-686-pae fails to boot in qemu
Hi, I investigated this a bit more and can confirm that this worked: linux-image-6.1.0-1-686-pae 6.1.4-1 while this one does not: linux-image-6.1.0-2-686-pae 6.1.7-1 To figure out where the regression comes from, I built 6.1.7 from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git with the kernel config from /boot/config-6.1.0-2-686-pae. Strangely enough, I was *not* able to confirm the bug. So I applied the quilt patch series from src:linux (= 6.1.7-1) on top and tried again. Now I saw the same output that Helmut posted. I tried the same with tag 6.1.4 from linux.git and the Debian patches and the problem did not show. So I bisected between 6.1.4 and 6.1.7, always applying the quilt patch series (which did not change between these two versions). If I did not make any mistakes, the following commit introduced this regression: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=92b0051217f23cba7541d6c4ecbca026477fb642 I can confirm that if I rebuild 6.1.7 with that patch reverted and the Debian patches on top, the error is gone. Does that seem plausible? Thanks! cheers, josch
Bug#1029270: linux-image-6.1.0-2-686-pae fails to boot in qemu
Control: severity -1 normal On Fri, Jan 20, 2023 at 03:55:53PM +0100, Helmut Grohne wrote: > A CI job of debvm started failing. debvm creates a minimalistic virtual > machine based on Debian unstable i386 and tries to run it in qemu. With > the previous kernel package that worked. Once updating to 6.1.7-1, it > fails to boot: I have one more data point. If you pass -enable-kvm to qemu, it actually boots. It only fails to boot when disabling kvm. That shouldn't affect that many users. It's still unclear what causes the issue. I'm also pulling qemu maintainer mjt into the discussion for possible input. Helmut
Bug#1029270: linux-image-6.1.0-2-686-pae fails to boot in qemu
Package: linux-image-6.1.0-2-686-pae Version: 6.1.7-1 Severity: important Control: affects -1 + debvm A CI job of debvm started failing. debvm creates a minimalistic virtual machine based on Debian unstable i386 and tries to run it in qemu. With the previous kernel package that worked. Once updating to 6.1.7-1, it fails to boot: https://salsa.debian.org/helmutg/debvm/-/jobs/3824112 | [1.158184] traps: PANIC: double fault, error_code: 0x0 | [1.158184] double fault: [#1] PREEMPT SMP PTI | [1.158184] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.1.0-2-686-pae #1 Debian 6.1.7-1 | [1.158184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014 | [1.158184] EIP: __kmem_cache_alloc_node+0xc4/0x350 | [1.158184] Code: 85 c9 0f 84 6e 02 00 00 8b 75 f0 8b 47 1c 01 f0 89 c1 8b 00 33 47 78 0f c9 31 c8 8d 4a 20 89 c3 89 f0 8b 37 64 0f c7 0e 75 ba <8b> 75 e8 8b 47 1c 8d 74 26 00 3e 8d 74 26 00 3e 8d 74 26 00 8b 47 | [1.158184] EAX: c11ed0c0 EBX: c11ed300 ECX: 0301 EDX: 02e1 | [1.158184] ESI: d6e1e978 EDI: c10013c0 EBP: c1145e4c ESP: c1145e30 | [1.158184] DS: 007b ES: 007b FS: 00d8 GS: SS: 0068 EFLAGS: c11ed2c2 | [1.158184] CR0: 80050033 CR2: CR3: 16e2c000 CR4: 06b0 | [1.158184] Call Trace: | [1.158184] __kmalloc+0x42/0x140 | [1.158184] ? cache_random_seq_create+0x81/0x130 | [1.158184] ? cache_random_seq_create+0x81/0x130 | [1.158184] cache_random_seq_create+0x81/0x130 | [1.158184] init_cache_random_seq+0x39/0x80 | [1.158184] __kmem_cache_create+0x10f/0x470 | [1.158184] kmem_cache_create_usercopy+0x158/0x2a0 | [1.158184] kmem_cache_create+0x17/0x20 | [1.158184] proto_register+0x183/0x240 | [1.158184] ? ipv4_offload_init+0x6e/0x6e | [1.158184] inet_init+0x37/0x261 | [1.158184] do_one_initcall+0x4b/0x1e0 | [1.158184] kernel_init_freeable+0x1a5/0x1e5 | [1.158184] ? rest_init+0xb0/0xb0 | [1.158184] kernel_init+0x17/0x100 | [1.158184] ret_from_fork+0x1c/0x28 | [1.158184] Modules linked in: | [1.158184] ---[ end trace ]--- | [1.158184] EIP: __kmem_cache_alloc_node+0xc4/0x350 | [1.158184] Code: 85 c9 0f 84 6e 02 00 00 8b 75 f0 8b 47 1c 01 f0 89 c1 8b 00 33 47 78 0f c9 31 c8 8d 4a 20 89 c3 89 f0 8b 37 64 0f c7 0e 75 ba <8b> 75 e8 8b 47 1c 8d 74 26 00 3e 8d 74 26 00 3e 8d 74 26 00 8b 47 | [1.158184] EAX: c11ed0c0 EBX: c11ed300 ECX: 0301 EDX: 02e1 | [1.158184] ESI: d6e1e978 EDI: c10013c0 EBP: c1145e4c ESP: c1145e30 | [1.158184] DS: 007b ES: 007b FS: 00d8 GS: SS: 0068 EFLAGS: c11ed2c2 | [1.158184] CR0: 80050033 CR2: CR3: 16e2c000 CR4: 06b0 | [1.158184] Kernel panic - not syncing: Fatal exception in interrupt You can easily reproduce this by installing debvm, running `debvm-create -a i386` (which will create a rootfs.ext4) and then `debvm-run`. The only other packages changed since the last successful run are: * openssl * glib2.0 * mmdebstrap We can rule out mmdebstrap as a cause by adding `-r bookworm` to the invocation and seeing that things boot. The other packages shouldn't be able to cause a kernel panic. Let me know if you need anything else. Helmut
