Dear Maintainer I just created another clone of the VM and instead of upgrading from Stretch to Buster I upgraded `nftables` and `linux-image-amd64` to `stretch-backports` with the following results:
| upgrading kernel | not upgrading kernel -----------------------+-------------------------+--------------------- upgrading nftables | fails to boot | ok not upgrading nftables | assert fails, but boots | ok Thus the issue appears to be related to differences between Linux 4.9 and Linux 4.19. This is the list of packages where the only difference to the production VM is the kernel from backports: > Start-Date: 2019-07-07 22:40:44 > Commandline: apt install -t stretch-backports linux-image-amd64 > linux-headers-amd64 > Install: linux-headers-4.19.0-0.bpo.5-amd64:amd64 (4.19.37-4~bpo9+1, > automatic), linux-image-4.19.0-0.bpo.5-amd64:amd64 (4.19.37-4~bpo9+1, > automatic), linux-kbuild-4.19:amd64 (4.19.37-4~bpo9+1, automatic), > linux-headers-4.19.0-0.bpo.5-common:amd64 (4.19.37-4~bpo9+1, automatic) > Upgrade: linux-compiler-gcc-6-x86:amd64 (4.9.168-1+deb9u3, 4.19.37-4~bpo9+1), > linux-image-amd64:amd64 (4.9+80+deb9u7, 4.19+105~bpo9+1), > linux-headers-amd64:amd64 (4.9+80+deb9u7, 4.19+105~bpo9+1) > End-Date: 2019-07-07 22:51:27 - > ii libapparmor1:amd64 2.11.0-3+deb9u2 amd64 > changehat AppArmor library > ii libnftnl4:amd64 1.0.7-1 amd64 > Netfilter nftables userspace API library > ii linux-image-4.19.0-0.bpo.5-amd64 4.19.37-4~bpo9+1 amd64 > Linux 4.19 for 64-bit PCs (signed) > ii linux-image-4.9.0-8-amd64 4.9.144-3.1 amd64 > Linux 4.9 for 64-bit PCs > ii linux-image-4.9.0-9-amd64 4.9.168-1+deb9u3 amd64 > Linux 4.9 for 64-bit PCs > ii linux-image-amd64 4.19+105~bpo9+1 amd64 > Linux for 64-bit PCs (meta-package) > ii nftables 0.7-1 amd64 > Program to control packet filtering rules by Netfilter project > ii zfs-dkms 0.7.12-1~bpo9+1 all > OpenZFS filesystem kernel modules for Linux > ii zfs-initramfs 0.7.12-1~bpo9+1 all > OpenZFS root filesystem capabilities for Linux - initramfs > ii zfs-zed 0.6.5.9-5 amd64 > OpenZFS Event Daemon I've attached the log with the failed assertion. The system booted just fine, but `nft list ruleset` hangs in uninterruptable sleep. Best regards Tim Düsterhus
Jul 07 22:52:15 buster-test-fresh kernel: Initializing XFRM netlink socket Jul 07 22:52:15 buster-test-fresh kernel: list_del corruption. prev->next should be ffff8b7d2d596260, but was 0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: ------------[ cut here ]------------ Jul 07 22:52:15 buster-test-fresh kernel: kernel BUG at /build/linux-v84tij/linux-4.19.37/lib/list_debug.c:53! Jul 07 22:52:15 buster-test-fresh kernel: invalid opcode: 0000 [#1] SMP PTI Jul 07 22:52:15 buster-test-fresh kernel: CPU: 0 PID: 948 Comm: nft Tainted: P OE 4.19.0-0.bpo.5-amd64 #1 Debian 4.19.37-4~bpo9+1 Jul 07 22:52:15 buster-test-fresh kernel: Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 Jul 07 22:52:15 buster-test-fresh kernel: RIP: 0010:__list_del_entry_valid+0x70/0x90 Jul 07 22:52:15 buster-test-fresh kernel: Code: a4 c8 a9 e8 30 aa d0 ff 0f 0b 48 89 fe 48 c7 c7 90 a4 c8 a9 e8 1f aa d0 ff 0f 0b 48 89 fe 48 c7 c7 c8 a4 c8 a9 e8 0e aa d0 ff <0f> 0b 48 89 fe 48 c7 c7 08 a5 c8 a9 e8 fd a9 d0 ff 0f 0b 90 90 90 Jul 07 22:52:15 buster-test-fresh kernel: RSP: 0018:ffffa1eec70df938 EFLAGS: 00010282 Jul 07 22:52:15 buster-test-fresh kernel: RAX: 0000000000000054 RBX: ffff8b7d2d596260 RCX: 0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: RDX: 0000000000000000 RSI: ffff8b7d3aa166b8 RDI: ffff8b7d3aa166b8 Jul 07 22:52:15 buster-test-fresh kernel: RBP: ffff8b7d2d6d0000 R08: 0000000000000000 R09: 00000000000001d5 Jul 07 22:52:15 buster-test-fresh kernel: R10: ffffa1eec70df990 R11: 0000000000000001 R12: ffffa1eec70df9d0 Jul 07 22:52:15 buster-test-fresh kernel: R13: ffff8b7d2d3985a8 R14: 00000000fffffff5 R15: ffff8b7d2d398000 Jul 07 22:52:15 buster-test-fresh kernel: FS: 00007f074971eb40(0000) GS:ffff8b7d3aa00000(0000) knlGS:0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 07 22:52:15 buster-test-fresh kernel: CR2: 000055f67597b5f0 CR3: 000000006dd1a004 CR4: 00000000003606f0 Jul 07 22:52:15 buster-test-fresh kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Jul 07 22:52:15 buster-test-fresh kernel: Call Trace: Jul 07 22:52:15 buster-test-fresh kernel: nf_tables_unbind_set+0x1a/0xc0 [nf_tables] Jul 07 22:52:15 buster-test-fresh kernel: nf_tables_expr_destroy+0x1a/0x40 [nf_tables] Jul 07 22:52:15 buster-test-fresh kernel: nf_tables_rule_destroy+0x4f/0x80 [nf_tables] Jul 07 22:52:15 buster-test-fresh kernel: nf_tables_newrule+0x4d5/0x8d0 [nf_tables] Jul 07 22:52:15 buster-test-fresh kernel: ? do_wp_page+0x131/0x5f0 Jul 07 22:52:15 buster-test-fresh kernel: nfnetlink_rcv_batch+0x4a2/0x720 [nfnetlink] Jul 07 22:52:15 buster-test-fresh kernel: ? refcount_inc_checked+0x5/0x30 Jul 07 22:52:15 buster-test-fresh kernel: ? apparmor_capable+0x70/0xb0 Jul 07 22:52:15 buster-test-fresh kernel: ? security_capable+0x35/0x50 Jul 07 22:52:15 buster-test-fresh kernel: ? nla_parse+0x32/0x130 Jul 07 22:52:15 buster-test-fresh kernel: nfnetlink_rcv+0x119/0x13c [nfnetlink] Jul 07 22:52:15 buster-test-fresh kernel: netlink_unicast+0x1bc/0x250 Jul 07 22:52:15 buster-test-fresh kernel: netlink_sendmsg+0x2d0/0x3c0 Jul 07 22:52:15 buster-test-fresh kernel: sock_sendmsg+0x36/0x40 Jul 07 22:52:15 buster-test-fresh kernel: ___sys_sendmsg+0x2e9/0x300 Jul 07 22:52:15 buster-test-fresh kernel: ? __alloc_pages_nodemask+0x11d/0x2a0 Jul 07 22:52:15 buster-test-fresh kernel: ? mem_cgroup_commit_charge+0x80/0x520 Jul 07 22:52:15 buster-test-fresh kernel: ? mem_cgroup_try_charge+0xfd/0x1a0 Jul 07 22:52:15 buster-test-fresh kernel: ? refcount_inc_checked+0x5/0x30 Jul 07 22:52:15 buster-test-fresh kernel: ? apparmor_capable+0x70/0xb0 Jul 07 22:52:15 buster-test-fresh kernel: ? security_capable+0x35/0x50 Jul 07 22:52:15 buster-test-fresh kernel: ? release_sock+0x19/0x90 Jul 07 22:52:15 buster-test-fresh kernel: ? __sys_sendmsg+0x63/0xa0 Jul 07 22:52:15 buster-test-fresh kernel: __sys_sendmsg+0x63/0xa0 Jul 07 22:52:15 buster-test-fresh kernel: do_syscall_64+0x55/0x120 Jul 07 22:52:15 buster-test-fresh kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Jul 07 22:52:15 buster-test-fresh kernel: RIP: 0033:0x7f074873cdc7 Jul 07 22:52:15 buster-test-fresh kernel: Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb cd 66 0f 1f 44 00 00 8b 05 4a 49 2b 00 85 c0 75 2e 48 63 ff 48 63 d2 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 a1 f0 2a 00 f7 d8 64 89 02 48 Jul 07 22:52:15 buster-test-fresh kernel: RSP: 002b:00007fffbe9d8168 EFLAGS: 00000246 ORIG_RAX: 000000000000002e Jul 07 22:52:15 buster-test-fresh kernel: RAX: ffffffffffffffda RBX: 00007fffbe9d8180 RCX: 00007f074873cdc7 Jul 07 22:52:15 buster-test-fresh kernel: RDX: 0000000000000000 RSI: 00007fffbe9d91e0 RDI: 0000000000000003 Jul 07 22:52:15 buster-test-fresh kernel: RBP: 00007fffbe9d92e0 R08: 0000000000000004 R09: 0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: R10: 00007fffbe9d91cc R11: 0000000000000246 R12: 00007fffbe9d8180 Jul 07 22:52:15 buster-test-fresh kernel: R13: 00007fffbe9d8190 R14: 0000000000000003 R15: 00007fffbe9d91e0 Jul 07 22:52:15 buster-test-fresh kernel: Modules linked in: nft_limit nft_ct nf_conntrack nf_defrag_ipv6 xfrm_user nf_defrag_ipv4 libcrc32c crc32c_generic xfrm4_tunnel tunnel4 ipcomp nf_tables_set xfrm_ipcomp nf_tables esp4 ah4 nfnetlink af_key xfrm_algo bochs_drm ttm nfit libnvdimm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel drm pcspkr serio_raw evdev virtio_console virtio_balloon joydev sg qemu_fw_cfg button ip_tables x_tables autofs4 hid_generic usbhid hid zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) sd_mod virtio_net net_failover failover virtio_scsi sr_mod cdrom ata_generic crc32c_intel uhci_hcd ehci_hcd ata_piix libata aesni_intel aes_x86_64 crypto_simd cryptd glue_helper usbcore scsi_mod psmouse floppy virtio_pci virtio_ring virtio usb_common i2c_piix4 Jul 07 22:52:15 buster-test-fresh kernel: ---[ end trace 46db01c96d793a42 ]--- Jul 07 22:52:15 buster-test-fresh kernel: RIP: 0010:__list_del_entry_valid+0x70/0x90 Jul 07 22:52:15 buster-test-fresh kernel: Code: a4 c8 a9 e8 30 aa d0 ff 0f 0b 48 89 fe 48 c7 c7 90 a4 c8 a9 e8 1f aa d0 ff 0f 0b 48 89 fe 48 c7 c7 c8 a4 c8 a9 e8 0e aa d0 ff <0f> 0b 48 89 fe 48 c7 c7 08 a5 c8 a9 e8 fd a9 d0 ff 0f 0b 90 90 90 Jul 07 22:52:15 buster-test-fresh kernel: RSP: 0018:ffffa1eec70df938 EFLAGS: 00010282 Jul 07 22:52:15 buster-test-fresh kernel: RAX: 0000000000000054 RBX: ffff8b7d2d596260 RCX: 0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: RDX: 0000000000000000 RSI: ffff8b7d3aa166b8 RDI: ffff8b7d3aa166b8 Jul 07 22:52:15 buster-test-fresh kernel: RBP: ffff8b7d2d6d0000 R08: 0000000000000000 R09: 00000000000001d5 Jul 07 22:52:15 buster-test-fresh kernel: R10: ffffa1eec70df990 R11: 0000000000000001 R12: ffffa1eec70df9d0 Jul 07 22:52:15 buster-test-fresh kernel: R13: ffff8b7d2d3985a8 R14: 00000000fffffff5 R15: ffff8b7d2d398000 Jul 07 22:52:15 buster-test-fresh kernel: FS: 00007f074971eb40(0000) GS:ffff8b7d3aa00000(0000) knlGS:0000000000000000 Jul 07 22:52:15 buster-test-fresh kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 07 22:52:16 buster-test-fresh kernel: CR2: 000055f67597b5f0 CR3: 000000006dd1a004 CR4: 00000000003606f0 Jul 07 22:52:16 buster-test-fresh kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jul 07 22:52:16 buster-test-fresh kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400