Package: linux-image Version: 6.17+ Severity: important Kernel oops following NULL pointer dereference in aa_file_perm() when running containers with podman + crun under Incus, triggered during UNIX socket file-descriptor passing (SCM_RIGHTS).
This appears to be an AppArmor regression somewhere around 6.17, and seems likely related to AppArmor AF_UNIX mediation and refactoring ? https://git.kernel.org/pub/scm/linux/kernel/git/jj/linux-apparmor.git/commit/?h=apparmor-next&id=c05e705812d179f4b85aeacc34a555a42bc4f9ac Confirmed not working: 6.17.8+deb13, 6.18.4 (non debian kernel) Confirmed working: 6.16.12+deb13 Steps I used to reproduce: - Starting with a clean Debian 13/Trixie install (VM or bare metal) running kernel version >= ~6.17 - Install Incus (latest 6.20 for reference) - Create a non-privileged debian 13 container under incus with 'security.nesting=true' enabled - Install podman in to the incus container (from debian distribution v5.4.2 / apt get podman) - Attempt to run a rootful woodpecker-ci pod: # podman run --rm -v /run/podman/podman.sock:/var/run/docker.sock -e 'WOODPECKER_SERVER=xxxxx' -e 'WOODPECKER_AGENT_SECRET=xxxx'-p 3000:3000 docker.io/woodpeckerci/woodpecker-agent:v3 Key points that trigger the issue: - Podman is running nested inside a non-privileged container - The podman container bind mounts the /run/podman/podman.sock UNIX socket (this is within the incus container) - Accessing the podman UNIX socket from within the nested podman container is what triggers the oops What does work: - Podman on its own without nesting works fine - Using crun instead of runc (I understand crun makes more use of FD passing which is what appears to trigger the issue) - Kernels earlier than 6.17 Full trace below Jan 16 11:06:59 incus-podman kernel: BUG: kernel NULL pointer dereference, address: 0000000000000018 Jan 16 11:06:59 incus-podman kernel: #PF: supervisor read access in kernel mode Jan 16 11:06:59 incus-podman kernel: #PF: error_code(0x0000) - not-present page Jan 16 11:06:59 incus-podman kernel: PGD 0 P4D 0 Jan 16 11:06:59 incus-podman kernel: Oops: Oops: 0000 [#1] SMP PTI Jan 16 11:06:59 incus-podman kernel: CPU: 1 UID: 1000000 PID: 981 Comm: crun Not tainted 6.18.4-zabbly+ #debian13 PREEMPT(voluntary) Jan 16 11:06:59 incus-podman kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/Incus, BIOS unknown 02/02/2022 Jan 16 11:06:59 incus-podman kernel: RIP: 0010:aa_file_perm+0xc0/0x5d0 Jan 16 11:06:59 incus-podman kernel: Code: 45 31 c9 c3 cc cc cc cc 49 8b 46 20 41 8b 57 10 0f b7 00 66 25 00 f0 66 3d 00 c0 75 1c 41 f7 c4 46 00 10 00 75 13 49 8b 46 18 <48> 8b 40 18 66 83 78 10 01 0f 84 d9 02 00 00 89 d0 f7 d0 44 21 e0 Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efb5f0 EFLAGS: 00010246 Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX: ffff898294ff8180 RCX: ffff898283610b40 Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI: ffff898282ae13c0 RDI: ffffffffa88e8430 Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efb6a0 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R13: ffff898294ff8180 R14: ffff898283610b40 R15: ffff898282e6d3d0 Jan 16 11:06:59 incus-podman kernel: FS: 00007f3616418840(0000) GS:ffff898340c3c000(0000) knlGS:0000000000000000 Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018 CR3: 0000000103626002 CR4: 0000000000372ef0 Jan 16 11:06:59 incus-podman kernel: Call Trace: Jan 16 11:06:59 incus-podman kernel: <TASK> Jan 16 11:06:59 incus-podman kernel: ? __slab_free+0xdf/0x2c0 Jan 16 11:06:59 incus-podman kernel: common_file_perm+0x69/0x1b0 Jan 16 11:06:59 incus-podman kernel: apparmor_file_receive+0x42/0x80 Jan 16 11:06:59 incus-podman kernel: security_file_receive+0x4a/0x120 Jan 16 11:06:59 incus-podman kernel: receive_fd+0x1d/0xf0 Jan 16 11:06:59 incus-podman kernel: scm_detach_fds+0xad/0x1c0 Jan 16 11:06:59 incus-podman kernel: __scm_recv_common.isra.0+0x66/0x180 Jan 16 11:06:59 incus-podman kernel: scm_recv_unix+0x30/0x130 Jan 16 11:06:59 incus-podman kernel: ? unix_destroy_fpl+0x3a/0xa0 Jan 16 11:06:59 incus-podman kernel: __unix_dgram_recvmsg+0x2ac/0x450 Jan 16 11:06:59 incus-podman kernel: unix_seqpacket_recvmsg+0x43/0x70 Jan 16 11:06:59 incus-podman kernel: sock_recvmsg+0xe1/0xf0 Jan 16 11:06:59 incus-podman kernel: ____sys_recvmsg+0xa0/0x230 Jan 16 11:06:59 incus-podman kernel: ___sys_recvmsg+0xc7/0xf0 Jan 16 11:06:59 incus-podman kernel: __sys_recvmsg+0x89/0x100 Jan 16 11:06:59 incus-podman kernel: __x64_sys_recvmsg+0x1d/0x30 Jan 16 11:06:59 incus-podman kernel: x64_sys_call+0x840/0x2350 Jan 16 11:06:59 incus-podman kernel: do_syscall_64+0x80/0x590 Jan 16 11:06:59 incus-podman kernel: ? ___sys_recvmsg+0xd2/0xf0 Jan 16 11:06:59 incus-podman kernel: ? ____sys_recvmsg+0x10e/0x230 Jan 16 11:06:59 incus-podman kernel: ? __sys_recvmsg+0x89/0x100 Jan 16 11:06:59 incus-podman kernel: ? __x64_sys_recvmsg+0x1d/0x30 Jan 16 11:06:59 incus-podman kernel: ? x64_sys_call+0x840/0x2350 Jan 16 11:06:59 incus-podman kernel: ? do_syscall_64+0xb8/0x590 Jan 16 11:06:59 incus-podman kernel: ? __sys_recvmsg+0x89/0x100 Jan 16 11:06:59 incus-podman kernel: ? __x64_sys_recvmsg+0x1d/0x30 Jan 16 11:06:59 incus-podman kernel: ? x64_sys_call+0x840/0x2350 Jan 16 11:06:59 incus-podman kernel: ? do_syscall_64+0xb8/0x590 Jan 16 11:06:59 incus-podman kernel: ? irqentry_exit_to_user_mode+0x2e/0x2a0 Jan 16 11:06:59 incus-podman kernel: ? irqentry_exit+0x43/0x50 Jan 16 11:06:59 incus-podman kernel: ? clear_bhb_loop+0x50/0xa0 Jan 16 11:06:59 incus-podman kernel: ? clear_bhb_loop+0x50/0xa0 Jan 16 11:06:59 incus-podman kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e Jan 16 11:06:59 incus-podman kernel: RIP: 0033:0x7f361659c687 Jan 16 11:06:59 incus-podman kernel: Code: 48 89 fa 4c 89 df e8 58 b3 00 00 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 1a 5b c3 0f 1f 84 00 00 00 00 00 48 8b 44 24 10 0f 05 <5b> c3 0f 1f 80 00 00 00 00 83 e2 39 83 fa 08 75 de e8 23 ff ff ff Jan 16 11:06:59 incus-podman kernel: RSP: 002b:00007fff89de51f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002f Jan 16 11:06:59 incus-podman kernel: RAX: ffffffffffffffda RBX: 00007f3616418840 RCX: 00007f361659c687 Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI: 00007fff89de5240 RDI: 0000000000000009 Jan 16 11:06:59 incus-podman kernel: RBP: 00007fff89de5240 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007fff89de58c0 Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000007 R14: 00007fff89de58c0 R15: 000000000000000c Jan 16 11:06:59 incus-podman kernel: </TASK> Jan 16 11:06:59 incus-podman kernel: Modules linked in: nft_nat nft_ct nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib overlay veth nft_masq nft_chain_nat nf_nat nf_conntrack n f_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables vhost_vsock vhost vhost_iotlb binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency_common in tel_pmc_core pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel virtio_snd rapl snd_pcsp virtio_gpu snd_pcm virtio_dma_buf drm_shmem_helper vmw_vsock_virtio_transport drm_client_lib 9p snd_timer vmw_vsock_virtio_transport_common 9pnet_virtio drm_kms_help er 9pnet snd netfs vsock virtio_input soundcore vmgenid joydev input_leds mac_hid cfg80211 sch_fq_codel efi_pstore drm nfnetlink dmi_sysfs qemu_fw_cfg virtio_rng ip_tables x_t ables autofs4 iTCO_wdt intel_pmc_bxt iTCO_vendor_support psmouse i2c_i801 i2c_mux serio_raw i2c_smbus ahci libahci lpc_ich Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018 Jan 16 11:06:59 incus-podman kernel: ---[ end trace 0000000000000000 ]--- Jan 16 11:06:59 incus-podman kernel: RIP: 0010:aa_file_perm+0xc0/0x5d0 Jan 16 11:06:59 incus-podman kernel: Code: 45 31 c9 c3 cc cc cc cc 49 8b 46 20 41 8b 57 10 0f b7 00 66 25 00 f0 66 3d 00 c0 75 1c 41 f7 c4 46 00 10 00 75 13 49 8b 46 18 <48> 8 b 40 18 66 83 78 10 01 0f 84 d9 02 00 00 89 d0 f7 d0 44 21 e0 Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efb5f0 EFLAGS: 00010246 Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX: ffff898294ff8180 RCX: ffff898283610b40 Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI: ffff898282ae13c0 RDI: ffffffffa88e8430 Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efb6a0 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R13: ffff898294ff8180 R14: ffff898283610b40 R15: ffff898282e6d3d0 Jan 16 11:06:59 incus-podman kernel: FS: 00007f3616418840(0000) GS:ffff898340c3c000(0000) knlGS:0000000000000000 Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 16 11:06:59 incus-podman kernel: CR2: 0000000000000018 CR3: 0000000103626002 CR4: 0000000000372ef0 Jan 16 11:06:59 incus-podman kernel: note: crun[981] exited with irqs disabled Jan 16 11:06:59 incus-podman kernel: ------------[ cut here ]------------ Jan 16 11:06:59 incus-podman kernel: Voluntary context switch within RCU read-side critical section! Jan 16 11:06:59 incus-podman kernel: WARNING: CPU: 0 PID: 981 at kernel/rcu/tree_plugin.h:332 rcu_note_context_switch+0x523/0x590 Jan 16 11:06:59 incus-podman kernel: Modules linked in: nft_nat nft_ct nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib overlay veth nft_masq nft_chain_nat nf_nat nf_conntrack n f_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables vhost_vsock vhost vhost_iotlb binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency_common in tel_pmc_core pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel virtio_snd rapl snd_pcsp virtio_gpu snd_pcm virtio_dma_buf drm_shmem_helper vmw_vsock_virtio_transport drm_client_lib 9p snd_timer vmw_vsock_virtio_transport_common 9pnet_virtio drm_kms_help er 9pnet snd netfs vsock virtio_input soundcore vmgenid joydev input_leds mac_hid cfg80211 sch_fq_codel efi_pstore drm nfnetlink dmi_sysfs qemu_fw_cfg virtio_rng ip_tables x_t ables autofs4 iTCO_wdt intel_pmc_bxt iTCO_vendor_support psmouse i2c_i801 i2c_mux serio_raw i2c_smbus ahci libahci lpc_ich Jan 16 11:06:59 incus-podman kernel: CPU: 0 UID: 1000000 PID: 981 Comm: crun Tainted: G D 6.18.4-zabbly+ #debian13 PREEMPT(voluntary) Jan 16 11:06:59 incus-podman kernel: Tainted: [D]=DIE Jan 16 11:06:59 incus-podman kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/Incus, BIOS unknown 02/02/2022 Jan 16 11:06:59 incus-podman kernel: RIP: 0010:rcu_note_context_switch+0x523/0x590 Jan 16 11:06:59 incus-podman kernel: Code: ff 49 89 96 a8 00 00 00 e9 35 fd ff ff 45 85 ff 75 ef e9 2b fd ff ff 48 c7 c7 f0 db 7f a8 c6 05 25 4a 2c 02 01 e8 9d 36 f2 ff <0f> 0 b e9 23 fb ff ff 4d 8b 74 24 20 4c 89 f7 e8 09 97 fa 00 41 c6 Jan 16 11:06:59 incus-podman kernel: RSP: 0018:ffffcc4900efbc50 EFLAGS: 00010046 Jan 16 11:06:59 incus-podman kernel: RAX: 0000000000000000 RBX: ffff898295dc9b80 RCX: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: RBP: ffffcc4900efbc78 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8982ea633600 Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000000 R14: ffffcc4900efbe00 R15: ffff898295dca980 Jan 16 11:06:59 incus-podman kernel: FS: 0000000000000000(0000) GS:ffff898340bfc000(0000) knlGS:0000000000000000 Jan 16 11:06:59 incus-podman kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 16 11:06:59 incus-podman kernel: CR2: 00007f2eb92ec4a8 CR3: 000000005e234006 CR4: 0000000000372ef0 Jan 16 11:06:59 incus-podman kernel: Call Trace: Jan 16 11:06:59 incus-podman kernel: <TASK> Jan 16 11:06:59 incus-podman kernel: __schedule+0xc6/0x1310 Jan 16 11:06:59 incus-podman kernel: ? try_to_wake_up+0x392/0x8a0 Jan 16 11:06:59 incus-podman kernel: ? kthread_insert_work+0xb8/0xe0 Jan 16 11:06:59 incus-podman kernel: schedule+0x27/0xf0 Jan 16 11:06:59 incus-podman kernel: synchronize_rcu_expedited+0x1c2/0x220 Jan 16 11:06:59 incus-podman kernel: ? __pfx_autoremove_wake_function+0x10/0x10 Jan 16 11:06:59 incus-podman kernel: ? __pfx_wait_rcu_exp_gp+0x10/0x10 Jan 16 11:06:59 incus-podman kernel: namespace_unlock+0x295/0x380 Jan 16 11:06:59 incus-podman kernel: put_mnt_ns+0x79/0xb0 Jan 16 11:06:59 incus-podman kernel: free_nsproxy+0x16/0x190 Jan 16 11:06:59 incus-podman kernel: switch_task_namespaces+0x74/0xa0 Jan 16 11:06:59 incus-podman kernel: exit_task_namespaces+0x10/0x20 Jan 16 11:06:59 incus-podman kernel: do_exit+0x2a5/0xa20 Jan 16 11:06:59 incus-podman kernel: make_task_dead+0x93/0xa0 Jan 16 11:06:59 incus-podman kernel: rewind_stack_and_make_dead+0x16/0x20 Jan 16 11:06:59 incus-podman kernel: RIP: 0033:0x7f361659c687 Jan 16 11:06:59 incus-podman kernel: Code: Unable to access opcode bytes at 0x7f361659c65d. Jan 16 11:06:59 incus-podman kernel: RSP: 002b:00007fff89de51f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002f Jan 16 11:06:59 incus-podman kernel: RAX: ffffffffffffffda RBX: 00007f3616418840 RCX: 00007f361659c687 Jan 16 11:06:59 incus-podman kernel: RDX: 0000000000000000 RSI: 00007fff89de5240 RDI: 0000000000000009 Jan 16 11:06:59 incus-podman kernel: RBP: 00007fff89de5240 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:06:59 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007fff89de58c0 Jan 16 11:06:59 incus-podman kernel: R13: 0000000000000007 R14: 00007fff89de58c0 R15: 000000000000000c Jan 16 11:06:59 incus-podman kernel: </TASK> Jan 16 11:06:59 incus-podman kernel: ---[ end trace 0000000000000000 ]--- Jan 16 11:07:58 incus-podman kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Jan 16 11:07:58 incus-podman kernel: rcu: Tasks blocked on level-0 rcu_node (CPUs 0-7): P981/1:b..l Jan 16 11:07:58 incus-podman kernel: rcu: (detected by 0, t=60013 jiffies, g=8493, q=965 ncpus=2) Jan 16 11:07:58 incus-podman kernel: task:crun state:D stack:0 pid:981 tgid:981 ppid:980 task_flags:0x40014c flags:0x00080001 Jan 16 11:07:58 incus-podman kernel: Call Trace: Jan 16 11:07:58 incus-podman kernel: <TASK> Jan 16 11:07:58 incus-podman kernel: __schedule+0x468/0x1310 Jan 16 11:07:58 incus-podman kernel: ? try_to_wake_up+0x392/0x8a0 Jan 16 11:07:58 incus-podman kernel: schedule+0x27/0xf0 Jan 16 11:07:58 incus-podman kernel: synchronize_rcu_expedited+0x1c2/0x220 Jan 16 11:07:58 incus-podman kernel: ? __pfx_autoremove_wake_function+0x10/0x10 Jan 16 11:07:58 incus-podman kernel: ? __pfx_wait_rcu_exp_gp+0x10/0x10 Jan 16 11:07:58 incus-podman kernel: namespace_unlock+0x295/0x380 Jan 16 11:07:58 incus-podman kernel: put_mnt_ns+0x79/0xb0 Jan 16 11:07:58 incus-podman kernel: free_nsproxy+0x16/0x190 Jan 16 11:07:58 incus-podman kernel: switch_task_namespaces+0x74/0xa0 Jan 16 11:07:58 incus-podman kernel: exit_task_namespaces+0x10/0x20 Jan 16 11:07:58 incus-podman kernel: do_exit+0x2a5/0xa20 Jan 16 11:07:58 incus-podman kernel: make_task_dead+0x93/0xa0 Jan 16 11:07:58 incus-podman kernel: rewind_stack_and_make_dead+0x16/0x20 Jan 16 11:07:58 incus-podman kernel: RIP: 0033:0x7f361659c687 Jan 16 11:07:58 incus-podman kernel: RSP: 002b:00007fff89de51f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002f Jan 16 11:07:58 incus-podman kernel: RAX: ffffffffffffffda RBX: 00007f3616418840 RCX: 00007f361659c687 Jan 16 11:07:58 incus-podman kernel: RDX: 0000000000000000 RSI: 00007fff89de5240 RDI: 0000000000000009 Jan 16 11:07:58 incus-podman kernel: RBP: 00007fff89de5240 R08: 0000000000000000 R09: 0000000000000000 Jan 16 11:07:58 incus-podman kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007fff89de58c0 Jan 16 11:07:58 incus-podman kernel: R13: 0000000000000007 R14: 00007fff89de58c0 R15: 000000000000000c Jan 16 11:07:58 incus-podman kernel: </TASK> Jan 16 11:08:00 incus-podman kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P981 } 61834 jiffies s: 873 root: 0x0/T Jan 16 11:08:00 incus-podman kernel: rcu: blocking rcu_node structures (internal RCU debug):
