I enabled kmemleak and got a bunch of call traces like:
=====================================================================
unreferenced object 0xffff99144b13f600 (size 232):
comm "cat", pid 1100, jiffies 4294739144
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace (crc 8540ec4f):
kmemleak_alloc+0x4a/0x90
kmem_cache_alloc_node+0x2ea/0x390
__alloc_skb+0x174/0x1b0
audit_log_start+0x198/0x3d0
audit_log_proctitle+0x32/0x160
audit_log_exit+0x6c6/0x780
__audit_syscall_exit+0xee/0x140
syscall_exit_work+0x12b/0x150
syscall_exit_to_user_mode_prepare+0x39/0x80
syscall_exit_to_user_mode+0x11/0x260
do_syscall_64+0x8c/0x180
entry_SYSCALL_64_after_hwframe+0x78/0x80
======================================================================
It indicates the skb allocated in audit_log_start is leaking.
Look into audit_log_end, when freeing skb, all skb are dequeued one by one from
ab->skb_list and call __audit_log_end:
=====================================================
void audit_log_end(struct audit_buffer *ab)
{
struct sk_buff *skb;
if (!ab)
return;
while ((skb = skb_dequeue(&ab->skb_list)))
__audit_log_end(skb);
audit_buffer_free(ab);
}
=====================================================
In __audit_log_end:
=====================================================
static void __audit_log_end(struct sk_buff *skb)
{
struct nlmsghdr *nlh;
if (audit_rate_check()) {
/* setup the netlink header, see the comments in
* kauditd_send_multicast_skb() for length quirks */
nlh = nlmsg_hdr(skb);
nlh->nlmsg_len = skb->len - NLMSG_HDRLEN;
/* queue the netlink packet and poke the kauditd thread */
skb_queue_tail(&audit_queue, skb);
wake_up_interruptible(&kauditd_wait);
} else
audit_log_lost("rate limit exceeded");
}
=====================================================
If it doesn’t exceed the rate limit (audit_rate_check), the skb will be added
to audit_queue and handled by the worker.
But if it exceeds the rate limit, nothing happens, then the skb is leak,
because it’s already been dequeued from the list and no one is able to free it.
I checked the upstream lsm stacking, the latest version still has this issue:
https://github.com/cschaufler/lsm-stacking/blob/audit-6.17-rc1/kernel/audit.c#L2631
It should check rate limit first, if it doesn’t exceed the limit, then
dequeue the skb from the list, if it exceeds the limit, it should go to
audit_buffer_free which will free all skb in ab->skb_list
I’ve modified the code to check if it exceeds the rate limit before dequeuing
the first skb from the skb_list and build a test kernel here:
https://launchpad.net/~gerald-yang-tw/+archive/ubuntu/audit-memleak
This is not ideal, because in audit_rate_check, it only considers adding one
skb at a time and checks if it exceeds audit_rate_limit.
We should take all skb in skb_list into account instead of just one skb,
otherwise it could exceed the rate limit a little bit.
But I think this is where the slab leak happens, I can not reproduce the issue
with the above test kernel.
I'm checking if there is a better way to fix it and send a patch upstream.
If you would like to help run some tests, please use the above test
kernel and see if it fixes the slab leak, thanks!
To install the test kernel, please add the above test PPA and install the
following test packages:
linux-image-unsigned-6.8.0-81-generic
linux-modules-6.8.0-81-generic
linux-modules-extra-6.8.0-81-generic
Regards,
Gerald
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2098730
Title:
Kernel 6.8.0 memory leak
Status in linux package in Ubuntu:
Confirmed
Status in linux source package in Noble:
Confirmed
Status in linux source package in Oracular:
Won't Fix
Status in linux source package in Plucky:
Confirmed
Bug description:
We discovered a kernel memory leak in the Ubuntu 24.04 distribution
using the 6.8.0 series kernel while testing on our cluster.
After performing a fresh installation of Ubuntu Server 24.04 from the
official ISO: `https://ubuntu.com/download/server/thank-
you?version=24.04.1&architecture=amd64<s=true`, with the built-in
kernel `linux-image-6.8.0-51-generic`, we observed that Slab consumes
all available memory due to the kernel threads `kmalloc-rnd-(*)-2k`
and `skbuff_head_cache` when the audit subsystem is enabled. The value
of `SUnreclaim` in `/proc/meminfo` nearly reaches the system's total
memory.
We validated this bug on both amd64 and aarch64 architectures,
affecting bare metal and virtual machines alike across all platforms.
Here’s the stack information obtained using the eBPF tool `memleak`:
"https://github.com/iovisor/bcc/blob/master/tools/memleak.py"; to
detect the kernel memory leak:
195362816 bytes in 5962 allocations from stack
0xffffffff96652314 __alloc_pages+0x264 [kernel]
0xffffffff96652314 __alloc_pages+0x264 [kernel]
0xffffffff9665b0a8 allocate_slab+0xa8 [kernel]
0xffffffff9665b3b8 new_slab+0x38 [kernel]
0xffffffff9665e3d5 ___slab_alloc+0x435 [kernel]
0xffffffff9665f62b __kmalloc_node_track_caller+0x18b [kernel]
0xffffffff970d8317 kmalloc_reserve+0x67 [kernel]
0xffffffff970db4aa __alloc_skb+0x8a [kernel]
0xffffffff96457f98 audit_log_start+0x198 [kernel]
0xffffffff96462103 audit_log_exit+0x433 [kernel]
0xffffffff96462dbe __audit_syscall_exit+0xee [kernel]
0xffffffff963f111b syscall_exit_work+0x12b [kernel]
0xffffffff963f1189 syscall_exit_to_user_mode_prepare+0x39
[kernel]
0xffffffff97430ae1 syscall_exit_to_user_mode+0x11 [kernel]
0xffffffff97428fec do_syscall_64+0x8c [kernel]
0xffffffff97600130 entry_SYSCALL_64_after_hwframe+0x78 [kernel]
To reproduce this memory leak issue, follow these steps:
1. Install and start auditd.
2. Add audit rules in `/etc/audit/rules.d/audit.rules`:
```
-D
-b 8192
-f 1
-r 100
-a always,exit -F arch=b64 -S openat -S truncate -S ftruncate -F
exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
```
3. Reboot or run `augenrules --load`, then execute `auditctl -l` to verify
that the audit rule above has been loaded.
4. Run command `while :; do cat /proc/1/environ; done` as a normal user (uid
>= 1000) to triger the audit events.
5. Monitor kernel memory allocation by running either:
```
watch -d -n 1 'cat /proc/meminfo | grep -i SUnreclaim'
```
or simply use
```
slabtop -s c
```
We tested several kernels within the 6.8.0 series, and this bug was
present in all of them, including when installing HWE Kernel 6.8.0 in
Ubuntu 22.04 via `apt install linux-generic-hwe-22.04`. However, after
installing mainline kernels v6.8.1 or higher from
https://kernel.ubunt.com/mainline/, this bug disappears, indicating it
may have been fixed on upstream.
Therefore, it is essential to update your stable repository's kernel and
refresh your ISO accordingly.
---
ProblemType: Bug
ApportVersion: 2.28.1-0ubuntu3.3
Architecture: arm64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer',
'/dev/snd/seq', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D0c', '/dev/snd/hwC0D0',
'/dev/snd/controlC0', '/dev/snd/by-path'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 24.04
InstallationDate: Installed on 2024-08-13 (191 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release arm64
(20240423)
IwConfig:
lo no wireless extensions.
enp0s5 no wireless extensions.
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 203a:fffc PARALLELS Virtual Mouse
Bus 003 Device 003: ID 203a:fffb PARALLELS Virtual Keyboard
MachineType: Parallels International GmbH. Parallels ARM Virtual Machine
NonfreeKernelModules: prl_fs_freeze prl_tg
Package: linux (not installed)
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=tmux-256color
ProcFB: 0 virtio_gpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.8.0-53-generic
root=UUID=0c00eea2-089b-4002-90e1-ca43db4a623e ro
ProcVersionSignature: Ubuntu 6.8.0-53.55-generic 6.8.12
RelatedPackageVersions:
linux-restricted-modules-6.8.0-53-generic N/A
linux-backports-modules-6.8.0-53-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.9
RfKill:
Tags: noble
Uname: Linux 6.8.0-53-generic aarch64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: False
dmi.bios.date: Mon, 23 Dec 2024 15:54:53
dmi.bios.release: 0.1
dmi.bios.vendor: Parallels International GmbH.
dmi.bios.version: 20.2.0 (55872)
dmi.board.asset.tag: None
dmi.board.name: Parallels ARM Virtual Platform
dmi.board.vendor: Parallels ARM Virtual Machine
dmi.board.version: 0.1
dmi.chassis.type: 2
dmi.chassis.vendor: Parallels International GmbH.
dmi.modalias:
dmi:bvnParallelsInternationalGmbH.:bvr20.2.0(55872):bdMon,23Dec2024155453:br0.1:svnParallelsInternationalGmbH.:pnParallelsARMVirtualMachine:pvr0.1:rvnParallelsARMVirtualMachine:rnParallelsARMVirtualPlatform:rvr0.1:cvnParallelsInternationalGmbH.:ct2:cvr:skuParallels_ARM_VM:
dmi.product.family: Parallels VM
dmi.product.name: Parallels ARM Virtual Machine
dmi.product.sku: Parallels_ARM_VM
dmi.product.version: 0.1
dmi.sys.vendor: Parallels International GmbH.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098730/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
