It appears that the combination of our audit rules being enabled and apparmor=0 is what triggers the leak. My test case is repeatedly compiling Go from source and running its self tests (cloning https://go.googlesource.com/go, then 'cd go/src; while true; do ./all.bash; done'). On a VM configured as one of our machines (with audit rules), this leaks visibly. On a basically stock 22.04 VM (and thus with no audit rules), this doesn't leak. If I disable auditd on our configuration, it stops leaking. On the stock configuration, if I install auditd and our rules, and enable auditd (and reboot), it immediately starts leaking with rapid growth in kmalloc-2k. Taken from slabtop on the stock VM + auditd:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 81360 81355 99% 2.00K 5085 16 162720K kmalloc-2k 44010 39386 89% 1.15K 1630 27 52160K ext4_inode_cache [...] This is on a VM that's been up only 24 minutes so far; this is the top slab entry, far ahead of the second placed one I've also shown. And just in the process of writing this comment, it's grown to 206080K. Stopping auditd on the our-setup VM seems to stop further kmalloc-2k slab growth but doesn't reduce the current size. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987430 Title: Ubuntu 22.04 kernel 5.15.0-46-generic leaks kernel memory in kmalloc-2k slabs Status in linux package in Ubuntu: Confirmed Bug description: Since updating to kernel 5.15.0-46-generic (package version 5.15.0-46.49), all of our Ubuntu 22.04 LTS servers are leaking kernel memory; our first server with 8 GB of RAM just fatally OOMed, causing us to detect this. Inspection of OOM reports, /proc/meminfo, and /proc/slabinfo says that it's mostly going to unreclaimable kmalloc-2k slabs: Aug 23 12:51:11 cluster kernel: [361299.864757] Unreclaimable slab info: Aug 23 12:51:11 cluster kernel: [361299.864757] Name Used Total [...] Aug 23 12:51:11 cluster kernel: [361299.864924] kmalloc-2k 6676584KB 6676596KB Most of our machines appear to be leaking slab memory at a rate of around 20 to 40 Mbytes/hour, with some machines leaking much faster; the champions are leaking kernel memory at 145 Mbytes/hour and 237 Mbytes/hour. We aren't running any proprietary kernel modules and our only unusual kernel configuration is that we've disabled AppArmor with 'apparmor=0' on the kernel command line. /proc/version_signature: Ubuntu 5.15.0-46.49-generic 5.15.39 Full kernel command line from the Dell R240 system that fatally OOMd: BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic root=UUID=3165564f-a2dd-4b39-935b-114f3e23ff54 ro console=ttyS0,115200 console=tty0 apparmor=0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987430/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
