Performing verification for Jammy.

I started a fresh Jammy VM, and installed 5.15.0-67-generic from
-updates.

I appended apparmor=0 to GRUB_CMDLINE_LINUX_DEFAULT, updated grub, and
rebooted.

>From there, I installed auditd, and stress-ng. I edited
/etc/audit/rules.d/audit.rules to include:

-a exit,always -F arch=b64 -S execve
-a exit,always -F arch=b32 -S execve

I then rebooted, and started stress-ng.

I checked the kmalloc-2k slab with:

$ watch "sudo cat /proc/meminfo | grep SUnreclaim"
$ watch "sudo cat /proc/slabinfo | grep kmalloc-2k"
$ sudo slabtop

Now, since this bug has been worked around by:

39cce16cfeed UBUNTU: SAUCE: LSM: Change Landlock from LSMBLOB_NEEDED to
LSMBLOB_NOT_NEEDED

in 5.15.0-53-generic, the kmalloc-k slab did not uncontrollably increase, but
that is okay, it is just the leak isn't reachable with landlock not using the
task_getsecid_obj hook.

This means this verification is more of a smoke test than checking root
cause.

I then enabled -proposed, and installed 5.15.0-68-generic, and rebooted.

I again ran stress-ng and checked the kmalloc-2k slab, and all was well, there
was no memory leak.

The core issue was fixed, but again, masked by the previous fix in 
5.15.0-53-generic. 

There was no smoke, and things ran as expected. Marking verified for
Jammy.

** Tags removed: verification-needed-jammy
** Tags added: verification-done-jammy

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1987430

Title:
  Ubuntu 22.04 kernel 5.15.0-46-generic leaks kernel memory in
  kmalloc-2k slabs

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Jammy:
  Fix Committed
Status in linux source package in Kinetic:
  Fix Released

Bug description:
  Since updating to kernel 5.15.0-46-generic (package version
  5.15.0-46.49), all of our Ubuntu 22.04 LTS servers are leaking kernel
  memory; our first server with 8 GB of RAM just fatally OOMed, causing
  us to detect this. Inspection of OOM reports, /proc/meminfo, and
  /proc/slabinfo says that it's mostly going to unreclaimable kmalloc-2k
  slabs:

          Aug 23 12:51:11 cluster kernel: [361299.864757] Unreclaimable slab 
info:
          Aug 23 12:51:11 cluster kernel: [361299.864757] Name                  
    Used          Total
          [...]
          Aug 23 12:51:11 cluster kernel: [361299.864924] kmalloc-2k           
6676584KB    6676596KB

  Most of our machines appear to be leaking slab memory at a rate of
  around 20 to 40 Mbytes/hour, with some machines leaking much faster;
  the champions are leaking kernel memory at 145 Mbytes/hour and 237
  Mbytes/hour.

  We aren't running any proprietary kernel modules and our only unusual
  kernel configuration is that we've disabled AppArmor with 'apparmor=0'
  on the kernel command line.

  /proc/version_signature:
  Ubuntu 5.15.0-46.49-generic 5.15.39

  Full kernel command line from the Dell R240 system that fatally OOMd:
  BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic 
root=UUID=3165564f-a2dd-4b39-935b-114f3e23ff54 ro console=ttyS0,115200 
console=tty0 apparmor=0

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987430/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to