** Summary changed: - Memory leak in 6.8.0-87-generic when using a cifs mount + cifs: Fix memory leak of a folio every call to cifs_writepages_begin()
** Description changed: + BugLink: https://bugs.launchpad.net/bugs/2131213 + [Impact] - In a 24.04 VM running kernel 6.8.0-87-generic, a memory leak is experienced when performing I/O in a cifs mountpoint. Free memory keeps decreasing until reaching a very low value. - This behavior is not seen in the latest 24.04 HWE kernel - 6.14.0-35-generic. + The fix for CVE-2024-27036 introduced a slow memory leak to the cifs + subsystem: - [Test case] - 1) Create 2 multipass Ubuntu 24.04 VMs, one acting as a samba server, and the other acting as a samba client. + commit f3dc1bdb6b0b0693562c7c54a6c28bafa608ba3c + Author: David Howells <[email protected]> + Date: Thu Feb 22 11:20:26 2024 +0000 + Subject: cifs: Fix writeback data corruption + Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f3dc1bdb6b0b0693562c7c54a6c28bafa608ba3c - 2) From the client, mount the samba share with default settings: - sudo mount -t cifs //<IP>/shared /mnt/cifs-share + The changes made to folios caused folios used in cifs_writepages_begin() to never + folio_put() the references, leading to a leak of a folio per call to + cifs_writepages_begin(). This presents itself as a slow leak, slowly leaking + memory on each cifs operation until the system is exhausted of memory and hangs, + which can be from several hours to several days. - 3) From the client, create a small file in /mnt/cifs-share: - fallocate -l 1k d1.data + This greatly impacts system stability, and increases the risk of outages and + data corruption when the system eventually runs out of memory. - 4) From the client, keep zipping the file in an endless loop: - while true ; do zstd --force d1.data ; done + There is no workarounds. - 5) From the client, watch the free memory decrease to a very low value + [Fix] + + This bug does not actually affect mainline Linux, due to the cifs subsystem + undergoing a major refactor in 6.10 to switch to using netfslib to manage folios. + + This makes this bug only present in stable trees. It was fixed in + 6.6.107 by: + + commit 5de7b4141af107c184f4f86594ea0763aecd764e + From: Yang Erkun <[email protected]> + Date: Fri, 12 Sep 2025 09:41:50 +0800 + Subject: cifs: fix pagecache leak when do writepages + Link: https://github.com/gregkh/linux/commit/5de7b4141af107c184f4f86594ea0763aecd764e + + This has already been applied to -azure since 6.8.0-1040.46, but this is needed + in -generic. + + [Testcase] + + 1) Create a fresh VM for the server, using noble. + 2) sudo apt update + 3) sudo apt upgrade + 4) sudo hostnamectl set-hostname samba-dc + 5) sudo vim /etc/hosts + Add an entry with its IP address, e.g.: + 192.168.122.124 samba-dc samba-dc.example.com + 6) sudo apt install -y samba smbclient winbind libpam-winbind libnss-winbind krb5-kdc libpam-krb5 cifs-utils + Note: skip config of kerberos KDC. + 7) sudo rm /etc/krb5.conf + 8) sudo rm /etc/samba/smb.conf + 9) sudo samba-tool domain provision --server-role=dc --use-rfc2307 --dns-backend=SAMBA_INTERNAL --realm=samba-dc.EXAMPLE.COM --domain=SAMBA --adminpass=Password1 + 10) sudo cp /var/lib/samba/private/krb5.conf /etc/krb5.conf + 11) sudo systemctl mask smbd nmbd winbind + 12) sudo systemctl disable smbd nmbd winbind + 13) sudo systemctl stop smbd nmbd winbind + 14) sudo systemctl unmask samba-ad-dc + 15) sudo systemctl start samba-ad-dc + 16) sudo systemctl enable samba-ad-dc + 17) sudo reboot + 18) sudo systemctl stop systemd-resolved + 19) sudo systemctl disable systemd-resolved + 20) cat << EOF >> /etc/resolv.conf + nameserver 192.168.122.124 + search SAMBA + EOF + sudo vim /etc/samba/smb.conf + Change forwarder to 8.8.8.8 + 21) sudo reboot + 22) host -t SRV _ldap._tcp.samba-dc.example.com + _ldap._tcp.samba-dc.example.com has SRV record 0 100 389 samba-dc.samba-dc.example.com. + 23) $ smbclient -L localhost -N + Anonymous login successful + + Sharename Type Comment + --------- ---- ------- + sysvol Disk + netlogon Disk + IPC$ IPC IPC Service (Samba 4.13.17-Ubuntu) + SMB1 disabled -- no workgroup available + 24) $ smbclient //localhost/netlogon -UAdministrator -c 'ls' + Enter SAMBA\Administrator's password: + . D 0 Mon Feb 28 04:23:22 2022 + .. D 0 Mon Feb 28 04:23:27 2022 + + 9983232 blocks of size 1024. 7995324 blocks available + 25) kinit administrator + Password for [email protected]: + Warning: Your password will expire in 41 days on Wed May 21 02:51:02 2025 + 26) klist + Ticket cache: FILE:/tmp/krb5cc_1000 + Default principal: [email protected] + + Valid starting Expires Service principal + 04/09/25 02:53:27 04/09/25 12:53:27 krbtgt/[email protected] + renew until 04/10/25 02:53:22 + 27) Create a share: + 28) sudo mkdir -p /srv/samba/Demo/ + 29) sudo vim /etc/samba/smb.conf + [Demo] + path = /srv/samba/Demo/ + read only = no + 30) sudo chmod 0770 /srv/samba/Demo/ + 31) smbclient -U Administrator //samba-dc.example.com/demo + Password for [SAMBA\Administrator]: + Try "help" to get a list of possible commands. + smb: \> + 32) smbclient -U Administrator --use-krb5-ccache=/tmp/krb5cc_1000 //samba-dc.example.com/demo + Try "help" to get a list of possible commands. + smb: \> + + Make a fresh VM for the client: + + $ sudo vim /etc/hosts + Add an entry with its IP address, e.g.: + 192.168.122.124 samba-dc samba-dc.example.com + $ sudo apt install cifs-utils + $ sudo -s + # kinit [email protected] + Password for [email protected]: + Warning: Your password will expire in 41 days on Wed May 21 02:51:02 2025 + # klist + Ticket cache: FILE:/tmp/krb5cc_0 + Default principal: [email protected] + + Valid starting Expires Service principal + 04/09/25 03:26:10 04/09/25 13:26:10 krbtgt/[email protected] + renew until 04/10/25 03:26:06 + # mkdir /mnt/testshare1 + # mount -t cifs -o cruid=root,user=root,sec=krb5i,uid=0,gid=0,cred=/tmp/krb5cc_0 //samba-dc.example.com/demo /mnt/testshare1 + Create some small files, 6 should be enough. + # fallocate -l 1k d1.data + # fallocate -l 1k d2.data + # fallocate -l 1k d3.data + # fallocate -l 1k d4.data + # fallocate -l 1k d5.data + # fallocate -l 1k d6.data + Check the free memory: + root@noble-client:/home/ubuntu# free -m + total used free shared buff/cache available + Mem: 3915 393 3519 1 224 3521 + Swap: 0 0 0 + Open up six terminals, and run this in each: + # while true ; do zstd --force d1.data ; done + # while true ; do zstd --force d2.data ; done + # while true ; do zstd --force d3.data ; done + # while true ; do zstd --force d4.data ; done + # while true ; do zstd --force d5.data ; done + # while true ; do zstd --force d6.data ; done + Watch free memory slowly decrease in the "free" column. + root@noble-client:/home/ubuntu# free -m + total used free shared buff/cache available + Mem: 3915 529 2783 1 393 3385 + Swap: 0 0 0 + root@noble-client:/home/ubuntu# free -m + total used free shared buff/cache available + Mem: 3915 534 1590 1 483 3278 + Swap: 0 0 0 + + This may take several hours. + + There is a test kernel in the following ppa: + + https://launchpad.net/~mruffell/+archive/ubuntu/lp2131213-test + + If you enable the ppa and install the test kernel, the memory leak will no longer + occur. + + [Where problems can occur] + + We are changing how folios are managed and freed. This could lead to additional + memory leaks or data corruption if a regression were to occur. + + Since this affects core writeback operations, if a regression were to occur, it + would likely impact all cifs users, and there likely would not be any workarounds. [Other info] - Originally reported in closed Discourse post at https://discourse.ubuntu.com/t/ubuntu-24-04-file-operations-on-top-of-a-cifs-mount-leak-memory/56439 - --- - ProblemType: Bug - AlsaDevices: - total 0 - crw-rw---- 1 root audio 116, 1 Nov 13 10:37 seq - crw-rw---- 1 root audio 116, 33 Nov 13 10:37 timer - AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' - ApportVersion: 2.28.1-0ubuntu3.8 - Architecture: amd64 - ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' - AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: - CRDA: N/A - CasperMD5CheckResult: unknown - CloudArchitecture: x86_64 - CloudBuildName: server - CloudID: multipass - CloudName: multipass - CloudPlatform: nocloud - CloudSerial: 20251031 - CloudSubPlatform: config-disk (/dev/sr0) - CurrentDmesg: Error: command ['dmesg'] failed with exit code 1: dmesg: read kernel buffer failed: Operation not permitted - DistroRelease: Ubuntu 24.04 - IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' - Lsusb: Error: command ['lsusb'] failed with exit code 1: - Lsusb-t: - Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: - MachineType: QEMU Standard PC (i440FX + PIIX, 1996) - Package: linux (not installed) - PciMultimedia: + Reported in the following discourse posts: + https://discourse.ubuntu.com/t/ubuntu-24-04-file-operations-on-top-of-a-cifs-mount-leak-memory/56439 + https://discourse.ubuntu.com/t/ubuntu-24-04-file-operations-on-top-of-a-cifs-mount-leak-memory/62985 - ProcEnviron: - LANG=C.UTF-8 - PATH=(custom, no user) - SHELL=/bin/bash - TERM=xterm-256color - XDG_RUNTIME_DIR=<set> - ProcFB: 0 simpledrmdrmfb - ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-87-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 - ProcVersionSignature: User Name 6.8.0-87.88-generic 6.8.12 - RelatedPackageVersions: - linux-restricted-modules-6.8.0-87-generic N/A - linux-backports-modules-6.8.0-87-generic N/A - linux-firmware N/A - RfKill: Error: [Errno 2] No such file or directory: 'rfkill' - Tags: cloud-image noble - Uname: Linux 6.8.0-87-generic x86_64 - UpgradeStatus: No upgrade log present (probably fresh install) - UserGroups: adm cdrom dip lxd sudo - _MarkForUpload: True - acpidump: + Fixed recently in -azure 6.8.0-1040.46: + https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2122719 - dmi.bios.date: 02/06/2015 - dmi.bios.release: 0.0 - dmi.bios.vendor: EFI Development Kit II / OVMF - dmi.bios.version: 0.0.0 - dmi.chassis.type: 1 - dmi.chassis.vendor: QEMU - dmi.chassis.version: pc-i440fx-8.2 - dmi.modalias: dmi:bvnEFIDevelopmentKitII/OVMF:bvr0.0.0:bd02/06/2015:br0.0:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-8.2:cvnQEMU:ct1:cvrpc-i440fx-8.2:sku: - dmi.product.name: Standard PC (i440FX + PIIX, 1996) - dmi.product.version: pc-i440fx-8.2 - dmi.sys.vendor: QEMU + Upstream mailing list discussions: + V3: + https://lore.kernel.org/all/[email protected]/ + V4: + https://lore.kernel.org/linux-cifs/[email protected]/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2131213 Title: cifs: Fix memory leak of a folio every call to cifs_writepages_begin() To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2131213/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
