[Bug 1860813] Re: LXC container reports spike in swap occasionally
This bug was fixed in the package lxcfs - 3.0.3-0ubuntu1~18.04.2 --- lxcfs (3.0.3-0ubuntu1~18.04.2) bionic; urgency=medium * d/p/0001-better-handle-swapfree-calculation.patch: (LP: #1860813) - bindings: fix swapFree calculation with swap accounting enabled -- Kellen Renshaw Tue, 31 Mar 2020 16:20:51 + ** Changed in: lxcfs (Ubuntu Bionic) Status: Fix Committed => Fix Released ** Changed in: lxcfs (Ubuntu Eoan) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
This bug was fixed in the package lxcfs - 3.0.4-2ubuntu1 --- lxcfs (3.0.4-2ubuntu1) eoan; urgency=medium * d/p/0001-better-handle-swapfree-calculation.patch: (LP: #1860813) - bindings: fix swapFree calculation with swap accounting enabled -- Kellen Renshaw Tue, 31 Mar 2020 16:32:11 + -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
SRU Verification, Eoan/19.10 Update, forgot to enable swapaccount=1, corrected verification below LXD deb in Eoan uses the snap, which includes it's own version of LXCFS. Testing altered to verify that LXCFS in Eoan/universe is fixed. root@test-lxcfs-eoan:~# dpkg -l lxcfs Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==---= ii lxcfs 3.0.4-2 amd64FUSE based filesystem for LXC Create LXCFS filesystem for testing: root@test-lxcfs-eoan:~# lxcfs -f -p /run/tmp_lxcfs ./test/ mount namespace: 5 hierarchies: 0: fd: 6: memory 1: fd: 7: blkio 2: fd: 8: rdma 3: fd: 9: hugetlb 4: fd: 10: pids 5: fd: 11: devices 6: fd: 12: net_cls,net_prio 7: fd: 13: freezer 8: fd: 14: perf_event 9: fd: 15: cpu,cpuacct 10: fd: 16: cpuset 11: fd: 17: name=systemd 12: fd: 18: unified Test session 1: watch -n0.1 "grep SwapFree ./test/proc/meminfo" Test session 2: root@test-lxcfs-eoan:~# while true; do grep SwapFree ./test/proc/meminfo ; done ... SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree: 0 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB ... Problem does reproduce with the current package root@test-lxcfs-eoan:~# dpkg -l lxcfs Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name VersionArchitecture Description +++-==-==--= ii lxcfs 3.0.4-2ubuntu1 amd64FUSE based filesystem for LXC Create LXCFS filesystem for testing: root@test-lxcfs-eoan:~# lxcfs -f -p /run/tmp_lxcfs ./test/ mount namespace: 5 hierarchies: 0: fd: 6: memory 1: fd: 7: blkio 2: fd: 8: rdma 3: fd: 9: hugetlb 4: fd: 10: pids 5: fd: 11: devices 6: fd: 12: net_cls,net_prio 7: fd: 13: freezer 8: fd: 14: perf_event 9: fd: 15: cpu,cpuacct 10: fd: 16: cpuset 11: fd: 17: name=systemd 12: fd: 18: unified Test session 1: watch -n0.1 "grep SwapFree ./test/proc/meminfo" Test session 2: root@test-lxcfs-eoan:~# while true; do grep SwapFree ./test/proc/meminfo ; done ... SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB Problem doesn't reproduce with the package in -proposed ** Tags removed: verification-needed verification-needed-eoan ** Tags added: verification-done verification-done-eoan -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
SRU Verification, Eoan/19.10 LXD deb in Eoan uses the snap, which includes it's own version of LXCFS. Testing altered to verify that LXCFS in Eoan/universe is fixed. root@test-lxcfs-eoan:~# dpkg -l lxcfs Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name VersionArchitecture Description +++-==-==--= ii lxcfs 3.0.4-2ubuntu1 amd64FUSE based filesystem for LXC Create LXCFS filesystem for testing: root@test-lxcfs-eoan:~# lxcfs -f -p /run/tmp_lxcfs ./test/ mount namespace: 5 hierarchies: 0: fd: 6: memory 1: fd: 7: blkio 2: fd: 8: rdma 3: fd: 9: hugetlb 4: fd: 10: pids 5: fd: 11: devices 6: fd: 12: net_cls,net_prio 7: fd: 13: freezer 8: fd: 14: perf_event 9: fd: 15: cpu,cpuacct 10: fd: 16: cpuset 11: fd: 17: name=systemd 12: fd: 18: unified Test session 1: watch -n0.1 "grep SwapFree ./test/proc/meminfo" Test session 2: root@test-lxcfs-eoan:~# while true; do grep SwapFree ./test/proc/meminfo ; done ... SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB SwapFree:2097148 kB Problem doesn't reproduce with the package in -proposed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
SRU Verification, Bionic/18.04 ubuntu@lxcfs-test-bionic:~$ dpkg -l | grep lxcfs ii lxcfs 3.0.3-0ubuntu1~18.04.2 amd64FUSE based filesystem for LXC ubuntu@lxcfs-test-bionic:~$ lxc launch ubuntu:b sru-test-bionic To start your first container, try: lxc launch ubuntu:18.04 Creating sru-test-bionic Starting sru-test-bionic ubuntu@lxcfs-test-bionic:~$ lxc shell sru-test-bionic #opened two in separate sessions mesg: ttyname failed: No such device Shell 1: root@sru-test-bionic:~# watch -n 0.1 "grep Swap /proc/meminfo" Shell 2: root@sru-test-bionic:~# while true; do free | grep Swap; done ... Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 Swap: 1466364 0 1466364 ... Problem no longer appears in Bionic with lxcfs 3.0.3-0ubuntu1~18.04.2 ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Hello Kellen, or anyone else affected, Accepted lxcfs into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxcfs/3.0.4-2ubuntu1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-eoan. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: lxcfs (Ubuntu Eoan) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-eoan ** Changed in: lxcfs (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
[sts-sponsor] Sponsored for E/B. Thanks Kellen for your contribution. - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Test for 19.10/eoan Autopkgtests: autopkgtest [16:38:08]: test exercise: ---] autopkgtest [16:38:08]: test exercise: - - - - - - - - - - results - - - - - - - - - - exercise PASS autopkgtest [16:38:08]: summary exercise PASS Manual test: ubuntu@test-lxcfs-eoan:~$ lxc launch ubuntu:e test1 ubuntu@test-lxcfs-eoan:~$ lxc shell test1 # opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 ... # note, doesn't seem to be reproducing on this machine, has for others ubuntu@test-lxcfs-eoan:~$ sudo apt install ./lxcfs_3.0.4-2ubuntu1_amd64.deb ubuntu@test-lxcfs-eoan:~$ dpkg -l | grep lxcfs ii lxcfs3.0.4-2ubuntu1 amd64FUSE based filesystem for LXC ubuntu@test-lxcfs-eoan:~$ lxc stop test1 ubuntu@test-lxcfs-eoan:~$ lxc start test1 ubuntu@test-lxcfs-eoan:~$ lxc shell test1 # opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 ... ** Patch added: "lp1860813_eoan.debdiff" https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5343854/+files/lp1860813_eoan.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Revised debdiff, test information for 18.04/bionic Autopkgtest: autopkgtest [16:25:54]: test exercise: ---] autopkgtest [16:25:54]: test exercise: - - - - - - - - - - results - - - - - - - - - - exercise PASS autopkgtest [16:25:54]: summary exercise PASS Manual testing: ubuntu@test-lxcfs-bionic:~$ dpkg -l | grep lxcfs ii lxcfs 3.0.3-0ubuntu1~18.04.1 amd64FUSE based filesystem for LXC ubuntu@test-lxcfs-bionic:~$ lxc launch ubuntu:b test1 ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 Swap: 2097148 0 2097148 ... ubuntu@test-lxcfs-bionic:~$ lxc stop test1 ubuntu@test-lxcfs-bionic:~$ sudo apt install ./lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'lxcfs' instead of './lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb' The following packages will be upgraded: lxcfs 1 upgraded, 0 newly installed, 0 to remove and 6 not upgraded. Need to get 0 B/44.2 kB of archives. After this operation, 0 B of additional disk space will be used. Get:1 /home/ubuntu/lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb lxcfs amd64 3.0.3-0ubuntu1~18.04.2 [44.2 kB] (Reading database ... 66998 files and directories currently installed.) Preparing to unpack .../lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb ... Unpacking lxcfs (3.0.3-0ubuntu1~18.04.2) over (3.0.3-0ubuntu1~18.04.1) ... Setting up lxcfs (3.0.3-0ubuntu1~18.04.2) ... Processing triggers for ureadahead (0.100.0-21) ... Processing triggers for systemd (237-3ubuntu10.39) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... ubuntu@test-lxcfs-bionic:~$ lxc start test1 ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 ... Issue doesn't occur ** Patch added: "lp1860813_bionic.debdiff" https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5343837/+files/lp1860813_bionic.debdiff ** Patch removed: "lp1860813_bionic.debdiff" https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5342354/+files/lp1860813_bionic.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Could not reproduce with reproducer on Xenial, with lxcfs 2.0.8-0ubuntu1~16.04.2 running on the GA 4.4 and HWE 4.15 kernels. Xenial does not appear to be affected. ** Changed in: lxcfs (Ubuntu Xenial) Status: New => Invalid ** Changed in: lxcfs (Ubuntu Eoan) Status: New => In Progress ** Changed in: lxcfs (Ubuntu Eoan) Importance: Undecided => Medium ** Changed in: lxcfs (Ubuntu Eoan) Assignee: (unassigned) => Kellen Renshaw (krenshaw) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
** Also affects: lxcfs (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: lxcfs (Ubuntu Eoan) Importance: Undecided Status: New ** Also affects: lxcfs (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: lxcfs (Ubuntu Bionic) Assignee: (unassigned) => Kellen Renshaw (krenshaw) ** Changed in: lxcfs (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: lxcfs (Ubuntu Bionic) Status: New => In Progress ** Tags added: sts-sponsor-slashd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Attaching debdiff ** Patch added: "lp1860813_bionic.debdiff" https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+attachment/5342354/+files/lp1860813_bionic.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Test for 18.04/bionic Autopkgtests: autopkgtest [21:29:26]: test exercise: ---] autopkgtest [21:29:27]: test exercise: - - - - - - - - - - results - - - - - - - - - - exercise PASS autopkgtest [21:29:27]: summary exercise PASS autopkgtest [21:29:27]: Binaries: resetting testbed apt configuration Manual testing: ubuntu@test-lxcfs-bionic:~$ dpkg -l | grep lxcfs ii lxcfs 3.0.3-0ubuntu1~18.04.1 amd64FUSE based filesystem for LXC ubuntu@test-lxcfs-bionic:~$ lxc launch ubuntu:b test1 ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 Swap: 2097148 0 2097148 ... ubuntu@test-lxcfs-bionic:~$ lxc stop test1 ubuntu@test-lxcfs-bionic:~$ sudo apt install ./lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'lxcfs' instead of './lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb' The following packages will be upgraded: lxcfs 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/44.1 kB of archives. After this operation, 0 B of additional disk space will be used. Get:1 /home/ubuntu/lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb lxcfs amd64 3.0.3-0ubuntu1~18.04.2 [44.1 kB] (Reading database ... 66998 files and directories currently installed.) Preparing to unpack .../lxcfs_3.0.3-0ubuntu1~18.04.2_amd64.deb ... Unpacking lxcfs (3.0.3-0ubuntu1~18.04.2) over (3.0.3-0ubuntu1~18.04.1) ... Setting up lxcfs (3.0.3-0ubuntu1~18.04.2) ... Processing triggers for ureadahead (0.100.0-21) ... Processing triggers for systemd (237-3ubuntu10.39) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... ubuntu@test-lxcfs-bionic:~$ lxc start test1 ubuntu@test-lxcfs-bionic:~$ lxc shell test1 #opened two in separate sessions Session 1 root@test1:~# watch -n 0.1 "grep Swap /proc/meminfo" Session 2 root@test1:~# while true; do free | grep Swap; done ... Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 ... Issue doesn't occur -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
** Changed in: lxcfs (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
** Changed in: lxcfs (Ubuntu) Status: Fix Released => New ** Changed in: lxcfs (Ubuntu) Assignee: (unassigned) => Kellen Renshaw (krenshaw) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Previous description, for context: lxcfs provides a container-specific view of /proc/meminfo. Occasionally, with near zero or zero swap usage *and* swap accounting turned on (kernel parameter swapaccount=1), the container will report 100% swap utilization. Example from Ubuntu 18.04 and lxcfs 3.0.3-0ubuntu1~18.04.1, using the output of "while true; do free |grep Swap >> swap.log; done" with the "free" command's header for clarity: totalusedfree shared buff/cache available Mem:2040052 53028 1839908 176 147116 1987024 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 <--- The issue Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Additionally, "watch -n 0.1 "grep Swap /proc/meminfo"" was running in another terminal on the container, this seems to stimulate the behavior. This appears to come from proc_meminfo_read in bindings.c in lxcfs. >From the kernel documentation (https://github.com/torvalds/linux/blob/master/Documentation/admin- guide/cgroup-v1/memory.rst), under 5.5 usage_in_bytes: "For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2)." When swap accounting is on and swap usage is 0 or very close to 0, the kernel reported mem+swap usage and the mem usage numbers can cross over, causing the assignments below to set swapfree to 0, triggering the issue: swapusage = memswusage - memusage, swapfree = swapusage < swaptotal ? swaptotal - swapusage : 0; ** Description changed: - lxcfs provides a container-specific view of /proc/meminfo. Occasionally, - with near zero or zero swap usage *and* swap accounting turned on - (kernel parameter swapaccount=1), the container will report 100% swap - utilization. + [Impact] - Example from Ubuntu 18.04 and lxcfs 3.0.3-0ubuntu1~18.04.1, using the output of "while true; do free |grep Swap >> swap.log; done" with the "free" command's header for clarity: - totalusedfree shared buff/cache available - Mem:2040052 53028 1839908 176 147116 1987024 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 2097148 0 <--- The issue - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 + * lxcfs provides a container-specific view of /proc/meminfo. + Occasionally, with near zero or zero swap usage *and* swap accounting + turned on (kernel parameter swapaccount=1), the container will report + 100% swap utilization. - Additionally, "watch -n 0.1 "grep Swap /proc/meminfo"" was running in - another terminal on the container, this seems to stimulate the behavior. + * This issue has been encountered and could result in unecessary alerts + or potential automated attempts at remediating a non-existent "full + swap" issue. - This appears to come from proc_meminfo_read in bindings.c in lxcfs. + * This fix changed the logic used for SwapFree when swap accounting is + enabled to better handle situations where memswusage is less than + memusage, caused by the fuzziness of the usage_in_bytes counters used as + the source. Specifically, it added a check for memusage being larger + than memswusage and if s
[Bug 1860813] Re: LXC container reports spike in swap occasionally
** Changed in: lxcfs (Ubuntu) Status: New => Incomplete ** Changed in: lxcfs (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1860813] Re: LXC container reports spike in swap occasionally
Proposed fix: https://github.com/lxc/lxcfs/pull/316 Tested on Ubuntu 18.04 by replacing the lxcfs binary and library, and retrying the reproducer there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
