Re: [lxc-users] ?==?utf-8?q? LXD 2.14 - Ubuntu 16.04 - kernel 4.4.0-57-generic - SWAP continuing to grow
Most likely your database cache is simply set too large. I've been experiencing similar issues with MySQL (please read in detail): https://stackoverflow.com/questions/43259136/mysqld-out-of-memory-with-plenty-of-memory/43259820 It finally went away after I've been lowering MySQL cache by a few GBs from each OOM to OOM, until it stopped happenin Tomasz Chmielewski https://lxadm.com On Saturday, July 15, 2017 18:36 JST, Saint Michael wrote: > I have a lot of memory management issues using pure LXC. In my case, my box > has only one container. I use LXC to be able to move my app around, not to > squeeze performance out of hardware. What happens is my database gets > killed the OOM manager, although there are gigabytes of RAM used for cache. > The memory manager kills applications instead of reclaiming memory from > disc cache. How can this be avoided? > > My config at the host is: > > vm.hugepages_treat_as_movable=0 > vm.hugetlb_shm_group=27 > vm.nr_hugepages=2500 > vm.nr_hugepages_mempolicy=2500 > vm.nr_overcommit_hugepages=0 > vm.overcommit_memory=0 > vm.swappiness=0 > vm.vfs_cache_pressure=150 > vm.dirty_ratio=10 > vm.dirty_background_ratio=5 > > This shows the issue > [9449866.130270] Node 0 hugepages_total=1250 hugepages_free=1250 > hugepages_surp=0 hugepages_size=2048kB > [9449866.130271] Node 1 hugepages_total=1250 hugepages_free=1248 > hugepages_surp=0 hugepages_size=2048kB > [9449866.130271] 46181 total pagecache pages > [9449866.130273] 33203 pages in swap cache > [9449866.130274] Swap cache stats: add 248571542, delete 248538339, find > 69031185/100062903 > [9449866.130274] Free swap = 0kB > [9449866.130275] Total swap = 8305660kB > [9449866.130276] 20971279 pages RAM > [9449866.130276] 0 pages HighMem/MovableOnly > [9449866.130276] 348570 pages reserved > [9449866.130277] 0 pages cma reserved > [9449866.130277] 0 pages hwpoisoned > [9449866.130278] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds > swapents oom_score_adj name > [9449866.130286] [ 618] 0 61887181 135 168 3 >3 0 systemd-journal > [9449866.130288] [ 825] 0 82511343 130 25 3 >0 0 systemd-logind > [9449866.130289] [ 830] 0 830 1642 31 8 3 >0 0 mcelog > [9449866.130290] [ 832] 996 83226859 51 23 3 > 47 0 chronyd > [9449866.130292] [ 834] 0 834 4905 100 12 3 >0 0 irqbalance > [9449866.130293] [ 835] 0 835 6289 177 15 3 >0 0 smartd > [9449866.130295] [ 837]81 83728499 258 28 3 > 149 -900 dbus-daemon > [9449866.130296] [ 857] 0 857 1104 16 7 3 >0 0 rngd > [9449866.130298] [ 859] 0 859 19246337114 224 4 > 40630 0 NetworkManager > [9449866.130300] [ 916] 0 91625113 229 50 3 >0 -1000 sshd > [9449866.130302] [ 924] 0 924 6490 50 17 3 >0 0 atd > [9449866.130303] [ 929] 0 92935327 199 20 3 > 284 0 agetty > [9449866.130305] [ 955] 0 95522199 3185 43 3 > 312 0 dhclient > [9449866.130307] [ 1167] 0 1167 6125 88 17 3 >2 0 lxc-autostart > [9449866.130309] [ 1176] 0 117610818 275 24 3 > 38 0 systemd > [9449866.130310] [ 1188] 0 118813303 1980 29 3 > 36 0 systemd-journal > [9449866.130312] [ 1372]99 1372 38812 12 3 > 45 0 dnsmasq > [9449866.130313] [ 1375]81 1375 6108 77 17 3 > 39 -900 dbus-daemon > [9449866.130315] [ 1394] 0 1394 6175 46 15 3 > 168 0 systemd-logind > [9449866.130316] [ 1395] 0 139578542 1142 69 3 >4 0 rsyslogd > [9449866.130317] [ 1397] 0 1397 1614 32 8 3 >0 0 agetty > [9449866.130319] [ 1398] 0 1398 1614 31 8 3 >0 0 agetty > [9449866.130320] [ 1400] 0 1400 1614 31 8 3 >0 0 agetty > [9449866.130321] [ 1401] 0 1401 16142 8 3 > 30 0 agetty > [9449866.130322] [ 1402] 0 1402 16142 8 3 > 29 0 agetty > [9449866.130324] [ 1403] 0 1403 1614 31 8 3 >0 0 agetty > [9449866.130325] [ 1404] 0 1404 1614 32 8 3 >0 0 agetty > [9449866.130327] [ 1405] 0 1405 1614 32 8 3 >0 0 agetty > [9449866.130328] [ 1406] 0 1406 16142 8 3 > 29 0 agetty > [9449866.13032
[lxc-users] Networking Issues
Hi all, We are trying to convert our libvirt-lxc containers to LXC containers on CentOS 7 due to race conditions that are causing a lot of instability during our testing of the former. I am currently testing against LXC 1.0.9 because it was easily available to me in the EPEL repo, but I hope to upgrade to an LXC 2.x.x version in the future. For the most part, this is working great. I can even run lxc-start or lxc-autostart manually and see everything-including our five veth interfaces-successfully come online for the container. The problem, however, is that if I allow the default lxc.service systemd service, or my custom systemd service, to automatically start the container on boot, only a single interface comes online (as in, gets an IP address...though I can attach to the container and do an ip addr show and see that all interfaces are up, just with no IP addresses). Further, if I try to restart the container manually after systemd automatically starts the container, only a single interface continues to come online. I currently have SELinux disabled to try to eliminate at least one possibly source of issue. Thinking that maybe networking isn't up yet (even though the services require networking-online.service), I have tried adding in a four minute delay prior to letting the service attempt to auto-start LXC containers to no avail. Interestingly, after disabling all the above-discussed LXC services, I also tried using the @reboot "hack" to run lxc-autostart as root to no avail. For what it is worth, this container (and others that will follow) can/should be run as root (unprivledged?). We plan to use SELinux to "sandbox" them. I'll paste the current configuration file I'm working with below. If there are specific log files that would be nice to see as well, please let me know. Any help/pointers/etc would be deeply appreciated. Thanks so much, Luke # Distribution configuration lxc.include = /usr/share/lxc/config/centos.common.conf lxc.arch = x86_64 # Container specific configuration lxc.rootfs = /var/lib/libvirt/filesystems/newcontainer lxc.utsname = newcontainer # SELinux context #lxc.se_context = system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 lxc.logfile = /newcontainer.log lxc.loglevel = 1 # Enable auto start lxc.start.auto = 1 # Set up some system limits #lxc.cgroup.memory.limit_in_bytes = 488282K #lxc.cgroup.memory.memsw.limit_in_bytes = 50K #lxc.cgroup.cpuset.cpus = 0 # Network configuration lxc.network.0.type = veth lxc.network.0.flags = up lxc.network.0.link = br-base lxc.network.0.hwaddr = 52:54:00:af:83:80 lxc.network.1.type = veth lxc.network.1.flags = up lxc.network.1.link = br-eth0 lxc.network.1.hwaddr = 52:54:00:6e:5a:47 lxc.network.2.type = veth lxc.network.2.flags = up lxc.network.2.link = br-eth2 lxc.network.2.hwaddr = 52:54:00:4d:b5:6e lxc.network.3.type = veth lxc.network.3.flags = up lxc.network.3.link = br-eth3 lxc.network.3.hwaddr = 52:54:00:eb:a3:24 lxc.network.4.type = veth lxc.network.4.flags = up lxc.network.4.link = br-eth4 lxc.network.4.hwaddr = 52:54:00:ec:a4:4a This E-mail is confidential. It may also be legally privileged. If you are not the addressee you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return E-mail. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] LXD 2.14 - Ubuntu 16.04 - kernel 4.4.0-57-generic - SWAP continuing to grow
I have a lot of memory management issues using pure LXC. In my case, my box has only one container. I use LXC to be able to move my app around, not to squeeze performance out of hardware. What happens is my database gets killed the OOM manager, although there are gigabytes of RAM used for cache. The memory manager kills applications instead of reclaiming memory from disc cache. How can this be avoided? My config at the host is: vm.hugepages_treat_as_movable=0 vm.hugetlb_shm_group=27 vm.nr_hugepages=2500 vm.nr_hugepages_mempolicy=2500 vm.nr_overcommit_hugepages=0 vm.overcommit_memory=0 vm.swappiness=0 vm.vfs_cache_pressure=150 vm.dirty_ratio=10 vm.dirty_background_ratio=5 This shows the issue [9449866.130270] Node 0 hugepages_total=1250 hugepages_free=1250 hugepages_surp=0 hugepages_size=2048kB [9449866.130271] Node 1 hugepages_total=1250 hugepages_free=1248 hugepages_surp=0 hugepages_size=2048kB [9449866.130271] 46181 total pagecache pages [9449866.130273] 33203 pages in swap cache [9449866.130274] Swap cache stats: add 248571542, delete 248538339, find 69031185/100062903 [9449866.130274] Free swap = 0kB [9449866.130275] Total swap = 8305660kB [9449866.130276] 20971279 pages RAM [9449866.130276] 0 pages HighMem/MovableOnly [9449866.130276] 348570 pages reserved [9449866.130277] 0 pages cma reserved [9449866.130277] 0 pages hwpoisoned [9449866.130278] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [9449866.130286] [ 618] 0 61887181 135 168 3 3 0 systemd-journal [9449866.130288] [ 825] 0 82511343 130 25 3 0 0 systemd-logind [9449866.130289] [ 830] 0 830 1642 31 8 3 0 0 mcelog [9449866.130290] [ 832] 996 83226859 51 23 3 47 0 chronyd [9449866.130292] [ 834] 0 834 4905 100 12 3 0 0 irqbalance [9449866.130293] [ 835] 0 835 6289 177 15 3 0 0 smartd [9449866.130295] [ 837]81 83728499 258 28 3 149 -900 dbus-daemon [9449866.130296] [ 857] 0 857 1104 16 7 3 0 0 rngd [9449866.130298] [ 859] 0 859 19246337114 224 4 40630 0 NetworkManager [9449866.130300] [ 916] 0 91625113 229 50 3 0 -1000 sshd [9449866.130302] [ 924] 0 924 6490 50 17 3 0 0 atd [9449866.130303] [ 929] 0 92935327 199 20 3 284 0 agetty [9449866.130305] [ 955] 0 95522199 3185 43 3 312 0 dhclient [9449866.130307] [ 1167] 0 1167 6125 88 17 3 2 0 lxc-autostart [9449866.130309] [ 1176] 0 117610818 275 24 3 38 0 systemd [9449866.130310] [ 1188] 0 118813303 1980 29 3 36 0 systemd-journal [9449866.130312] [ 1372]99 1372 38812 12 3 45 0 dnsmasq [9449866.130313] [ 1375]81 1375 6108 77 17 3 39 -900 dbus-daemon [9449866.130315] [ 1394] 0 1394 6175 46 15 3 168 0 systemd-logind [9449866.130316] [ 1395] 0 139578542 1142 69 3 4 0 rsyslogd [9449866.130317] [ 1397] 0 1397 1614 32 8 3 0 0 agetty [9449866.130319] [ 1398] 0 1398 1614 31 8 3 0 0 agetty [9449866.130320] [ 1400] 0 1400 1614 31 8 3 0 0 agetty [9449866.130321] [ 1401] 0 1401 16142 8 3 30 0 agetty [9449866.130322] [ 1402] 0 1402 16142 8 3 29 0 agetty [9449866.130324] [ 1403] 0 1403 1614 31 8 3 0 0 agetty [9449866.130325] [ 1404] 0 1404 1614 32 8 3 0 0 agetty [9449866.130327] [ 1405] 0 1405 1614 32 8 3 0 0 agetty [9449866.130328] [ 1406] 0 1406 16142 8 3 29 0 agetty [9449866.130329] [ 1408] 0 1408 16142 8 3 30 0 agetty [9449866.130330] [ 1409] 0 1409 1614 30 7 3 0 0 agetty [9449866.130332] [18224] 0 18224264560 43 4 404 0 VGAuthService [9449866.130333] [18225] 0 1822561032 95 58 3 258 0 vmtoolsd [9449866.130335] [28660] 0 2866026372 44 54 4 202 -1000 sshd [9449866.130337] [18992] 998 18992 132859 876 54 3 13 0 polkitd [9449866.130339] [23849] 0 2384910744 370 23 3
