Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
What kind of memory savings did you get using UKSM? Did you run it on an LXD server? I finally got KSM working as expected on my containers using the ksm_preload technique (thanks Fajar!). Unfortunately, the RAM savings is not as high as I expected. I have about 50 containers - each running nginx and php7 (identical). I modified the startup scripts for nginx and php7 to use the “ksm-wrapper” tool (which uses the ksm_preload technique) and restarted all the containers. After everything settled down, the best I could get was about 150MB of saved memory. Much less than I expected. I looked into UKSM, but that requires a custom kernel (either build your own or use one of the pre-built ones: pf-kernel). Since these are production servers, I am very hesitant to use anything but a standard kernel. -Ron On Jun 6, 2017, at 5:13 AM, Andreas Freudenberg wrote: Hi, as stated by Tomasz, KSM will only work for applications which support it. If you want a KSM for all apllications you could try UKSM [1]. Worked pretty well here .. AF [1] http://kerneldedup.org/en/projects/uksm/introduction/ Am 05.06.2017 um 03:18 schrieb Tomasz Chmielewski: > KSM only works with applications which support it: > > KSM only operates on those areas of address space which an application > has advised to be likely candidates > for merging, by using the madvise(2) system call: int madvise(addr, > length, MADV_MERGEABLE. > > This means that doing: > >echo 1 > /sys/kernel/mm/ksm/run > > will be enough for KVM, but will not do anything for applications like bash, > nginx, apache, php-fpm and so on. > > > Please refer to "Enabling memory deduplication libraries in containers" on > https://openvz.org/KSM_(kernel_same-page_merging) - you will have to use > ksm_preload mentioned. I haven't personally used it with LXD. > > > Tomasz Chmielewski > https://lxadm.com > > > On Monday, June 05, 2017 09:48 JST, Ron Kelley wrote: > >> Thanks Fajar. >> >> This is on-site with our own physical servers, storage, etc. The goal is to >> get the most containers per server as possible. While our servers have lots >> of RAM, we need to come up with a long-term scaling plan and hope KSM can >> help us scale beyond the standard numbers. >> >> As for the openvz link; I read that a few times but I don’t get any positive >> results using those methods. This leads me to believe (a) LXD does not >> support KSM or (b) the applications are not registering w/the KSM part of >> the kernel. >> >> I am going to run through some tests this week to see if I can get KSM >> working outside the LXD environment then try to replicate the same tests >> inside LXD. >> >> Thanks again for the feedback. >> >> >> >>> On Jun 4, 2017, at 6:15 PM, Fajar A. Nugraha wrote: >>> >>> On Sun, Jun 4, 2017 at 11:16 PM, Ron Kelley wrote: >>> (Reviving the thread about Container Scaling: >>> https://lists.linuxcontainers.org/pipermail/lxc-users/2016-May/011607.html) >>> >>> We have hit critical mass with LXD 2.12 and I need to get Kernel Samepage >>> Merging (KSM) working as soon as possible. All my research has come to a >>> dead-end, and I am reaching out to the group at large for suggestions. >>> >>> Background: We have 5 host servers - each running U16.04 >>> (4.4.0-57-generic), 8G RAM, 20G SWAP, and 50 containers (exact configs per >>> server - nginx and php 7). >>> >>> >>> Is this a cloud, or on-site setup? >>> >>> For cloud, there are a lot of options that could get you running with MUCH >>> more memory, which would save you lots of headaches getting KSM to work. My >>> favorite is EC2 spot instance on AWS. >>> >>> On another note, I now setup most of my hosts with no swap, since >>> performance plummets whenever swap is used. YMMV. >>> >>> >>> I am trying to get KSM working since each container is an identical replica >>> of the other (other than hostname/IP). I have read a ton of information on >>> the ‘net about Ubuntu and KSM, yet I can’t seem to get any pages to share >>> on the host. I am not sure if this is a KSM config issue or if LXD won’t >>> allow KSM between containers. >>> >>> >>> >>> >>> >>> Here is what I have done thus far: >>> -- >>> * Installed the ksmtuned utility and verified ksmd is running on each host. >>> * Created the ksm_preload and ksm-wrapper tools per this site (the >>> https://github.com/unbrice/ksm_preload). >>> * Created 50 identical Ubuntu 16.04 containers running nginx >>> * Modified the nginx startup script on each container to include the >>> ksm_preload.so library; no issues running nginx. >>> >>> (Note: since I could not find the ksm_preload library for Ubuntu, I had to >>> use the ksm-wrapper tool listed above) >>> >>> All the relevant files under /sys/kernel/mm/ksm still show 0 (pages_shared, >>> pages_sharing, etc) regardless of what I do. >>> >>> >>> Can any (@stgraber @brauner) confirm if KSM is supported
Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
On Tue, Jun 6, 2017 at 4:13 PM, Andreas Freudenberg wrote: > Hi, > > as stated by Tomasz, KSM will only work for applications which support it. > ... or when you use ksm_preload > If you want a KSM for all apllications you could try UKSM [1]. > > UKSM is not in upstream kernel yet, is it. Is there a ppa or ubuntu package for it? -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
Hi, as stated by Tomasz, KSM will only work for applications which support it. If you want a KSM for all apllications you could try UKSM [1]. Worked pretty well here .. AF [1] http://kerneldedup.org/en/projects/uksm/introduction/ Am 05.06.2017 um 03:18 schrieb Tomasz Chmielewski: > KSM only works with applications which support it: > > KSM only operates on those areas of address space which an application > has advised to be likely candidates > for merging, by using the madvise(2) system call: int madvise(addr, > length, MADV_MERGEABLE. > > This means that doing: > > echo 1 > /sys/kernel/mm/ksm/run > > will be enough for KVM, but will not do anything for applications like bash, > nginx, apache, php-fpm and so on. > > > Please refer to "Enabling memory deduplication libraries in containers" on > https://openvz.org/KSM_(kernel_same-page_merging) - you will have to use > ksm_preload mentioned. I haven't personally used it with LXD. > > > Tomasz Chmielewski > https://lxadm.com > > > On Monday, June 05, 2017 09:48 JST, Ron Kelley wrote: > >> Thanks Fajar. >> >> This is on-site with our own physical servers, storage, etc. The goal is to >> get the most containers per server as possible. While our servers have lots >> of RAM, we need to come up with a long-term scaling plan and hope KSM can >> help us scale beyond the standard numbers. >> >> As for the openvz link; I read that a few times but I don’t get any positive >> results using those methods. This leads me to believe (a) LXD does not >> support KSM or (b) the applications are not registering w/the KSM part of >> the kernel. >> >> I am going to run through some tests this week to see if I can get KSM >> working outside the LXD environment then try to replicate the same tests >> inside LXD. >> >> Thanks again for the feedback. >> >> >> >>> On Jun 4, 2017, at 6:15 PM, Fajar A. Nugraha wrote: >>> >>> On Sun, Jun 4, 2017 at 11:16 PM, Ron Kelley wrote: >>> (Reviving the thread about Container Scaling: >>> https://lists.linuxcontainers.org/pipermail/lxc-users/2016-May/011607.html) >>> >>> We have hit critical mass with LXD 2.12 and I need to get Kernel Samepage >>> Merging (KSM) working as soon as possible. All my research has come to a >>> dead-end, and I am reaching out to the group at large for suggestions. >>> >>> Background: We have 5 host servers - each running U16.04 >>> (4.4.0-57-generic), 8G RAM, 20G SWAP, and 50 containers (exact configs per >>> server - nginx and php 7). >>> >>> >>> Is this a cloud, or on-site setup? >>> >>> For cloud, there are a lot of options that could get you running with MUCH >>> more memory, which would save you lots of headaches getting KSM to work. My >>> favorite is EC2 spot instance on AWS. >>> >>> On another note, I now setup most of my hosts with no swap, since >>> performance plummets whenever swap is used. YMMV. >>> >>> >>> I am trying to get KSM working since each container is an identical replica >>> of the other (other than hostname/IP). I have read a ton of information on >>> the ‘net about Ubuntu and KSM, yet I can’t seem to get any pages to share >>> on the host. I am not sure if this is a KSM config issue or if LXD won’t >>> allow KSM between containers. >>> >>> >>> >>> >>> >>> Here is what I have done thus far: >>> -- >>> * Installed the ksmtuned utility and verified ksmd is running on each host. >>> * Created the ksm_preload and ksm-wrapper tools per this site (the >>> https://github.com/unbrice/ksm_preload). >>> * Created 50 identical Ubuntu 16.04 containers running nginx >>> * Modified the nginx startup script on each container to include the >>> ksm_preload.so library; no issues running nginx. >>> >>> (Note: since I could not find the ksm_preload library for Ubuntu, I had to >>> use the ksm-wrapper tool listed above) >>> >>> All the relevant files under /sys/kernel/mm/ksm still show 0 (pages_shared, >>> pages_sharing, etc) regardless of what I do. >>> >>> >>> Can any (@stgraber @brauner) confirm if KSM is supported with LXD? If so, >>> what is the “magic” to make it work? We really want to get 2-3x more sites >>> per container if possible. >>> >>> >>> Have you read https://openvz.org/KSM_(kernel_same-page_merging) ? Some info >>> might be relevant. For example, it mentions something which you did not >>> wrote: >>> >>> To start ksmd, issue >>> [root@HN ~]# echo 1 > /sys/kernel/mm/ksm/run >>> >>> Also the section about Tuning and Caveats. >>> >>> -- >>> Fajar >>> ___ >>> lxc-users mailing list >>> lxc-users@lists.linuxcontainers.org >>> http://lists.linuxcontainers.org/listinfo/lxc-users >> >> ___ >> lxc-users mailing list >> lxc-users@lists.linuxcontainers.org >> http://lists.linuxcontainers.org/listinfo/lxc-users > > > ___ lxc-users mailing list lxc-users@li
Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
On Monday, June 05, 2017 10:58 JST, "Fajar A. Nugraha" wrote: > On Mon, Jun 5, 2017 at 7:48 AM, Ron Kelley wrote: > > > > > As for the openvz link; I read that a few times but I don’t get any > > positive results using those methods. This leads me to believe (a) LXD > > does not support KSM or (b) the applications are not registering w/the KSM > > part of the kernel. > > > > > > Works for me here on a simple test, in 2 privileged containers. That's valuable info. I wonder if you also checked in unprivileged containers? -- Tomasz Chmielewski ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
Thanks. I tried the ksm_preload option but it does not seem to work. As I mentioned below, I had to creat the ksm_preload and ksm-wrapper tools manually since I could not find them in the “normal” Ubuntu repositories. Does anyone know if the ksm_preload libraries are native to Ubuntu (CentOS calls it the "ksm_preload-0.10-3.el6.x86_64.rpm” package)? > On Jun 4, 2017, at 9:18 PM, Tomasz Chmielewski wrote: > > KSM only works with applications which support it: > > KSM only operates on those areas of address space which an application > has advised to be likely candidates > for merging, by using the madvise(2) system call: int madvise(addr, > length, MADV_MERGEABLE. > > This means that doing: > >echo 1 > /sys/kernel/mm/ksm/run > > will be enough for KVM, but will not do anything for applications like bash, > nginx, apache, php-fpm and so on. > > > Please refer to "Enabling memory deduplication libraries in containers" on > https://openvz.org/KSM_(kernel_same-page_merging) - you will have to use > ksm_preload mentioned. I haven't personally used it with LXD. > > > Tomasz Chmielewski > https://lxadm.com > > > On Monday, June 05, 2017 09:48 JST, Ron Kelley wrote: > >> Thanks Fajar. >> >> This is on-site with our own physical servers, storage, etc. The goal is to >> get the most containers per server as possible. While our servers have lots >> of RAM, we need to come up with a long-term scaling plan and hope KSM can >> help us scale beyond the standard numbers. >> >> As for the openvz link; I read that a few times but I don’t get any positive >> results using those methods. This leads me to believe (a) LXD does not >> support KSM or (b) the applications are not registering w/the KSM part of >> the kernel. >> >> I am going to run through some tests this week to see if I can get KSM >> working outside the LXD environment then try to replicate the same tests >> inside LXD. >> >> Thanks again for the feedback. >> >> >> >>> On Jun 4, 2017, at 6:15 PM, Fajar A. Nugraha wrote: >>> >>> On Sun, Jun 4, 2017 at 11:16 PM, Ron Kelley wrote: >>> (Reviving the thread about Container Scaling: >>> https://lists.linuxcontainers.org/pipermail/lxc-users/2016-May/011607.html) >>> >>> We have hit critical mass with LXD 2.12 and I need to get Kernel Samepage >>> Merging (KSM) working as soon as possible. All my research has come to a >>> dead-end, and I am reaching out to the group at large for suggestions. >>> >>> Background: We have 5 host servers - each running U16.04 >>> (4.4.0-57-generic), 8G RAM, 20G SWAP, and 50 containers (exact configs per >>> server - nginx and php 7). >>> >>> >>> Is this a cloud, or on-site setup? >>> >>> For cloud, there are a lot of options that could get you running with MUCH >>> more memory, which would save you lots of headaches getting KSM to work. My >>> favorite is EC2 spot instance on AWS. >>> >>> On another note, I now setup most of my hosts with no swap, since >>> performance plummets whenever swap is used. YMMV. >>> >>> >>> I am trying to get KSM working since each container is an identical replica >>> of the other (other than hostname/IP). I have read a ton of information on >>> the ‘net about Ubuntu and KSM, yet I can’t seem to get any pages to share >>> on the host. I am not sure if this is a KSM config issue or if LXD won’t >>> allow KSM between containers. >>> >>> >>> >>> >>> >>> Here is what I have done thus far: >>> -- >>> * Installed the ksmtuned utility and verified ksmd is running on each host. >>> * Created the ksm_preload and ksm-wrapper tools per this site (the >>> https://github.com/unbrice/ksm_preload). >>> * Created 50 identical Ubuntu 16.04 containers running nginx >>> * Modified the nginx startup script on each container to include the >>> ksm_preload.so library; no issues running nginx. >>> >>> (Note: since I could not find the ksm_preload library for Ubuntu, I had to >>> use the ksm-wrapper tool listed above) >>> >>> All the relevant files under /sys/kernel/mm/ksm still show 0 (pages_shared, >>> pages_sharing, etc) regardless of what I do. >>> >>> >>> Can any (@stgraber @brauner) confirm if KSM is supported with LXD? If so, >>> what is the “magic” to make it work? We really want to get 2-3x more sites >>> per container if possible. >>> >>> >>> Have you read https://openvz.org/KSM_(kernel_same-page_merging) ? Some info >>> might be relevant. For example, it mentions something which you did not >>> wrote: >>> >>> To start ksmd, issue >>> [root@HN ~]# echo 1 > /sys/kernel/mm/ksm/run >>> >>> Also the section about Tuning and Caveats. >>> >>> -- >>> Fajar >>> ___ >>> lxc-users mailing list >>> lxc-users@lists.linuxcontainers.org >>> http://lists.linuxcontainers.org/listinfo/lxc-users >> >> ___ >> lxc-users mailing list >> lxc-users@lists.linuxcontai
Re: [lxc-users] ?==?utf-8?q? LXD and Kernel Samepage Merging (KSM)
KSM only works with applications which support it: KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call: int madvise(addr, length, MADV_MERGEABLE. This means that doing: echo 1 > /sys/kernel/mm/ksm/run will be enough for KVM, but will not do anything for applications like bash, nginx, apache, php-fpm and so on. Please refer to "Enabling memory deduplication libraries in containers" on https://openvz.org/KSM_(kernel_same-page_merging) - you will have to use ksm_preload mentioned. I haven't personally used it with LXD. Tomasz Chmielewski https://lxadm.com On Monday, June 05, 2017 09:48 JST, Ron Kelley wrote: > Thanks Fajar. > > This is on-site with our own physical servers, storage, etc. The goal is to > get the most containers per server as possible. While our servers have lots > of RAM, we need to come up with a long-term scaling plan and hope KSM can > help us scale beyond the standard numbers. > > As for the openvz link; I read that a few times but I don’t get any positive > results using those methods. This leads me to believe (a) LXD does not > support KSM or (b) the applications are not registering w/the KSM part of the > kernel. > > I am going to run through some tests this week to see if I can get KSM > working outside the LXD environment then try to replicate the same tests > inside LXD. > > Thanks again for the feedback. > > > > > On Jun 4, 2017, at 6:15 PM, Fajar A. Nugraha wrote: > > > > On Sun, Jun 4, 2017 at 11:16 PM, Ron Kelley wrote: > > (Reviving the thread about Container Scaling: > > https://lists.linuxcontainers.org/pipermail/lxc-users/2016-May/011607.html) > > > > We have hit critical mass with LXD 2.12 and I need to get Kernel Samepage > > Merging (KSM) working as soon as possible. All my research has come to a > > dead-end, and I am reaching out to the group at large for suggestions. > > > > Background: We have 5 host servers - each running U16.04 > > (4.4.0-57-generic), 8G RAM, 20G SWAP, and 50 containers (exact configs per > > server - nginx and php 7). > > > > > > Is this a cloud, or on-site setup? > > > > For cloud, there are a lot of options that could get you running with MUCH > > more memory, which would save you lots of headaches getting KSM to work. My > > favorite is EC2 spot instance on AWS. > > > > On another note, I now setup most of my hosts with no swap, since > > performance plummets whenever swap is used. YMMV. > > > > > > I am trying to get KSM working since each container is an identical replica > > of the other (other than hostname/IP). I have read a ton of information on > > the ‘net about Ubuntu and KSM, yet I can’t seem to get any pages to share > > on the host. I am not sure if this is a KSM config issue or if LXD won’t > > allow KSM between containers. > > > > > > > > > > > > Here is what I have done thus far: > > -- > > * Installed the ksmtuned utility and verified ksmd is running on each host. > > * Created the ksm_preload and ksm-wrapper tools per this site (the > > https://github.com/unbrice/ksm_preload). > > * Created 50 identical Ubuntu 16.04 containers running nginx > > * Modified the nginx startup script on each container to include the > > ksm_preload.so library; no issues running nginx. > > > > (Note: since I could not find the ksm_preload library for Ubuntu, I had to > > use the ksm-wrapper tool listed above) > > > > All the relevant files under /sys/kernel/mm/ksm still show 0 (pages_shared, > > pages_sharing, etc) regardless of what I do. > > > > > > Can any (@stgraber @brauner) confirm if KSM is supported with LXD? If so, > > what is the “magic” to make it work? We really want to get 2-3x more sites > > per container if possible. > > > > > > Have you read https://openvz.org/KSM_(kernel_same-page_merging) ? Some info > > might be relevant. For example, it mentions something which you did not > > wrote: > > > > To start ksmd, issue > > [root@HN ~]# echo 1 > /sys/kernel/mm/ksm/run > > > > Also the section about Tuning and Caveats. > > > > -- > > Fajar > > ___ > > lxc-users mailing list > > lxc-users@lists.linuxcontainers.org > > http://lists.linuxcontainers.org/listinfo/lxc-users > > ___ > lxc-users mailing list > lxc-users@lists.linuxcontainers.org > http://lists.linuxcontainers.org/listinfo/lxc-users -- Tomasz Chmielewski ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
