Re: [lxc-users] limits.memory - possible to set per group of containers?
On 2019-06-18 11:59, Stéphane Graber wrote: So we have plans to introduce project quotas which will allow placing such restrictions in a clean way through LXD. Until then you can manually tweak /sys/fs/cgroup/memory/lxc or /sys/fs/cgroup/memory/lxc.payload (depending on version of liblxc) as all containers reside under there and limits are hierarchical. It's pretty similar to what systemd would attempt to do except that liblxc/lxd bypass systemd's expected cgroup so placing the limit through systemd wouldn't work. I was just going to test systemd - because systemd-cgls shows these: 1) this is just the monitor process, so setting any limits on it won't have the desired effect: ├─lxc.monitor │ ├─shard01d │ │ └─26025 [lxc monitor] /var/snap/lxd/common/lxd/containers shard01d │ └─uni01-2019-06-18-01-09-11 │ └─11442 [lxc monitor] /var/snap/lxd/common/lxd/containers uni01-2019-06-18-01-09-11 2) Container processes seem to be all under lxc.payload tree: ─lxc.payload │ ├─shard01d │ │ ├─ 488 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid │ │ ├─ 1821 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d │ │ ├─ 3158 /lib/systemd/systemd-udevd │ │ ├─ 3878 /usr/sbin/rsyslogd -n │ │ ├─ 4592 /sbin/rpcbind -f -w (...) │ └─uni01-2019-06-18-01-09-11 │ ├─ 2875 /usr/bin/php cli.php Jiradaemon │ ├─ 3446 /usr/bin/php cli.php Jiradaemon │ ├─ 4022 pickup -l -t unix -u -c │ ├─ 4180 /usr/bin/php cli.php Jiradaemon (...) But since there isn't any "lxc.payload" systemd service - then yes, like you said - I'd need to dive into /sys/fs/cgroup/memory/ for now. Great to hear project quotas are in the plans! Tomasz Chmielewski ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] limits.memory - possible to set per group of containers?
On Tue, Jun 18, 2019 at 09:47:19AM +0900, Tomasz Chmielewski wrote: > Let's say I have a host with 32 GB RAM. > > To make sure the host is not affected by any weird memory consumption > patterns, I've set the following in the container: > > limits.memory: 29GB > > This works quite well - where previously, several processes with high memory > usage, forking rapidly (a forkbomb to test, but also i.e. a supervisor in > normal usage) running in the container could make the host very slow or even > unreachable - with the above setting, everything (on the host) is just > smooth no matter what the container does. > > However, that's just with one container. > > With two (or more) containers having "limits.memory: 29GB" set - it's easy > for each of them to consume i.e. 20 GB, leading to host unavailability. > > Is it possible to set a global, or per-container group "limits.memory: > 29GB"? > > For example, if I add "MemoryMax=29G" to > /etc/systemd/system/snap.lxd.daemon.service - would I achieve a desired > effect? > > > > Tomasz Chmielewski > https://lxadm.com So we have plans to introduce project quotas which will allow placing such restrictions in a clean way through LXD. Until then you can manually tweak /sys/fs/cgroup/memory/lxc or /sys/fs/cgroup/memory/lxc.payload (depending on version of liblxc) as all containers reside under there and limits are hierarchical. It's pretty similar to what systemd would attempt to do except that liblxc/lxd bypass systemd's expected cgroup so placing the limit through systemd wouldn't work. -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: PGP signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] limits.memory - possible to set per group of containers?
On 2019-06-18 11:34, Fajar A. Nugraha wrote: You could probably just use nested lxd instead: https://stgraber.org/2016/12/07/running-snaps-in-lxd-containers/ Set the outer container memory limit to 29GB, and put other containers inside that one. Yeah, but that's a bit "hackish" (to achieve the desired result - memory limit per group of containers; nothing wrong with nesting itself), and may be even hard to implement on existing setups. Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] limits.memory - possible to set per group of containers?
On Tue, Jun 18, 2019 at 7:47 AM Tomasz Chmielewski wrote: > Let's say I have a host with 32 GB RAM. > > To make sure the host is not affected by any weird memory consumption > patterns, I've set the following in the container: > >limits.memory: 29GB > > This works quite well - where previously, several processes with high > memory usage, forking rapidly (a forkbomb to test, but also i.e. a > supervisor in normal usage) running in the container could make the host > very slow or even unreachable - with the above setting, everything (on > the host) is just smooth no matter what the container does. > > However, that's just with one container. > > With two (or more) containers having "limits.memory: 29GB" set - it's > easy for each of them to consume i.e. 20 GB, leading to host > unavailability. > > Is it possible to set a global, or per-container group "limits.memory: > 29GB"? > > For example, if I add "MemoryMax=29G" to > /etc/systemd/system/snap.lxd.daemon.service - would I achieve a desired > effect? > > > You could probably just use nested lxd instead: https://stgraber.org/2016/12/07/running-snaps-in-lxd-containers/ Set the outer container memory limit to 29GB, and put other containers inside that one. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] limits.memory - possible to set per group of containers?
Let's say I have a host with 32 GB RAM. To make sure the host is not affected by any weird memory consumption patterns, I've set the following in the container: limits.memory: 29GB This works quite well - where previously, several processes with high memory usage, forking rapidly (a forkbomb to test, but also i.e. a supervisor in normal usage) running in the container could make the host very slow or even unreachable - with the above setting, everything (on the host) is just smooth no matter what the container does. However, that's just with one container. With two (or more) containers having "limits.memory: 29GB" set - it's easy for each of them to consume i.e. 20 GB, leading to host unavailability. Is it possible to set a global, or per-container group "limits.memory: 29GB"? For example, if I add "MemoryMax=29G" to /etc/systemd/system/snap.lxd.daemon.service - would I achieve a desired effect? Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] [Fwd: Re: unprivileged Debian Buster container on Debian Buster host fail to start: no cgroups, no controllers]
Sorry, I don't understand why it doesn't work... Same config is
successful with my buster host.
Did you try it on a fresh buster install? Did you tweak things around?
Xavier
Le 17/06/2019 à 18:47, Lukas Pirl a écrit :
Ping… no ideas on this?
Forwarded Message
From: Lukas Pirl
To: lxc-users@lists.linuxcontainers.org
Subject: Re: [lxc-users] unprivileged Debian Buster container on Debian Buster
host fail to start: no cgroups, no controllers
Date: Wed, 29 May 2019 00:11:29 +0200
On Tue, 2019-05-28 21:50 +0200, Xavier Gendre wrote as excerpted:
Hello Lukas,
unprivileged buster containers on a buster host run like a charm. Your
config includes a lot of stuff that are not suited for an unprivileged
container (apparmor, ...). First, you should try with a simpler
configuration file as the following one.
---%<--%<--%<---
lxc.idmap = u 0 165536 65536
lxc.idmap = g 0 165536 65536
lxc.net.0.type = empty
--->%-->%-->%---
Then,
lxc-create -n test -f config.file -t download -- --dist debian --release
buster --arch amd64
lxc-start -n test
Thanks for your reply, Xavier. No luck. This is what I have/see now:
$ egrep -v '^#' test.config
lxc.net.0.type = empty
lxc.idmap = u 0 165536 65536
lxc.idmap = g 0 165536 65536
$ lxc-create -n test -f test.config -t download -- --dist debian \
--release buster --arch amd64
Permission denied - Failed to open ttyPermission denied - Failed to open
ttyPermission denied - Failed to open ttycat: /proc/1/uid_map: No such file
or
directory
Using image from local cache
Unpacking the rootfs
---
You just created a Debian buster amd64 (20190509_05:24) container.
To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
$ ll /dev/tty
crw-rw-rw- 1 root tty 5, 0 2019-05-28 23:41:32 /dev/tty /dev/tty[0-9]*
crw--w 1 root tty 4, 0 2019-05-28 11:15:54 /dev/tty0
… # all tty have the same permissions
$
Should the user ``lxc`` be in the group ``tty``?
Apparently, ``lxc-create`` queries ``/proc/1/{u,g}id_map`` which it is not
allowed to (proc mounted with hidepid=2) instead of
``/proc/self/{u,g}id_map``, no?
$ cat /proc/self/{u,g}id_map
0 0 4294967295
0 0 4294967295
$ egrep -v '^#' test/config
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64
lxc.idmap = u 0 165536 65536
lxc.idmap = g 0 165536 65536
lxc.rootfs.path = dir:/home/lxc/test/rootfs
lxc.uts.name = test
lxc.net.0.type = empty
$ lxc-start -n test -F
Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
[!!] Failed to mount API filesystems.
Exiting PID 1...
$
Okay, let's first try to make ``/dev/tty`` read-writable for the user
``lxc``
(just for testing, ignoring if that is appropriate or not):
$ chmod a+rw /dev/tty /dev/tty[0-9]*
$ ll /dev/tty /dev/tty[0-9]*
crw-rw-rw- 1 root tty 5, 0 2019-05-28 23:41:32 /dev/tty
crw-rw-rw- 1 root tty 4, 0 2019-05-28 11:15:54 /dev/tty0
…
$ lxc-destroy -n test
$ lxc-create -n test -f test.config -t download -- --dist debian --release
buster --arch amd64
Permission denied - Failed to open ttyPermission denied - Failed to open
ttyPermission denied - Failed to open ttycat: /proc/1/uid_map: No such file
or
directory
Using image from local cache
Unpacking the rootfs
---
You just created a Debian buster amd64 (20190509_05:24) container.
To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.
$
The error when starting persist as above.
Hm, okay let's mount ``/proc`` with ``hidepid=0``. Destroy and create: The
tty
error persists, the cat error is gone. Start: errors persist as above.
Alright, last thing, let's try to remove the includes from the config file
(changes until now still in place):
$ egrep -v '^#' test/config
lxc.arch = linux64
lxc.idmap = u 0 165536 65536
lxc.idmap = g 0 165536 65536
lxc.rootfs.path = dir:/home/lxc/test/rootfs
lxc.uts.name = test
lxc.net.0.type = empty
$ lxc-start -n test -F
Failed to lookup module alias 'autofs4': Function not implemented
Failed to lookup module alias 'unix': Function not implemented
Failed to mount sysfs at /sys: Operation not permitted
Failed to mount proc at /proc: Operation not permitted
Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
[!!] Failed to mount API filesystems.
Exiting PID 1...
My lxc.conf does not include anything exotic as well:
$ egrep -v '^#' .config/lxc/lxc.conf
lxc.lxcpath = /home/lxc
Any ideas?
Cheers,
Lukas
Le 28/05/2019 à 15:54, Lukas Pirl a écrit :
Dear all,
first, thanks for the friendly and supportive help you all provide in
issue
trackers, on mailing lists, etc. – it is very helpful to find all this
online.
However, I struggle to run unprivileged (Debian Buster) containers (on a
Debian Buster host). LXC does not seem to mount the cgroup mount points
for
the container, thus the container's systemd tries to mount those and
fails
due
to
[lxc-users] [Fwd: Re: unprivileged Debian Buster container on Debian Buster host fail to start: no cgroups, no controllers]
Ping… no ideas on this?
Forwarded Message
From: Lukas Pirl
To: lxc-users@lists.linuxcontainers.org
Subject: Re: [lxc-users] unprivileged Debian Buster container on Debian Buster
host fail to start: no cgroups, no controllers
Date: Wed, 29 May 2019 00:11:29 +0200
> On Tue, 2019-05-28 21:50 +0200, Xavier Gendre wrote as excerpted:
> > Hello Lukas,
> >
> > unprivileged buster containers on a buster host run like a charm. Your
> > config includes a lot of stuff that are not suited for an unprivileged
> > container (apparmor, ...). First, you should try with a simpler
> > configuration file as the following one.
> >
> > ---%<--%<--%<---
> > lxc.idmap = u 0 165536 65536
> > lxc.idmap = g 0 165536 65536
> > lxc.net.0.type = empty
> > --->%-->%-->%---
> >
> > Then,
> > lxc-create -n test -f config.file -t download -- --dist debian --release
> > buster --arch amd64
> > lxc-start -n test
>
> Thanks for your reply, Xavier. No luck. This is what I have/see now:
>
> $ egrep -v '^#' test.config
> lxc.net.0.type = empty
> lxc.idmap = u 0 165536 65536
> lxc.idmap = g 0 165536 65536
>
> $ lxc-create -n test -f test.config -t download -- --dist debian \
> --release buster --arch amd64
> Permission denied - Failed to open ttyPermission denied - Failed to open
> ttyPermission denied - Failed to open ttycat: /proc/1/uid_map: No such file
> or
> directory
> Using image from local cache
> Unpacking the rootfs
>
> ---
> You just created a Debian buster amd64 (20190509_05:24) container.
>
> To enable SSH, run: apt install openssh-server
> No default root or user password are set by LXC.
>
> $ ll /dev/tty
> crw-rw-rw- 1 root tty 5, 0 2019-05-28 23:41:32 /dev/tty /dev/tty[0-9]*
> crw--w 1 root tty 4, 0 2019-05-28 11:15:54 /dev/tty0
> … # all tty have the same permissions
>
> $
>
> Should the user ``lxc`` be in the group ``tty``?
>
> Apparently, ``lxc-create`` queries ``/proc/1/{u,g}id_map`` which it is not
> allowed to (proc mounted with hidepid=2) instead of
> ``/proc/self/{u,g}id_map``, no?
>
> $ cat /proc/self/{u,g}id_map
> 0 0 4294967295
> 0 0 4294967295
>
> $ egrep -v '^#' test/config
> lxc.include = /usr/share/lxc/config/common.conf
> lxc.include = /usr/share/lxc/config/userns.conf
> lxc.arch = linux64
> lxc.idmap = u 0 165536 65536
> lxc.idmap = g 0 165536 65536
> lxc.rootfs.path = dir:/home/lxc/test/rootfs
> lxc.uts.name = test
> lxc.net.0.type = empty
>
> $ lxc-start -n test -F
> Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
> [!!] Failed to mount API filesystems.
> Exiting PID 1...
> $
>
> Okay, let's first try to make ``/dev/tty`` read-writable for the user
> ``lxc``
> (just for testing, ignoring if that is appropriate or not):
>
> $ chmod a+rw /dev/tty /dev/tty[0-9]*
> $ ll /dev/tty /dev/tty[0-9]*
> crw-rw-rw- 1 root tty 5, 0 2019-05-28 23:41:32 /dev/tty
> crw-rw-rw- 1 root tty 4, 0 2019-05-28 11:15:54 /dev/tty0
> …
>
> $ lxc-destroy -n test
> $ lxc-create -n test -f test.config -t download -- --dist debian --release
> buster --arch amd64
> Permission denied - Failed to open ttyPermission denied - Failed to open
> ttyPermission denied - Failed to open ttycat: /proc/1/uid_map: No such file
> or
> directory
> Using image from local cache
> Unpacking the rootfs
>
> ---
> You just created a Debian buster amd64 (20190509_05:24) container.
>
> To enable SSH, run: apt install openssh-server
> No default root or user password are set by LXC.
> $
>
> The error when starting persist as above.
>
> Hm, okay let's mount ``/proc`` with ``hidepid=0``. Destroy and create: The
> tty
> error persists, the cat error is gone. Start: errors persist as above.
>
> Alright, last thing, let's try to remove the includes from the config file
> (changes until now still in place):
>
> $ egrep -v '^#' test/config
> lxc.arch = linux64
> lxc.idmap = u 0 165536 65536
> lxc.idmap = g 0 165536 65536
> lxc.rootfs.path = dir:/home/lxc/test/rootfs
> lxc.uts.name = test
> lxc.net.0.type = empty
>
> $ lxc-start -n test -F
> Failed to lookup module alias 'autofs4': Function not implemented
> Failed to lookup module alias 'unix': Function not implemented
> Failed to mount sysfs at /sys: Operation not permitted
> Failed to mount proc at /proc: Operation not permitted
> Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
> [!!] Failed to mount API filesystems.
> Exiting PID 1...
>
> My lxc.conf does not include anything exotic as well:
>
> $ egrep -v '^#' .config/lxc/lxc.conf
> lxc.lxcpath = /home/lxc
>
> Any ideas?
>
> Cheers,
>
> Lukas
>
> > Le 28/05/2019 à 15:54, Lukas Pirl a écrit :
> > > Dear all,
> > >
> > > first, thanks for the friendly and supportive help you all provide in
> > > issue
> > > trackers, on mailing lists, etc. – it is very helpful to find all this
> > > online.
> > >
> > > However, I struggle to run unprivileged (Debian Buster) containers (on a
> > > Debian Buster host). LXC
