For now to overcome this issue, we've added a patch to an internal mirror of lxc debian repo https://anonscm.debian.org/git/pkg-lxc/lxc.git If you guys think adding a config parameter to not mask out the isolcpus in cpuset:/lxc would be welcome, please let me know. I can work on a patch and submit the same.
On Wed, Jun 6, 2018 at 9:34 AM Emmanuel Livingstone < [email protected]> wrote: > Hi, > > We use lxc along with qemu/kvm in our boxes to provide containers and VMs. > We rely on isolcpus to ensure that the kernel scheduler by default does not > schedule OS processes on the cores that we've reserved for customer's > virtual instances(containers/VMs). > As per these 2 PRs(https://github.com/lxc/lxc/pull/1282, > https://github.com/lxc/lxc/pull/1309), cpuset.cpus inside cpuset:/lxc is > configured to have only those cpus that are not present in isolcpus. This > effectively means setting lxc.cgroup.cpuset.cpus for a lxc container to any > cpu that is part of isolcpus setting causes the container not to start. > This is because cpuset cgroup allows cpuset.cpus to be set to a value that > is a subset of the parent's value. Quoting from > https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt > > For any cpuset, one can define child cpusets containing a subset > of the parents CPU and Memory Node resources. > > I've verified this behaviour > > root@fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test > root@fk-cloud-none-1080125:/home/iaas# echo 0-1 > > /sys/fs/cgroup/cpuset/test/cpuset.cpus > root@fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test/container > root@fk-cloud-none-1080125:/home/iaas# echo 0 > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > root@fk-cloud-none-1080125:/home/iaas# echo 1 > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > root@fk-cloud-none-1080125:/home/iaas# echo 2 > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > bash: echo: write error: Permission denied > > In our case, we have 48 core boxes on which we reserve 4 cores for OS > processes and remaining 44 cores for customers virtual instances. > It was working fine for us until we upgraded from debian 8 to debian 9, > i.e., kernel 3.16 to 4.9. In kernel 3.16 even if isolcpus was enabled, the > file /sys/devices/system/cpu/isolated was missing and is available in 4.9 > through this commit > https://github.com/torvalds/linux/commit/59f30abe94bff50636c8cad45207a01fdcb2ee49 > > I had come across a recommended way of overcoming this issue in this > comment > <https://discuss.linuxcontainers.org/t/lxc-lxd-2-0-11-isolcpu-limits-cpu/1845/6> > by stgraber where he had recommended that all system processes should be > put inside a restricted cpuset cgroup and the remaining cores to be used > for scheduling by lxc/lxd using a script. Is there alternate way in systemd > to do this without writing an ad-hoc startup script. > > Also since we use both libvirt(qemu/kvm) and lxc(without libvirt), I was > wondering why the behaviour of isolcpus is different in both these cases. >
_______________________________________________ lxc-users mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-users
