On Thu, Jun 28, 2018 at 11:07:38AM +0530, Emmanuel Livingstone wrote: > For now to overcome this issue, we've added a patch to an internal mirror > of lxc debian repo https://anonscm.debian.org/git/pkg-lxc/lxc.git > If you guys think adding a config parameter to not mask out the isolcpus in > cpuset:/lxc would be welcome, please let me know. I can work on a patch and > submit the same.
Hm, the main problem with this is that there's no nice way of adding a config item that would allow one to configure this. Otherwise I'd be fine with it. > > On Wed, Jun 6, 2018 at 9:34 AM Emmanuel Livingstone < > [email protected]> wrote: > > > Hi, > > > > We use lxc along with qemu/kvm in our boxes to provide containers and VMs. > > We rely on isolcpus to ensure that the kernel scheduler by default does not > > schedule OS processes on the cores that we've reserved for customer's > > virtual instances(containers/VMs). > > As per these 2 PRs(https://github.com/lxc/lxc/pull/1282, > > https://github.com/lxc/lxc/pull/1309), cpuset.cpus inside cpuset:/lxc is > > configured to have only those cpus that are not present in isolcpus. This > > effectively means setting lxc.cgroup.cpuset.cpus for a lxc container to any > > cpu that is part of isolcpus setting causes the container not to start. > > This is because cpuset cgroup allows cpuset.cpus to be set to a value that > > is a subset of the parent's value. Quoting from > > https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt > > > > For any cpuset, one can define child cpusets containing a subset > > of the parents CPU and Memory Node resources. > > > > I've verified this behaviour > > > > root@fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test > > root@fk-cloud-none-1080125:/home/iaas# echo 0-1 > > > /sys/fs/cgroup/cpuset/test/cpuset.cpus > > root@fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test/container > > root@fk-cloud-none-1080125:/home/iaas# echo 0 > > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > > root@fk-cloud-none-1080125:/home/iaas# echo 1 > > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > > root@fk-cloud-none-1080125:/home/iaas# echo 2 > > > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus > > bash: echo: write error: Permission denied > > > > In our case, we have 48 core boxes on which we reserve 4 cores for OS > > processes and remaining 44 cores for customers virtual instances. > > It was working fine for us until we upgraded from debian 8 to debian 9, > > i.e., kernel 3.16 to 4.9. In kernel 3.16 even if isolcpus was enabled, the > > file /sys/devices/system/cpu/isolated was missing and is available in 4.9 > > through this commit > > https://github.com/torvalds/linux/commit/59f30abe94bff50636c8cad45207a01fdcb2ee49 > > > > I had come across a recommended way of overcoming this issue in this > > comment > > <https://discuss.linuxcontainers.org/t/lxc-lxd-2-0-11-isolcpu-limits-cpu/1845/6> > > by stgraber where he had recommended that all system processes should be > > put inside a restricted cpuset cgroup and the remaining cores to be used > > for scheduling by lxc/lxd using a script. Is there alternate way in systemd > > to do this without writing an ad-hoc startup script. > > > > Also since we use both libvirt(qemu/kvm) and lxc(without libvirt), I was > > wondering why the behaviour of isolcpus is different in both these cases. > > > _______________________________________________ > lxc-users mailing list > [email protected] > http://lists.linuxcontainers.org/listinfo/lxc-users _______________________________________________ lxc-users mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-users
