Quoting Ben Warren (b...@skyportsystems.com): > Hi Serge, > > On May 8, 2017, at 11:55 AM, Serge E. Hallyn <se...@hallyn.com> wrote: > > > > Quoting Ben Warren (b...@skyportsystems.com > > <mailto:b...@skyportsystems.com>): > >> Hi Serge, > >> > >>> On May 4, 2017, at 9:00 AM, Serge E. Hallyn <serge at hallyn.com> wrote: > >>> > >>> Quoting Ben Warren (ben at skyportsystems.com): > >>>> Hi, > >>>> > >>>> I’m stuck with Ubuntu 14.04 for now and would like to be able to run > >>>> unprivileged containers that are systemd-based. I’ve found lots of > >>>> examples of problems that are close, but nothing exactly matches. I got > >>>> the lxc packages from trusty-backports. > >>>> > >>>> Versions: > >>>> > >>>> ben at ben-sc:~$ lxc-ls --version > >>>> 2.0.7 > >>>> ben at ben-sc:~$ cat /etc/lsb-release > >>>> DISTRIB_ID=Ubuntu > >>>> DISTRIB_RELEASE=14.04 > >>>> DISTRIB_CODENAME=trusty > >>>> DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS" > >>>> > >>>> To keep it simple, I created an unprivileged container of ‘trusty’ using > >>>> the download method: > >>>> > >>>> ben at ben-sc:~$ lxc-create -n cd-build -t download > >>>> > >>>> > >>>> When I try to start the container, it won’t work: > >>>> > >>>> ben at ben-sc:~$ lxc-start -n cd-build -d --logfile cd-build.log > >>>> lxc-start: tools/lxc_start.c: main: 366 The container failed to start. > >>>> lxc-start: tools/lxc_start.c: main: 368 To get more details, run the > >>>> container in foreground mode. > >>>> lxc-start: tools/lxc_start.c: main: 370 Additional information can be > >>>> obtained by setting the --logfile and --logpriority options. > >>>> > >>>> Logfile contents: > >>>> > >>>> lxc-start 20170503225525.382 ERROR lxc_cgfsng - > >>>> cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not > >>>> permitted - Error remounting > >>>> /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only > >>> > >>> This is odd, not the error I would have expected. > >>> > >>> Can you tell me the exact version and from which ppa? > >>> > >> $ dpkg -s lxc > >> Package: lxc > >> Status: install ok installed > >> Priority: extra > >> Section: oldlibs > >> Installed-Size: 77 > >> Maintainer: Ubuntu Developers <ubuntu-devel-disc...@lists.ubuntu.com> > >> Architecture: all > >> Version: 2.0.7-0ubuntu1~14.04.1 > >> Depends: lxc1 (>= 2.0.7-0ubuntu1~14.04.1) > >> > >> I got it from here: > >> > >> http://us.archive.ubuntu.com/ubuntu/ trusty-backports > >> > >> Here’s what gets installed: > >> > >> $ sudo apt-get install -t trusty-backports lxc > >> Reading package lists... Done > >> Building dependency tree > >> Reading state information... Done > >> The following extra packages will be installed: > >> bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info > >> euca2ools > >> libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc-common lxc-templates lxc1 > >> python-distro-info python-requestbuilder python3-lxc uidmap > >> Suggested packages: > >> shunit2 gnutls-bin btrfs-tools lvm2 lxctl > >> Recommended packages: > >> lxcfs libpam-cgfs > >> The following NEW packages will be installed: > >> bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info > >> euca2ools > >> libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc lxc-common lxc-templates > >> lxc1 python-distro-info python-requestbuilder python3-lxc uidmap > >> > >> As for the overall environment, this is a VM that was originally set up > >> almost 3 years ago, and as a lab machine has only been piecemeal updated > >> over time as needed. The problem is that I have probably a hundred > >> identical instances and am concerned that the package dependencies are > >> maybe not quite right. I’m certainly willing to update whatever > >> individual packages are necessary to get this going. I have the VM > >> snapshotted before trying this, so it’s trivial to reproduce. > >> > >>> Is there anything in syslog about the failed mount? > >>> > >> This is all I see. It’s at lxc install time, now when trying to start the > >> container: > >> > >> May 7 21:01:01 ben-sc kernel: [ 103.486718] type=1400 > >> audit(1494216061.420:68): apparmor="STATUS" operation="profile_load" > >> profile="unconfined" name="lxc-container-default" pid=5801 > >> comm="apparmor_parser" > >> May 7 21:01:01 ben-sc kernel: [ 103.486925] type=1400 > >> audit(1494216061.420:69): apparmor="STATUS" operation="profile_load" > >> profile="unconfined" name="lxc-container-default-cgns" pid=5801 > >> comm="apparmor_parser" > >> May 7 21:01:01 ben-sc kernel: [ 103.487100] type=1400 > >> audit(1494216061.420:70): apparmor="STATUS" operation="profile_load" > >> profile="unconfined" name="lxc-container-default-with-mounting" pid=5801 > >> comm="apparmor_parser" > >> May 7 21:01:01 ben-sc kernel: [ 103.487292] type=1400 > >> audit(1494216061.420:71): apparmor="STATUS" operation="profile_load" > >> profile="unconfined" name="lxc-container-default-with-nesting" pid=5801 > >> comm="apparmor_parser" > >> May 7 21:01:01 ben-sc kernel: [ 103.519003] type=1400 > >> audit(1494216061.452:72): apparmor="STATUS" operation="profile_load" > >> profile="unconfined" name="/usr/bin/lxc-start" pid=5835 > >> comm="apparmor_parser" > >> > >>> You might try some of the other cgroup auto-mount settings (see > >>> lxc.container.conf(5)0, maybe > >>> > >>> lxc.mount.auto = cgroup:rw > >>> > >> I tried that, and get: > >> > >> lxc-start 20170508041726.340 ERROR lxc_cgfsng - > >> cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not > >> permitted - Error remounting > >> /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only > >> lxc-start 20170508041726.340 ERROR lxc_conf - > >> conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error > >> mounting /sys/fs/cgroup > >> lxc-start 20170508041726.340 ERROR lxc_conf - > >> conf.c:lxc_setup:3885 - failed to setup the automatic mounts for 'cd-build' > >> lxc-start 20170508041726.340 ERROR lxc_start - > >> start.c:do_start:811 - Failed to setup container "cd-build". > >> lxc-start 20170508041726.340 ERROR lxc_sync - > >> sync.c:__sync_wait:57 - An error occurred in another process (expected > >> sequence number 3) > >> lxc-start 20170508041726.340 ERROR lxc_start - > >> start.c:__lxc_start:1346 - Failed to spawn container "cd-build". > >> > >>>> lxc-start 20170503225525.382 ERROR lxc_conf - > >>>> conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error > >>>> mounting /sys/fs/cgroup > >>>> lxc-start 20170503225525.382 ERROR lxc_conf - > >>>> conf.c:lxc_setup:3885 - failed to setup the automatic mounts for > >>>> 'cd-build' > >>>> lxc-start 20170503225525.382 ERROR lxc_start - > >>>> start.c:do_start:811 - Failed to setup container "cd-build". > >>>> lxc-start 20170503225525.382 ERROR lxc_sync - > >>>> sync.c:__sync_wait:57 - An error occurred in another process (expected > >>>> sequence number 3) > >>>> lxc-start 20170503225525.382 ERROR lxc_start - > >>>> start.c:__lxc_start:1346 - Failed to spawn container "cd-build". > >>>> lxc-start 20170503225530.922 ERROR lxc_start_ui - > >>>> tools/lxc_start.c:main:366 - The container failed to start. > >>>> lxc-start 20170503225530.923 ERROR lxc_start_ui - > >>>> tools/lxc_start.c:main:368 - To get more details, run the container in > >>>> foreground mode. > >>>> lxc-start 20170503225530.923 ERROR lxc_start_ui - > >>>> tools/lxc_start.c:main:370 - Additional information can be obtained by > >>>> setting the --logfile and --logpriority options. > >>>> > >>>> Also: > >>>> > >>>> ———————————— > >>>> > >>>> ben at ben-sc:~$ cat /proc/self/cgroup > >>>> 12:name=dsystemd:/ > >>>> 11:name=systemd:/user/1001.user/c2.session > >>>> 10:hugetlb:/user/1001.user/c2.session > >>>> 9:perf_event:/user/1001.user/c2.session > >>>> 8:blkio:/user/1001.user/c2.session > >>>> 7:freezer:/user/1001.user/c2.session > >>>> 6:devices:/user/1001.user/c2.session > >>>> 5:memory:/user/1001.user/c2.session > >>>> 4:cpuacct:/user/1001.user/c2.session > >>>> 3:cpu:/user/1001.user/c2.session > >>>> 2:cpuset:/ > > > > What the heck is cuasing this? When I log in on a trusty+backports system, > > I get: > > > > ubuntu@trusty:~$ cat /proc/self/cgroup > > 11:hugetlb:/user/1000.user/2.session > > 10:perf_event:/user/1000.user/2.session > > 9:blkio:/user/1000.user/2.session > > 8:freezer:/user/1000.user/2.session > > 7:devices:/user/1000.user/2.session > > 6:memory:/user/1000.user/2.session > > 5:cpuacct:/user/1000.user/2.session > > 4:cpu:/user/1000.user/2.session > > 3:cpuset:/user/1000.user/2.session > > 2:name=systemd:/user/1000.user/2.session > > > > (on second login) > > > > ubuntu@trusty:~$ dpkg -l | egrep -e "(cgroup|lxc|cgfs)" > > ii cgmanager 0.24-0ubuntu7.5 > > amd64 Central cgroup manager daemon > > ii cgroup-lite 1.11~ubuntu14.04.2 > > all Light-weight package to set up cgroups at system boot > > ii libcgmanager0:amd64 0.24-0ubuntu7.5 > > amd64 Central cgroup manager daemon (client library) > > ii liblxc1 2.0.7-0ubuntu1~14.04.1 > > amd64 Linux Containers userspace tools (library) > > ii libpam-cgfs 2.0.6-0ubuntu1~14.04.1 > > amd64 PAM module for managing cgroups for LXC > > ii lxc 2.0.7-0ubuntu1~14.04.1 > > all Transitional package for lxc1 > > ii lxc-common 2.0.7-0ubuntu1~14.04.1 > > amd64 Linux Containers userspace tools (common tools) > > ii lxc-templates 2.0.7-0ubuntu1~14.04.1 > > amd64 Linux Containers userspace tools (templates) > > ii lxc1 2.0.7-0ubuntu1~14.04.1 > > amd64 Linux Containers userspace tools > > ii lxcfs 2.0.6-0ubuntu1~14.04.1 > > amd64 FUSE based filesystem for LXC > > ii python3-lxc 2.0.7-0ubuntu1~14.04.1 > > amd64 Linux Containers userspace tools (Python 3.x bindings) > > > > Ah, now if I purge cgmanager, then upon login I get: > > > > ubuntu@trusty:~$ cat /proc/self/cgroup > > 11:name=systemd:/user/1000.user/1.session > > 10:hugetlb:/user/1000.user/1.session > > 9:perf_event:/user/1000.user/1.session > > 8:blkio:/user/1000.user/1.session > > 7:freezer:/user/1000.user/1.session > > 6:devices:/user/1000.user/1.session > > 5:memory:/user/1000.user/1.session > > 4:cpuacct:/user/1000.user/1.session > > 3:cpu:/user/1000.user/1.session > > 2:cpuset:/ > > > > which looks more like yours, but my container still starts... > > I’ve made some progress, but still don’t fully know what’s going on. When I > build lxc from source (top-of-tree github.com:lxc/lxc) and compile with full > cgmanager and libcap support, the generated binaries work, and I can start > not only my ‘trusty’ container, but also ones that are farther from the host, > such as ‘delian-stretch’, which is systemd-based. > > The difference I see in the log is which cgroup driver is used. > When I build using the binaries from ’trusty-backports’, I see this: > lxc-start 20170509054154.989 INFO lxc_cgroup - > cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for > cd-build > > When using the binaries I built from source, I see this: > lxc-start 20170509053256.861 INFO lxc_cgroup - > cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgmanager initing for cd-build > > Assuming cgmanager support is compiled in to the ‘trusty-backports’ version, > the following code determines if the cgmanager driver is used (non-NULL > return code means cgmanager is to be used): > > struct cgroup_ops *cgm_ops_init(void) > { > check_supports_multiple_controllers(-1); > if (!collect_subsystems()) > return NULL; > > if (api_version < CGM_SUPPORTS_MULT_CONTROLLERS) > cgm_all_controllers_same = false; > > // if root, try to escape to root cgroup > if (geteuid() == 0 && !cgm_escape(NULL)) { > free_subsystems(); > return NULL; > } > > return &cgmanager_ops; > } > > I have no context for how any of this is dependent on the environment, > although I’m sure you do :)
Mine were starting with cgfsng which yours is using also, so you don't *need* the cgmanager driver. But I'm pretty sure that if you build your own with it enabled it will work. Is it possible that you have lxc.cgroup.use set in /etc/lxc/lxc.conf or in ~/.config/lxc/lxc.conf, and that it includes 'cpu'? If so, assuming you don't need it, removing cpu should work around this failure. Does adding ',cpu" to the end of the pam_cgfs.so line in /etc/pam.d/common-session help? The other thing is back to your core problem - why is /sys/fs/cgroup/cpu not remountable read-only? It may be related to why you have a dsystemd cgroup hierarchy. Do you recall setting that up and/or why it's there? Can you show the contents of /proc/1/mounts and /proc/self/mounts on the host and a fresh host boot log? _______________________________________________ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
