> On May 9, 2017, at 8:10 AM, Serge E. Hallyn <se...@hallyn.com> wrote:
> 
<snip>
> 
>> 
>> I’ve made some progress, but still don’t fully know what’s going on.  When I 
>> build lxc from source (top-of-tree github.com:lxc/lxc) and compile with full 
>> cgmanager and libcap support, the generated binaries work, and I can start 
>> not only my ‘trusty’ container, but also ones that are farther from the 
>> host, such as ‘delian-stretch’, which is systemd-based.
>> 
>> The difference I see in the log is which cgroup driver is used.
>> When I build using the binaries from ’trusty-backports’, I see this:
>>      lxc-start 20170509054154.989 INFO     lxc_cgroup - 
>> cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for 
>> cd-build
>> 
>> When using the binaries I built from source, I see this:
>>      lxc-start 20170509053256.861 INFO     lxc_cgroup - 
>> cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgmanager initing for 
>> cd-build
>> 
>> Assuming cgmanager support is compiled in to the ‘trusty-backports’ version, 
>> the following code determines if the cgmanager driver is used (non-NULL 
>> return code means cgmanager is to be  used):
>> 
>> struct cgroup_ops *cgm_ops_init(void)
>> {
>>      check_supports_multiple_controllers(-1);
>>      if (!collect_subsystems())
>>              return NULL;
>> 
>>      if (api_version < CGM_SUPPORTS_MULT_CONTROLLERS)
>>              cgm_all_controllers_same = false;
>> 
>>      // if root, try to escape to root cgroup
>>      if (geteuid() == 0 && !cgm_escape(NULL)) {
>>              free_subsystems();
>>              return NULL;
>>      }
>> 
>>      return &cgmanager_ops;
>> }
>> 
>> I have no context for how any of this is dependent on the environment, 
>> although I’m sure you do :)
> 
> Mine were starting with cgfsng which yours is using also, so you don't *need*
> the cgmanager driver.  But I'm pretty sure that if you build your own with
> it enabled it will work.
> 
> Is it possible that you have lxc.cgroup.use set in /etc/lxc/lxc.conf or in
> ~/.config/lxc/lxc.conf, and that it includes 'cpu'?  If so, assuming you
> don't need it, removing cpu should work around this failure.
> 
Neither of these files is present.  This is it for config:

ben@ben-sc:~/tmp/lxc/src$ cat /etc/lxc/default.conf 
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
ben@ben-sc:~/tmp/lxc/src$ cat ~/.config/lxc/default.conf 
lxc.id_map = u 0 165536 65536
lxc.id_map = g 0 165536 65536

> Does adding ',cpu" to the end of the pam_cgfs.so line in 
> /etc/pam.d/common-session
> help?
> 
I added like this:

session optional    pam_cgfs.so -c freezer,memory,cpu,name=systemd

but it doesn’t seem to make a difference
> The other thing is back to your core problem - why is /sys/fs/cgroup/cpu not
> remountable read-only?  It may be related to why you have a dsystemd cgroup
> hierarchy.  Do you recall setting that up and/or why it's there?  Can you
> show the contents of /proc/1/mounts and /proc/self/mounts on the host and a
> fresh host boot log?

I think the dsystemd thing was left over from me trying something else.  It’s 
not there now, after reverting to before any LXC installation and just 
installing the backports version of lxc.

Here’s the current state.  If I run ‘lxc-start’ runtime-linked against the 
‘back ports’ shared libraries I get this message:
      lxc-start 20170509161114.691 INFO     lxc_conf - 
conf.c:mount_file_entries:1985 - mount points have been setup
      lxc-start 20170509161114.691 ERROR    lxc_cgfsng - 
cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted 
- Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpuset read-only
      lxc-start 20170509161114.691 ERROR    lxc_conf - 
conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting 
/sys/fs/cgroup

If I change LD_LIBRARY_PATH to use the .so that I built, the container start as 
previously mentioned, using cgmanager.

ben@ben-sc:~$ cat /proc/self/cgroup
11:name=systemd:/user/1001.user/c2.session
10:perf_event:/user/1001.user/c2.session
9:memory:/user/1001.user/c2.session
8:hugetlb:/user/1001.user/c2.session
7:freezer:/user/1001.user/c2.session
6:devices:/user/1001.user/c2.session
5:cpuacct:/user/1001.user/c2.session
4:blkio:/user/1001.user/c2.session
3:cpu:/user/1001.user/c2.session
2:cpuset:/user/1001.user/c2.session

ben@ben-sc:~$ cat /proc/1/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4073948k,nr_inodes=1018487,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=816968k,mode=755 0 0
/dev/disk/by-uuid/0fdaee58-1394-4338-9eed-95ab207f0de6 / ext4 
rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgmfs /run/cgmanager/fs tmpfs rw,relatime,size=100k,mode=755 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup 
rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct
 0 0
cgroup /sys/fs/cgroup/memory cgroup 
rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory 
0 0
cgroup /sys/fs/cgroup/devices cgroup 
rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices
 0 0
cgroup /sys/fs/cgroup/freezer cgroup 
rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer
 0 0
cgroup /sys/fs/cgroup/blkio cgroup 
rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup 
rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event
 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup 
rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb
 0 0
name=systemd /sys/fs/cgroup/systemd cgroup 
rw,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd
 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc 
rw,nosuid,nodev,noexec,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs 
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
gvfsd-fuse /run/user/1001/gvfs fuse.gvfsd-fuse 
rw,nosuid,nodev,relatime,user_id=1001,group_id=1001 0 0

ben@ben-sc:~$ cat /proc/self/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4073948k,nr_inodes=1018487,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=816968k,mode=755 0 0
/dev/disk/by-uuid/0fdaee58-1394-4338-9eed-95ab207f0de6 / ext4 
rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgmfs /run/cgmanager/fs tmpfs rw,relatime,size=100k,mode=755 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup 
rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct
 0 0
cgroup /sys/fs/cgroup/memory cgroup 
rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory 
0 0
cgroup /sys/fs/cgroup/devices cgroup 
rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices
 0 0
cgroup /sys/fs/cgroup/freezer cgroup 
rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer
 0 0
cgroup /sys/fs/cgroup/blkio cgroup 
rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup 
rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event
 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup 
rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb
 0 0
name=systemd /sys/fs/cgroup/systemd cgroup 
rw,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd
 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc 
rw,nosuid,nodev,noexec,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs 
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
gvfsd-fuse /run/user/1001/gvfs fuse.gvfsd-fuse 
rw,nosuid,nodev,relatime,user_id=1001,group_id=1001 0 0



_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Reply via email to