Quoting Peter Steele ([email protected]): > On 12/08/2015 08:00 AM, Serge Hallyn wrote: > >Ok, can you change the launch command in the scripts to > > > >lxc-start -n $containername -L /tmp/$containername.cout -l trace -o > >/tmp/$containername.dout -- /sbin/init log_target=console log_level=debug > > > >The console output will go into the .cout file and lxc debug output into > >.dout. > > > I've actually made some progress in reproducing this outside of my > framework. I originally thought the problem only occurred during the > first boot of the containers. I've discovered that it can happen any > time the server is rebooted and the containers are started when the > server comes up.
What do you mean by "when the server comes up"? If you bring up the server, let it set for 5 mins, then start them, they still fail? > I've only seen this problem when multiple > containers are starting at the same time. What lxc version are you using again? > I incorporated your modified start command into a test as follows: > > # for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace > -o /tmp/$vm.dout -- /sbin/init log_target=console log_level=debug; > done > > This starts all of my previously created containers at roughly the > same time, and when I do this some of the containers encounter the > systemd errors I've been seeing. Which containers hit these errors > vary from test to test. In looking at the .dout logs, I noticed the > following: > > lxc-start 1449591253.647 DEBUG lxc_conf - > conf.c:setup_rootfs:1295 - mounted '/hf/cs/vm-00/rootfs' on > '/usr/lib64/lxc/rootfs' > lxc-start 1449591253.647 INFO lxc_conf - > conf.c:setup_utsname:928 - 'pws-vm-00' hostname has been setup > lxc-start 1449591253.660 DEBUG lxc_conf - > conf.c:setup_hw_addr:2368 - mac address 'fe:d6:e8:96:7e:2d' on > 'eth0' has been setup > lxc-start 1449591253.660 DEBUG lxc_conf - > conf.c:setup_netdev:2595 - 'eth0' has been setup > lxc-start 1449591253.660 INFO lxc_conf - > conf.c:setup_network:2616 - network has been setup > lxc-start 1449591253.660 INFO lxc_conf - > conf.c:mount_autodev:1157 - Mounting container /dev > lxc-start 1449591253.661 INFO lxc_conf - > conf.c:mount_autodev:1179 - Mounted tmpfs onto > /usr/lib64/lxc/rootfs/dev > lxc-start 1449591253.661 INFO lxc_conf - > conf.c:mount_autodev:1197 - Mounted container /dev > lxc-start 1449591253.661 ERROR lxc_utils - > utils.c:open_without_symlink:1626 - No such file or directory - > Error examining fuse in > /usr/lib64/lxc/rootfs/sys/fs/fuse/connections Ok, so this shows that in the container 'sys/fs' existed, but fuse did not. This suggests that the fuse kernel module was not done loading yet. Could you add to /lib/systemd/system/lxc.service the line ExecStartPre=modprobe fuse and see if that helps? (I'm not sure if you'd also need to sleep a short time to give syfs time to catch up, or if the modprobe would wait... you could just use a script that waits until /sys/fs/fuse exists on the host) _______________________________________________ lxc-users mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-users
