[systemd-devel] systemd is trying to break mount ordering
I have the following setup on a freshly updated Fedora Rawhide machine with systemd-220-9.fc23.x86_64. # cat /etc/fstab [comments left out] UUID=d5ac823b-d0bd-4f7f-bf4b-5cc82d585a92 / btrfs subvol=root 0 0 UUID=ec79f233-055c-40fa-98e5-e2d77314913a /boot ext4 defaults1 2 UUID=d5ac823b-d0bd-4f7f-bf4b-5cc82d585a92 /home btrfs subvol=home 0 0 192.168.122.1:/srv/nfs /mnt/nfs nfs defaults 0 0 /var/tmp/test.iso /mnt/nfs/content iso9660 loop,ro 0 0 Notice the last two lines. There is an NFS mount mounted to /mnt/nfs and an ISO filesystem mounted into /mnt/nfs/content, which makes it dependent on the NFS mount. After booting the machine, there are the following lines in the journal: [snip...] Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on local-fs.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs-content.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on network.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Breaking ordering cycle by deleting job sockets.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: sockets.target: Job sockets.target/start deleted to break ordering cycle starting with firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found ordering cycle on network.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on systemd-networkd.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on dbus.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on dbus.socket/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on sysinit.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on fedora-autorelabel-mark.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on local-fs.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on mnt-nfs-content.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on mnt-nfs.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Found dependency on network.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: network.target: Breaking ordering cycle by deleting job systemd-networkd.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: systemd-networkd.service: Job systemd-networkd.service/start deleted to break ordering cycle starting with network.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found ordering cycle on firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on basic.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on dnf-makecache.timer/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on sysinit.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on fedora-autorelabel-mark.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on local-fs.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs-content.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on network.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Breaking ordering cycle by deleting job dnf-makecache.timer/start Jun 15 10:37:55 rawhide-virt systemd[1]: dnf-makecache.timer: Job dnf-makecache.timer/start deleted to break ordering cycle starting with firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found ordering cycle on firewalld.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on basic.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on sysinit.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on fedora-autorelabel-mark.service/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on local-fs.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs-content.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs.mount/start Jun
Re: [systemd-devel] systemd is trying to break mount ordering
On Mon, 2015-06-15 at 13:24 +0200, Jan Synáček wrote: 192.168.122.1:/srv/nfs /mnt/nfs nfs defaults 0 0 /var/tmp/test.iso /mnt/nfs/content iso9660 loop,ro 0 0 Notice the last two lines. There is an NFS mount mounted to /mnt/nfs and an ISO filesystem mounted into /mnt/nfs/content, which makes it dependent on the NFS mount. Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on local-fs.target/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs-content.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on mnt-nfs.mount/start Jun 15 10:37:55 rawhide-virt systemd[1]: firewalld.service: Found dependency on network.target/start Isn't systemd trying to delete too many jobs while resolving the cycles? I don't think the cycle breaking is to blame. It's simple and only considers one cycle at a time, but in this case I doubt there exists any good solution that could be found. The cycle breaking only potentially breaks non-mandatory dependencies (Wants). local-fs.target dependencies on mounts and (probably, didn't check) dependencies between mounts are Requires, so the dependency that's arguably wrong here cannot be broken. Once local-fs.target gets a hard dependency on network the situation is already pretty bad, and you probably shouldn't expect it to recover gracefully from that. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd is trying to break mount ordering
On Mon, 15.06.15 13:24, Jan Synáček (jsyna...@redhat.com) wrote: I have the following setup on a freshly updated Fedora Rawhide machine with systemd-220-9.fc23.x86_64. # cat /etc/fstab [comments left out] UUID=d5ac823b-d0bd-4f7f-bf4b-5cc82d585a92 / btrfs subvol=root 0 0 UUID=ec79f233-055c-40fa-98e5-e2d77314913a /boot ext4 defaults1 2 UUID=d5ac823b-d0bd-4f7f-bf4b-5cc82d585a92 /home btrfs subvol=home 0 0 192.168.122.1:/srv/nfs /mnt/nfs nfs defaults 0 0 /var/tmp/test.iso /mnt/nfs/content iso9660 loop,ro 0 0 Notice the last two lines. There is an NFS mount mounted to /mnt/nfs and an ISO filesystem mounted into /mnt/nfs/content, which makes it dependent on the NFS mount. Please add _netdev to the mount options of the ISO mount, to let systemd know that you need the network for that. Otherwise systemd assumes the ISO mount is in fact a local mount (which is hence ordered before local-fs.target which in turn is before basic.target), while correctly detecting that the nfs mount is a remote mount (which is henece ordered before remote-fs.target which in turn is usually assumed to be started much later than local-fs.target). Since however the local mount is ordered after the remote mounts you get a cyclic dep loop. Isn't systemd trying to delete too many jobs while resolving the cycles? Well, systemd removes jobs effectively randomly, since for the cycle breaking logic all units are the same. Of course, you might consider some jobs more important than others, but systemd doesn't know which ones those would be. There have been prior requests for a better cycle breaking strategy but so far I am not aware of any proposal that could really work and substantially better things. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
