Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Sat, Oct 28, 2023 at 12:46 AM Tony Rodriguez wrote: > On 10/27/23 07:06, Lennart Poettering wrote: > > On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: > > > >> Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 > and > >> RH9/Rocky9 system-252. Nothing was done on my end to create this > issue. In > >> other words, no custom mount/unit files or services, just your typical > ISO > >> install and rpm updates. > >> > >> An ordering cycle occurs, when "_netdev" is specified within /etc/fstab > for > >> systemd. This happens with systemd-239-14 and systemd-239-18 using > iSCSI > >> based file systems.Seems others are experiencing this as well (see > link > >> below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. > >> Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based > >> devices/file systems. The system may not boot, may not mount file > systems, > >> may not start services/unit files, and the system becomes slow during > system > >> boot. > >> > >> Does anyone know of a fix/patch and root cause for this? > >> > >> Please see this link: > >> > https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 > >> > >> # cat /etc/fstab > >> [... > > 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and > using tmpfs for "/tmp" appears to stop the dependency issue for > systemd-239 for systemd-252. However, RH8 and RH9 don't support > systemd-networkd, I am wondering how this can be overcome if removing > "/tmp" and using "tmpfs" aren't options? Would I have to modify various > services and targets? What would I need to add or remove within services > and targets to avoid these dependencies? > everything on the system depends on /tmp having behaviour and semantics of a local filesystem. it is literally part of ABI if you wish. it is hardcoded everywhere it must "be there" always and until the last minute. Don't do that then ! it is not only systemd.. What is exactly your problem ? you cannot commit a little ram to tmpfs ?
Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Mo, 30.10.23 10:17, Lennart Poettering (lenn...@poettering.net) wrote: > On Fr, 27.10.23 20:46, Tony Rodriguez (unixpro1...@gmail.com) wrote: > > > Andrea asked for more details so I have provide this verbose output. > > > > 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and using > > tmpfs for "/tmp" appears to stop the dependency issue for systemd-239 for > > systemd-252. However, RH8 and RH9 don't support systemd-networkd, I am > > wondering how this can be overcome if removing "/tmp" and using "tmpfs" > > aren't options? Would I have to modify various services and targets? What > > would I need to add or remove within services and targets to avoid these > > dependencies? > > This is something you'd have to ask your OS vendor. If they don't > support netwokrd, they will support something else, and maybe it has a > way to run in early boot or initrd. > > Booting without /tmp/ mounted during early boot is simply not > supported from upstream, sorry, can't help you there. Please contact > your OS vendor if they can help you. > > > 2) On another note, with RH9 systemd-252-14/systemd-252-18 and iscsi, new > > dependency issues occur if "_netdev" within /etc/fstab is specified for > > "/var" or "/usr". > > If /usr/ is split off it *must* be mounted even earlier than /tmp/: it > must be mounted in the initrd, nothing else is supported, sorry. > > If /var/ is split off it must be mounted at the same point as /tmp/, > i.e some time in early boot, not necessarily in the initrd though. Since we never documented this properly I have put together another piece of documentation that summarizes the requirements on mounts and when they must be available during boot: https://github.com/systemd/systemd/pull/29761 You can see the rendered version here (until the next PR push that is) https://github.com/systemd/systemd/blob/87828aae4712bdb300101b05911392c43d081a6b/docs/MOUNT_REQUIREMENTS.md Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Fr, 27.10.23 20:46, Tony Rodriguez (unixpro1...@gmail.com) wrote: > Andrea asked for more details so I have provide this verbose output. > > 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and using > tmpfs for "/tmp" appears to stop the dependency issue for systemd-239 for > systemd-252. However, RH8 and RH9 don't support systemd-networkd, I am > wondering how this can be overcome if removing "/tmp" and using "tmpfs" > aren't options? Would I have to modify various services and targets? What > would I need to add or remove within services and targets to avoid these > dependencies? This is something you'd have to ask your OS vendor. If they don't support netwokrd, they will support something else, and maybe it has a way to run in early boot or initrd. Booting without /tmp/ mounted during early boot is simply not supported from upstream, sorry, can't help you there. Please contact your OS vendor if they can help you. > 2) On another note, with RH9 systemd-252-14/systemd-252-18 and iscsi, new > dependency issues occur if "_netdev" within /etc/fstab is specified for > "/var" or "/usr". If /usr/ is split off it *must* be mounted even earlier than /tmp/: it must be mounted in the initrd, nothing else is supported, sorry. If /var/ is split off it must be mounted at the same point as /tmp/, i.e some time in early boot, not necessarily in the initrd though. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On 10/27/23 07:06, Lennart Poettering wrote: On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In other words, no custom mount/unit files or services, just your typical ISO install and rpm updates. An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI based file systems. Seems others are experiencing this as well (see link below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based devices/file systems. The system may not boot, may not mount file systems, may not start services/unit files, and the system becomes slow during system boot. Does anyone know of a fix/patch and root cause for this? Please see this link: https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 # cat /etc/fstab [...] /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rhel-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 # journalctl -b | grep deleted Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start deleted to break ordering cycle starting with basic.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job network-pre.target/start deleted to break ordering cycle starting with network.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start /tmp must be available during early boot already, and your NetworkManager service is apparently a late boot service. Hence you have a cycle: you want that /tmp/ is mounted after the network, but your network is configured really late. But /tmp is necessary during early boot. BOOM! Two ways out: 1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for it, like everyone else. 2. Upgrade to a better network management solution that has no problems with running in early boot, for example systemd-networkd. Lennart -- Lennart Poettering, Berlin Thank you Lennart and Andrei, Andrea asked for more details so I have provide this verbose output. 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and using tmpfs for "/tmp" appears to stop the dependency issue for systemd-239 for systemd-252. However, RH8 and RH9 don't support systemd-networkd, I am wondering how this can be overcome if removing "/tmp" and using "tmpfs" aren't options? Would I have to modify various services and targets? What would I need to add or remove within services and targets to avoid these dependencies? 2) On another note, with RH9 systemd-252-14/systemd-252-18 and iscsi, new dependency issues occur if "_netdev" within /etc/fstab is specified for "/var" or "/usr". My system will not reach a graphical state and will take a very long time to boot. The dbus related error is also strange as well. Have to ssh into the system, since the GUI doesn't start, and manually "init 5" to make things work. Only way I can boot, (without any issues), is by omitting "_netdev" for "/tmp", "/usr", and "/var". This doesn't make sense because "_netdev" is the recommended way of telling systemd to treat file systems as remote-fs targets. RH9 dependency errors are listed below: /etc/fstab UID=d21b12c1-0d2b-435d-bdf9-fb327a484539 /boot xfs defaults 0 0 UUID=C1FC-48AC /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/rh9--iscsi4--mp-root / xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-home /home xfs defaults,_netdev 0 0 #/dev/mapper/rh9--iscsi4--mp-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rh9--iscsi4--mp-usr /usr xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-swap none swap defaults 0 0 --- /var/log/messages (short -- longer one is listed below) -- Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: Dependency failed for Power Profiles daemon. Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: power-profiles-daemon.service: Job power-profiles-daemon.service/start failed with result 'dependency'. Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: Dependency failed for firewalld - dynamic firewall
Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: > Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and > RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In > other words, no custom mount/unit files or services, just your typical ISO > install and rpm updates. > > An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for > systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI > based file systems. Seems others are experiencing this as well (see link > below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. > Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based > devices/file systems. The system may not boot, may not mount file systems, > may not start services/unit files, and the system becomes slow during system > boot. > > Does anyone know of a fix/patch and root cause for this? > > Please see this link: > https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 > > # cat /etc/fstab > [...] > /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 > UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 > /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 > /dev/mapper/rhel-var /var xfs > defaults,_netdev,x-initrd.mount 0 0 > /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 > /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 > > # journalctl -b | grep deleted > Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start > deleted to break ordering cycle starting with basic.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job > network-pre.target/start deleted to break ordering cycle starting with > network.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job > dbus.socket/start deleted to break ordering cycle starting with > NetworkManager.service/start /tmp must be available during early boot already, and your NetworkManager service is apparently a late boot service. Hence you have a cycle: you want that /tmp/ is mounted after the network, but your network is configured really late. But /tmp is necessary during early boot. BOOM! Two ways out: 1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for it, like everyone else. 2. Upgrade to a better network management solution that has no problems with running in early boot, for example systemd-networkd. Lennart -- Lennart Poettering, Berlin
