Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On 10/27/23 07:06, Lennart Poettering wrote: On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In other words, no custom mount/unit files or services, just your typical ISO install and rpm updates. An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI based file systems. Seems others are experiencing this as well (see link below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based devices/file systems. The system may not boot, may not mount file systems, may not start services/unit files, and the system becomes slow during system boot. Does anyone know of a fix/patch and root cause for this? Please see this link: https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 # cat /etc/fstab [...] /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rhel-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 # journalctl -b | grep deleted Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start deleted to break ordering cycle starting with basic.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job network-pre.target/start deleted to break ordering cycle starting with network.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start /tmp must be available during early boot already, and your NetworkManager service is apparently a late boot service. Hence you have a cycle: you want that /tmp/ is mounted after the network, but your network is configured really late. But /tmp is necessary during early boot. BOOM! Two ways out: 1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for it, like everyone else. 2. Upgrade to a better network management solution that has no problems with running in early boot, for example systemd-networkd. Lennart -- Lennart Poettering, Berlin Thank you Lennart and Andrei, Andrea asked for more details so I have provide this verbose output. 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and using tmpfs for "/tmp" appears to stop the dependency issue for systemd-239 for systemd-252. However, RH8 and RH9 don't support systemd-networkd, I am wondering how this can be overcome if removing "/tmp" and using "tmpfs" aren't options? Would I have to modify various services and targets? What would I need to add or remove within services and targets to avoid these dependencies? 2) On another note, with RH9 systemd-252-14/systemd-252-18 and iscsi, new dependency issues occur if "_netdev" within /etc/fstab is specified for "/var" or "/usr". My system will not reach a graphical state and will take a very long time to boot. The dbus related error is also strange as well. Have to ssh into the system, since the GUI doesn't start, and manually "init 5" to make things work. Only way I can boot, (without any issues), is by omitting "_netdev" for "/tmp", "/usr", and "/var". This doesn't make sense because "_netdev" is the recommended way of telling systemd to treat file systems as remote-fs targets. RH9 dependency errors are listed below: /etc/fstab UID=d21b12c1-0d2b-435d-bdf9-fb327a484539 /boot xfs defaults 0 0 UUID=C1FC-48AC /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/rh9--iscsi4--mp-root / xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-home /home xfs defaults,_netdev 0 0 #/dev/mapper/rh9--iscsi4--mp-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rh9--iscsi4--mp-usr /usr xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-swap none swap defaults 0 0 --- /var/log/messages (short -- longer one is listed below) -- Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: Dependency failed for Power Profiles daemon. Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: power-profiles-daemon.service: Job power-profiles-daemon.service/start failed with result 'dependency'. Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: Dependency failed for firewalld - dynamic firewall
Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: > Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and > RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In > other words, no custom mount/unit files or services, just your typical ISO > install and rpm updates. > > An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for > systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI > based file systems. Seems others are experiencing this as well (see link > below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. > Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based > devices/file systems. The system may not boot, may not mount file systems, > may not start services/unit files, and the system becomes slow during system > boot. > > Does anyone know of a fix/patch and root cause for this? > > Please see this link: > https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 > > # cat /etc/fstab > [...] > /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 > UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 > /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 > /dev/mapper/rhel-var /var xfs > defaults,_netdev,x-initrd.mount 0 0 > /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 > /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 > > # journalctl -b | grep deleted > Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start > deleted to break ordering cycle starting with basic.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job > network-pre.target/start deleted to break ordering cycle starting with > network.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job > dbus.socket/start deleted to break ordering cycle starting with > NetworkManager.service/start /tmp must be available during early boot already, and your NetworkManager service is apparently a late boot service. Hence you have a cycle: you want that /tmp/ is mounted after the network, but your network is configured really late. But /tmp is necessary during early boot. BOOM! Two ways out: 1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for it, like everyone else. 2. Upgrade to a better network management solution that has no problems with running in early boot, for example systemd-networkd. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Reload simple
Andy Pieters writes: > Is there a good way to send a usr1 signal upon reload of a service > with type=simple? > > I don't really want to do KILL USR1 $(pidof process) as that could > pull other processes with similar names / instances. Use $MAINPID as per man systemd.service: ExecReload= Commands to execute to trigger a configuration reload in the service. This argument takes multiple command lines, following the same scheme as described for ExecStart= above. Use of this setting is optional. Specifier and environment variable substitution is supported here following the same scheme as for ExecStart=. One additional, special environment variable is set: if known, $MAINPID is set to the main process of the daemon, and may be used for command lines like the following: ExecReload=kill -HUP $MAINPID Note however that reloading a daemon by sending a signal (as with the example line above) is usually not a good choice, because this is an asynchronous operation and hence not suitable to order reloads of multiple services against each other. It is strongly recommended to set ExecReload= to a command that not only triggers a configuration reload of the daemon, but also synchronously waits for it to complete. For example, dbus- broker(1) uses the following: ExecReload=busctl call org.freedesktop.DBus \ /org/freedesktop/DBus org.freedesktop.DBus \ ReloadConfig -- Feri.
Re: [systemd-devel] Reload simple
On Fri, 27 Oct 2023 at 13:31, Ferenc Wágner wrote: > Andy Pieters writes: > > > Is there a good way to send a usr1 signal upon reload of a service > > with type=simple? > > > Use $MAINPID as per man systemd.service: > > ExecReload=kill -HUP $MAINPID > > I had actually misunderstood that part. I thought that was only available with type=fokring Tested this in my sample service and it works. Very happy with this Thank you
Re: [systemd-devel] Reload simple
Hi, Andy, On 10/27/23 1:06 PM, Andy Pieters wrote: > Hi > > Is there a good way to send a usr1 signal upon reload of a service with > type=simple? > > I don't really want to do KILL USR1 $(pidof process) as that could pull other > processes with similar names / instances. systemd.service(5) [...] ExecReload= [...] One additional, special environment variable is set: if known, $MAINPID is set to the main process of the daemon, and may be used for command lines like the following: /bin/kill -HUP $MAINPID Am I missing something? Best Donald > > Thanks -- Donald Buczek buc...@molgen.mpg.de Tel: +49 30 8413 1433
Re: [systemd-devel] Reload simple
On Fri, Oct 27, 2023 at 2:07 PM Andy Pieters wrote: > > Hi > > Is there a good way to send a usr1 signal upon reload of a service with > type=simple? > > I don't really want to do KILL USR1 $(pidof process) as that could pull other > processes with similar names / instances. > systemctl kill ... ?
[systemd-devel] Reload simple
Hi Is there a good way to send a usr1 signal upon reload of a service with type=simple? I don't really want to do KILL USR1 $(pidof process) as that could pull other processes with similar names / instances. Thanks
Re: [systemd-devel] Help! iSCSI based file systemd with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On Fri, Oct 27, 2023 at 5:01 AM Tony Rodriguez wrote: > > Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 > and RH9/Rocky9 system-252. Nothing was done on my end to create this > issue. In other words, no custom mount/unit files or services, just > your typical ISO install and rpm updates. > > An ordering cycle occurs, when "_netdev" is specified within /etc/fstab > for systemd. This happens with systemd-239-14 and systemd-239-18 using > iSCSI based file systems.Seems others are experiencing this as well > (see link below). I can also confirm this happens with systemd-252 > (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or > "/usr" iSCSI based devices/file systems. The system may not boot, may > not mount file systems, may not start services/unit files, and the > system becomes slow during system boot. > > Does anyone know of a fix/patch and root cause for this? > > Please see this link: > https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 > > # cat /etc/fstab > [...] > /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 > UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 > /dev/mapper/rhel-tmp /tmpxfs defaults,_netdev 0 0 > /dev/mapper/rhel-var /varxfs > defaults,_netdev,x-initrd.mount 0 0 > /dev/mapper/rhel-var_log /var/logxfs > defaults,_netdev 0 0 > /dev/mapper/rhel-var_tmp /var/tmpxfs > defaults,_netdev 0 0 > > # journalctl -b | grep deleted > Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start > deleted to break ordering cycle starting with basic.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job > network-pre.target/start deleted to break ordering cycle starting with > network.target/start > Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job > dbus.socket/start deleted to break ordering cycle starting with > NetworkManager.service/start > Showing the actual chain that caused the loop would be helpful. Did not systemd print it? > # mount | grep " /tmp " > --> not mounted > > # systemctl status dbus.socket > ● dbus.socket - D-Bus System Message Bus Socket > Loaded: loaded (/usr/lib/systemd/system/dbus.socket; static; vendor > preset: enabled) > Active: inactive (dead) > --> not started
