Re: [systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
On 10/27/23 07:06, Lennart Poettering wrote: On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1...@gmail.com) wrote: Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In other words, no custom mount/unit files or services, just your typical ISO install and rpm updates. An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI based file systems. Seems others are experiencing this as well (see link below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based devices/file systems. The system may not boot, may not mount file systems, may not start services/unit files, and the system becomes slow during system boot. Does anyone know of a fix/patch and root cause for this? Please see this link: https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 # cat /etc/fstab [...] /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rhel-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 # journalctl -b | grep deleted Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start deleted to break ordering cycle starting with basic.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job network-pre.target/start deleted to break ordering cycle starting with network.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start /tmp must be available during early boot already, and your NetworkManager service is apparently a late boot service. Hence you have a cycle: you want that /tmp/ is mounted after the network, but your network is configured really late. But /tmp is necessary during early boot. BOOM! Two ways out: 1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for it, like everyone else. 2. Upgrade to a better network management solution that has no problems with running in early boot, for example systemd-networkd. Lennart -- Lennart Poettering, Berlin Thank you Lennart and Andrei, Andrea asked for more details so I have provide this verbose output. 1) Lennart's recommendation of removing "/tmp" within /etc/fstab and using tmpfs for "/tmp" appears to stop the dependency issue for systemd-239 for systemd-252. However, RH8 and RH9 don't support systemd-networkd, I am wondering how this can be overcome if removing "/tmp" and using "tmpfs" aren't options? Would I have to modify various services and targets? What would I need to add or remove within services and targets to avoid these dependencies? 2) On another note, with RH9 systemd-252-14/systemd-252-18 and iscsi, new dependency issues occur if "_netdev" within /etc/fstab is specified for "/var" or "/usr". My system will not reach a graphical state and will take a very long time to boot. The dbus related error is also strange as well. Have to ssh into the system, since the GUI doesn't start, and manually "init 5" to make things work. Only way I can boot, (without any issues), is by omitting "_netdev" for "/tmp", "/usr", and "/var". This doesn't make sense because "_netdev" is the recommended way of telling systemd to treat file systems as remote-fs targets. RH9 dependency errors are listed below: /etc/fstab UID=d21b12c1-0d2b-435d-bdf9-fb327a484539 /boot xfs defaults 0 0 UUID=C1FC-48AC /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/rh9--iscsi4--mp-root / xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-home /home xfs defaults,_netdev 0 0 #/dev/mapper/rh9--iscsi4--mp-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rh9--iscsi4--mp-usr /usr xfs defaults,_netdev 0 0 /dev/mapper/rh9--iscsi4--mp-swap none swap defaults 0 0 --- /var/log/messages (short -- longer one is listed below) -- Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: Dependency failed for Power Profiles daemon. Oct 27 20:00:24 rh9-iscsi4-mp systemd[1]: power-profiles-daemon.servic
[systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)
Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In other words, no custom mount/unit files or services, just your typical ISO install and rpm updates. An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI based file systems. Seems others are experiencing this as well (see link below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based devices/file systems. The system may not boot, may not mount file systems, may not start services/unit files, and the system becomes slow during system boot. Does anyone know of a fix/patch and root cause for this? Please see this link: https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 # cat /etc/fstab [...] /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rhel-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 # journalctl -b | grep deleted Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start deleted to break ordering cycle starting with basic.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job network-pre.target/start deleted to break ordering cycle starting with network.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start # mount | grep " /tmp " --> not mounted # systemctl status dbus.socket ● dbus.socket - D-Bus System Message Bus Socket Loaded: loaded (/usr/lib/systemd/system/dbus.socket; static; vendor preset: enabled) Active: inactive (dead) --> not started
[systemd-devel] Help! iSCSI based file systemd with "_netdev" causing ordering cycles to occur (random services and mounts fail)
Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and RH9/Rocky9 system-252. Nothing was done on my end to create this issue. In other words, no custom mount/unit files or services, just your typical ISO install and rpm updates. An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for systemd. This happens with systemd-239-14 and systemd-239-18 using iSCSI based file systems. Seems others are experiencing this as well (see link below). I can also confirm this happens with systemd-252 (RH9/Rocky9)l. Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based devices/file systems. The system may not boot, may not mount file systems, may not start services/unit files, and the system becomes slow during system boot. Does anyone know of a fix/patch and root cause for this? Please see this link: https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22 # cat /etc/fstab [...] /dev/mapper/rhel-root / xfs defaults,_netdev 0 0 UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults,_netdev 0 0 /dev/mapper/rhel-var /var xfs defaults,_netdev,x-initrd.mount 0 0 /dev/mapper/rhel-var_log /var/log xfs defaults,_netdev 0 0 /dev/mapper/rhel-var_tmp /var/tmp xfs defaults,_netdev 0 0 # journalctl -b | grep deleted Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start deleted to break ordering cycle starting with basic.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job network-pre.target/start deleted to break ordering cycle starting with network.target/start Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job dbus.socket/start deleted to break ordering cycle starting with NetworkManager.service/start # mount | grep " /tmp " --> not mounted # systemctl status dbus.socket ● dbus.socket - D-Bus System Message Bus Socket Loaded: loaded (/usr/lib/systemd/system/dbus.socket; static; vendor preset: enabled) Active: inactive (dead) --> not started
Re: [systemd-devel] Help! Reached target Local File Systems order is incorrect
Believe I found a solution after some extra debugging. However, I appreciate materials/links you provided. It is definitely good stuff to know. Regards, Tony
[systemd-devel] Help! Reached target Local File Systems order is incorrect
Created a service that invokes a "systemctl daemon-reload". Goal is for a reload to occur early in the boot process, before other user made services are invoked. During additional testing, sometimes it is correct and other times it is out of order (incorrect - See steps C). It may work for 5 or 6 times after each reboot/shutdown, then randomly become incorrect. How can I make this more consistent? Already tried various keyword combinations (wants,before,after, etc) within the unit file, all without luck. Thought about something like "After=var.mount, etc" as well, but that is inflexible because I will not know filesystems users may create. A) Unit file [Unit] Description=Systemctl-Reload Wants=local-fs.target DefaultDependencies=yes [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/systemctl daemon-reload [Install] WantedBy=local-fs.target B) Correct order: ** Reached target Local File Systems is after all mounting is done. Sometimes it works. eg: journalctl --boot=0 | grep -E 'Mounting |Mounted |Local File' Oct 09 11:21:39 rocky8-linux-build systemd[1]: Mounting Kernel Configuration File System... Oct 09 11:21:39 rocky8-linux-build systemd[1]: Mounted Kernel Configuration File System. Oct 09 11:21:40 rocky8-linux-build systemd[1]: Reached target Local File Systems (Pre). Oct 09 11:21:40 rocky8-linux-build systemd[1]: Reached target Local File Systems. Oct 09 11:21:40 rocky8-linux-build systemd[1]: Mounting /sysroot... Oct 09 11:21:40 rocky8-linux-build kernel: XFS (dm-0): Mounting V5 Filesystem Oct 09 11:21:40 rocky8-linux-build systemd[1]: Mounted /sysroot. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounting /sysroot/usr... Oct 09 11:21:41 rocky8-linux-build kernel: XFS (dm-2): Mounting V5 Filesystem Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted /sysroot/usr. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Stopped target Local File Systems. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Stopped target Local File Systems (Pre). Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted POSIX Message Queue File System. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted Kernel Debug File System. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted Huge Pages File System. Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounting FUSE Control File System... Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounted FUSE Control File System. Oct 09 11:21:42 rocky8-linux-build systemd[1]: Reached target Local File Systems (Pre). Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounting /boot... Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounting /tmp... Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounting /var... Oct 09 11:21:42 rocky8-linux-build kernel: XFS (vda2): Mounting V5 Filesystem Oct 09 11:21:42 rocky8-linux-build kernel: XFS (dm-3): Mounting V5 Filesystem Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounting /home... Oct 09 11:21:42 rocky8-linux-build kernel: XFS (dm-4): Mounting V5 Filesystem Oct 09 11:21:42 rocky8-linux-build kernel: XFS (dm-5): Mounting V5 Filesystem Oct 09 11:21:42 rocky8-linux-build systemd[1]: Mounted /tmp. Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounted /boot. Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounting /boot/efi... Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounted /boot/efi. Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounted /home. Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounted /var. *** Oct 09 11:21:43 rocky8-linux-build systemd[1]: Reached target Local File Systems. <<--- This is correct Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounting RPC Pipe File System... Oct 09 11:21:43 rocky8-linux-build systemd[1]: Mounted RPC Pipe File System. C) Other times a mount is randomly out of order: eg: journalctl --boot=0 | grep -E 'Mounting |Mounted |Local File' Oct 09 11:21:39 rocky8-linux-build systemd[1]: Mounting Kernel Configuration File System... Oct 09 11:21:39 rocky8-linux-build systemd[1]: Mounted Kernel Configuration File System. Oct 09 11:21:40 rocky8-linux-build systemd[1]: Reached target Local File Systems (Pre). Oct 09 11:21:40 rocky8-linux-build systemd[1]: Reached target Local File Systems. Oct 09 11:21:40 rocky8-linux-build systemd[1]: Mounting /sysroot... Oct 09 11:21:40 rocky8-linux-build kernel: XFS (dm-0): Mounting V5 Filesystem Oct 09 11:21:40 rocky8-linux-build systemd[1]: Mounted /sysroot. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounting /sysroot/usr... Oct 09 11:21:41 rocky8-linux-build kernel: XFS (dm-2): Mounting V5 Filesystem Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted /sysroot/usr. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Stopped target Local File Systems. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Stopped target Local File Systems (Pre). Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted POSIX Message Queue File System. Oct 09 11:21:41 rocky8-linux-build systemd[1]: Mounted Kernel Debug File
[systemd-devel] find_device() and FOREACH_DEVICE_DEVLINK memory leaks on "systemd-249"
Valgrind is reporting "still reachable" memory leak (2 blocks) when calling find_device() and FOREACH_DEVICE_DEVLINK against "systemd-249". In my case, they are both called within fstab-generator.c on "systemd-249". Only code modifications, on my end, are within fstab-generator.c A) Block 1 Memory Leak with find_device() - Defined within ./src/udev/udevadm-util.h int find_device(const char *id, const char *prefix, sd_device **ret); Also using, "_cleanup_(sd_device_unrefp) sd_device *ret = NULL;" and passing that into find_device() B) Block 2 Memory Leak with FOREACH_DEVICE_DEVLINK - Think this leak is caused by find_device() issue. ./src/libsystemd/sd-device/device-util.h #define FOREACH_DEVICE_DEVLINK(device, devlink) \ for (devlink = sd_device_get_devlink_first(device); \ devlink; \ devlink = sd_device_get_devlink_next(device)) } ** However, Valgrind doesn't report any "still reachable" memory leak messages with my exact same code fstab-generator.c mods on "systemd-250". Somewhere within "systemd-250" code base, this leak issue was already resolved. Just want to understand what is causing this leak, perhaps a weblink that demonstrates/discusses the bug/issue, and how to fix it? This way I can create a patch for "systemd-249", unsure if we can upgrade to "systemd-250" anytime in the near future on our end. Below demonstrates the issue. Thanks, Tony SYSTEMD-249: [@build systemd-249]# valgrind --leak-check=full --show-leak-kinds=all ./build/systemd-fstab-generator ==234387== Memcheck, a memory error detector ==234387== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==234387== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info ==234387== Command: ./build/systemd-fstab-generator ==234387== ==234387== ==234387== HEAP SUMMARY: ==234387== in use at exit: 8,192 bytes in 2 blocks ==234387== total heap usage: 512 allocs, 510 frees, 4,296,424 bytes allocated ==234387== ==234387== 4,096 bytes in 1 blocks are still reachable in loss record 1 of 2 ==234387== at 0x484186F: malloc (vg_replace_malloc.c:380) ==234387== by 0x4A269E6: mempool_alloc_tile (mempool.c:46) ==234387== by 0x4A26A75: mempool_alloc0_tile (mempool.c:65) ==234387== by 0x4A146A7: hashmap_base_new (hashmap.c:777) ==234387== by 0x4A14886: hashmap_base_ensure_allocated (hashmap.c:828) ==234387== by 0x4A148FC: _ordered_hashmap_ensure_allocated (hashmap.c:841) ==234387== by 0x4ACEA06: device_add_property_aux (sd-device.c:100) ==234387== by 0x4ACE349: device_add_property_internal (device-internal.h:100) ==234387== by 0x4ACF533: device_set_syspath (sd-device.c:208) ==234387== by 0x4ACF6C0: sd_device_new_from_syspath (sd-device.c:228) ==234387== by 0x4ACFA52: sd_device_new_from_devnum (sd-device.c:247) ==234387== by 0x4AD0900: sd_device_new_from_stat_rdev (sd-device.c:410) ==234387== ==234387== 4,096 bytes in 1 blocks are still reachable in loss record 2 of 2 ==234387== at 0x484186F: malloc (vg_replace_malloc.c:380) ==234387== by 0x4A269E6: mempool_alloc_tile (mempool.c:46) ==234387== by 0x4A26A75: mempool_alloc0_tile (mempool.c:65) ==234387== by 0x4A146A7: hashmap_base_new (hashmap.c:777) ==234387== by 0x4A14886: hashmap_base_ensure_allocated (hashmap.c:828) ==234387== by 0x4A14926: _set_ensure_allocated (hashmap.c:845) ==234387== by 0x4A17279: _set_put_strdup_full (hashmap.c:1851) ==234387== by 0x4AD3DEB: device_add_devlink (sd-device.c:1186) ==234387== by 0x4AD432F: handle_db_line (sd-device.c:1259) ==234387== by 0x4AD4CB0: device_read_db_internal_filename (sd-device.c:1436) ==234387== by 0x4AD5147: device_read_db_internal (sd-device.c:1468) ==234387== by 0x4ACE09D: device_read_db (device-private.h:61) ==234387== ==234387== LEAK SUMMARY: ==234387== definitely lost: 0 bytes in 0 blocks ==234387== indirectly lost: 0 bytes in 0 blocks ==234387== possibly lost: 0 bytes in 0 blocks ==234387== still reachable: 8,192 bytes in 2 blocks ==234387== suppressed: 0 bytes in 0 blocks ==234387== ==234387== For lists of detected and suppressed errors, rerun with: -s ==234387== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) SYSTEMD-250: [build systemd-250]# valgrind --leak-check=full --show-leak-kinds=all ./build/systemd-fstab-generator ==234804== Memcheck, a memory error detector ==234804== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==234804== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info ==234804== Command: ./build/systemd-fstab-generator ==234804== ==234804== ==234804== HEAP SUMMARY: ==234804== in use at exit: 0 bytes in 0 blocks ==234804== total heap usage: 92 allocs, 92 frees, 28,743 bytes allocated ==234804== ==234804== All heap blocks were freed -- no leaks are possible ==234804== ==234804== For lists of detected and suppressed errors, rerun
[systemd-devel] Need a systemd unit example that checks /etc/fstab for modification and sends a text message
From my understanding, it is frowned by systemd developers to "automatically" reload systemd via "systemctl daemon-reload" when /etc/fstab is modified. Guessing this is why such functionality hasn't already been incorporated into systemd. However, I would like to send a simple text message. Instructing users to manually invoke "systemctl deamon-reload" after modifying /etc/fstab via dmesg command or inside /var/log/messages. Unsure how to do so inside a systemd UNIT. Will someone please provide an example how to do so? Thanks
Re: [systemd-devel] help: unmounting iscsi xfs filesystem without using netdev in /etc/fstab
Thanks but I did not misinterpret, I know _netdev is used as a hint/keyword tag. I was speaking in general terms regarding using _netdev. Yes I also believe the delay is caused by network-online as well. Even though I figured things out by myself, I still want to thank others that offered suggestions and tried to help. Thanks again. Tony On 8/27/19 11:06 AM, Andrei Borzenkov wrote: 27.08.2019 18:55, Tony Rodriguez пишет: Interesting that you mentioned the following. I actually added logic to do something like this early yesterday. Will discuss with my team to see if this is good enough. I'd rather not bypass _netdev either. Feels like I am re-eventing the wheel. I think the main issue regarding using _netdev is the amount of time it takes to check/mount iscsi devices. Believe it waits up to 90 seconds to timeout for an iscsi device/filesystem and I am not sure this can be bypassed. Is it possible to limit the amount of time _netdev will take? You seem to seriously misinterpret what you see. _netdev does not "do" anything. It only affects which After/Before/Wants directives are added to mount unit definition - nothing more. If you see 90 seconds timeout, most likely one of units that mount is ordered After fails to start - and my educated guess is that it is related to network-online. I tried setting x-systemd.device-timeout and x-systemd.mount-timeout with _netdev in /etc/fstab but it didn't seem to work. It also didn't propagate to /run/systemd/generators/remote-fs-target.requires I believe the systemd document mentioned such settings are ignored. "I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev" On 8/27/19 2:33 AM, Michael Chapman wrote: On Tue, 27 Aug 2019, Tony Rodriguez wrote: Managed to detect/mount iscsi devices without using _netdev keyword in /etc/fstab. Made changes within src/fstab-generator/ftstab-generator.c and it seems to work. The only problem is during shutdown/reboot, my iscsi xfs filesystem does not unmount cleanly before the network/iscs service/system is shutdown. When this happens I receive a xfs error/warning. However this doesn't happen when _netdev is specified in /etc/fstab for my iscsi device. Seems _netdev handles management of mounts/unmounts before killing things off. How exactly does _netdev manage unmounting filesystems during a shutdown/reboot? One of the "default dependencies" for a mount unit that systemd thinks is a network filesystem is After=network.target. During shutdown, this ensures the filesystem is unmounted before networking is brought down: https://github.com/systemd/systemd/blob/fef40ceb5dfbb76d4733e579846a380a224efd55/src/core/mount.c#L455-L482 If you're not using the _netdev keyword, and systemd does not otherwise think this is a remote filesystem, you will need to add this dependency manually. You'll probably also want: Wants=network-online.target After=network-online.target to match the code linked above. I would like to invoke that same netdev unmount code/logic myself from within systemd source but without using the _netdev keyword. Unfortunately it is a requirement to not use _netdev within /etc/fstab for iscsi. Seems _netdev takes a long time to timeout and continue when unable to mount. I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev Plus any other options you may need, of course; this directive is not "list-valued", so you can't use it to add just one option. Checked src/core/mount.c and src/core/unmount.c but not sure what to do to duplicate how _netdev manages unmounting before the system is shutdown or rebooted. Do I need a special before and after keyword in /etc/fstab so my xfs filesystems is unmounted before shutting down the iscsi service/network/system? If so, will such keywords also propagate to /run/systemd/generator/remote-fs-target.requires? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freede
Re: [systemd-devel] help: unmounting iscsi xfs filesystem without using netdev in /etc/fstab
My team mentioned adding _netdev is now acceptable (thank goodness) but wondering if there is a way to limit the timeout when using _netdev? On 8/27/19 8:55 AM, Tony Rodriguez wrote: Interesting that you mentioned the following. I actually added logic to do something like this early yesterday. Will discuss with my team to see if this is good enough. I'd rather not bypass _netdev either. Feels like I am re-eventing the wheel. I think the main issue regarding using _netdev is the amount of time it takes to check/mount iscsi devices. Believe it waits up to 90 seconds to timeout for an iscsi device/filesystem and I am not sure this can be bypassed. Is it possible to limit the amount of time _netdev will take? I tried setting x-systemd.device-timeout and x-systemd.mount-timeout with _netdev in /etc/fstab but it didn't seem to work. It also didn't propagate to /run/systemd/generators/remote-fs-target.requires I believe the systemd document mentioned such settings are ignored. "I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev" On 8/27/19 2:33 AM, Michael Chapman wrote: On Tue, 27 Aug 2019, Tony Rodriguez wrote: Managed to detect/mount iscsi devices without using _netdev keyword in /etc/fstab. Made changes within src/fstab-generator/ftstab-generator.c and it seems to work. The only problem is during shutdown/reboot, my iscsi xfs filesystem does not unmount cleanly before the network/iscs service/system is shutdown. When this happens I receive a xfs error/warning. However this doesn't happen when _netdev is specified in /etc/fstab for my iscsi device. Seems _netdev handles management of mounts/unmounts before killing things off. How exactly does _netdev manage unmounting filesystems during a shutdown/reboot? One of the "default dependencies" for a mount unit that systemd thinks is a network filesystem is After=network.target. During shutdown, this ensures the filesystem is unmounted before networking is brought down: https://github.com/systemd/systemd/blob/fef40ceb5dfbb76d4733e579846a380a224efd55/src/core/mount.c#L455-L482 If you're not using the _netdev keyword, and systemd does not otherwise think this is a remote filesystem, you will need to add this dependency manually. You'll probably also want: Wants=network-online.target After=network-online.target to match the code linked above. I would like to invoke that same netdev unmount code/logic myself from within systemd source but without using the _netdev keyword. Unfortunately it is a requirement to not use _netdev within /etc/fstab for iscsi. Seems _netdev takes a long time to timeout and continue when unable to mount. I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev Plus any other options you may need, of course; this directive is not "list-valued", so you can't use it to add just one option. Checked src/core/mount.c and src/core/unmount.c but not sure what to do to duplicate how _netdev manages unmounting before the system is shutdown or rebooted. Do I need a special before and after keyword in /etc/fstab so my xfs filesystems is unmounted before shutting down the iscsi service/network/system? If so, will such keywords also propagate to /run/systemd/generator/remote-fs-target.requires? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] help: unmounting iscsi xfs filesystem without using netdev in /etc/fstab
Interesting that you mentioned the following. I actually added logic to do something like this early yesterday. Will discuss with my team to see if this is good enough. I'd rather not bypass _netdev either. Feels like I am re-eventing the wheel. I think the main issue regarding using _netdev is the amount of time it takes to check/mount iscsi devices. Believe it waits up to 90 seconds to timeout for an iscsi device/filesystem and I am not sure this can be bypassed. Is it possible to limit the amount of time _netdev will take? I tried setting x-systemd.device-timeout and x-systemd.mount-timeout with _netdev in /etc/fstab but it didn't seem to work. It also didn't propagate to /run/systemd/generators/remote-fs-target.requires I believe the systemd document mentioned such settings are ignored. "I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev" On 8/27/19 2:33 AM, Michael Chapman wrote: On Tue, 27 Aug 2019, Tony Rodriguez wrote: Managed to detect/mount iscsi devices without using _netdev keyword in /etc/fstab. Made changes within src/fstab-generator/ftstab-generator.c and it seems to work. The only problem is during shutdown/reboot, my iscsi xfs filesystem does not unmount cleanly before the network/iscs service/system is shutdown. When this happens I receive a xfs error/warning. However this doesn't happen when _netdev is specified in /etc/fstab for my iscsi device. Seems _netdev handles management of mounts/unmounts before killing things off. How exactly does _netdev manage unmounting filesystems during a shutdown/reboot? One of the "default dependencies" for a mount unit that systemd thinks is a network filesystem is After=network.target. During shutdown, this ensures the filesystem is unmounted before networking is brought down: https://github.com/systemd/systemd/blob/fef40ceb5dfbb76d4733e579846a380a224efd55/src/core/mount.c#L455-L482 If you're not using the _netdev keyword, and systemd does not otherwise think this is a remote filesystem, you will need to add this dependency manually. You'll probably also want: Wants=network-online.target After=network-online.target to match the code linked above. I would like to invoke that same netdev unmount code/logic myself from within systemd source but without using the _netdev keyword. Unfortunately it is a requirement to not use _netdev within /etc/fstab for iscsi. Seems _netdev takes a long time to timeout and continue when unable to mount. I haven't been following your earlier threads too closely, but if you're stuck with stupid fstab requirements why not just bypass them with altogether with drop-in: # path-to-mountpoint.mount.d/20-Options.conf [Mount] Options=_netdev Plus any other options you may need, of course; this directive is not "list-valued", so you can't use it to add just one option. Checked src/core/mount.c and src/core/unmount.c but not sure what to do to duplicate how _netdev manages unmounting before the system is shutdown or rebooted. Do I need a special before and after keyword in /etc/fstab so my xfs filesystems is unmounted before shutting down the iscsi service/network/system? If so, will such keywords also propagate to /run/systemd/generator/remote-fs-target.requires? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] help: unmounting iscsi xfs filesystem without using netdev in /etc/fstab
Managed to detect/mount iscsi devices without using _netdev keyword in /etc/fstab. Made changes within src/fstab-generator/ftstab-generator.c and it seems to work. The only problem is during shutdown/reboot, my iscsi xfs filesystem does not unmount cleanly before the network/iscs service/system is shutdown. When this happens I receive a xfs error/warning. However this doesn't happen when _netdev is specified in /etc/fstab for my iscsi device. Seems _netdev handles management of mounts/unmounts before killing things off. How exactly does _netdev manage unmounting filesystems during a shutdown/reboot? I would like to invoke that same netdev unmount code/logic myself from within systemd source but without using the _netdev keyword. Unfortunately it is a requirement to not use _netdev within /etc/fstab for iscsi. Seems _netdev takes a long time to timeout and continue when unable to mount. Checked src/core/mount.c and src/core/unmount.c but not sure what to do to duplicate how _netdev manages unmounting before the system is shutdown or rebooted. Do I need a special before and after keyword in /etc/fstab so my xfs filesystems is unmounted before shutting down the iscsi service/network/system? If so, will such keywords also propagate to /run/systemd/generator/remote-fs-target.requires? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Help: Local and Remote targets
Understood but I don't have to worry about iscsi offloading for now. Some theories I have to attack the problem but each has a problem and may be complex. 1) Since I can obtain udev/disk information when the system is in a multi-user already booted stated. Check for "iscsi" devices and filesystems then create a file on "/" that has the device id and specify it as an iscsi device. When the system reboots then check for that file, if it exists set state to REMOTE-FS target within fstab-generator.c Problem: I will have to manage create/remove such files within systemd especially if contents within /etc/fstab are manually modified by the user. Where would be the best place to put such logic within systemd source? 2) Boot the system mounting only "/" and "/usr" and defer fstab-generatoration until the system is booted. Then find a way to manually re-start fstab-generation after iscsi/network are operational. After that restart systemd. Problem: Unsure if this is reasonable. Wondering if doing so may introduce some kind of side effect issues regarding mounting and other services. Note: Really hoping for an easier way instead of trying to implement points (1) or (2) Tony On 8/20/19 3:33 PM, Rick Beldin wrote: This might be problematic. What will you do with iscsi devices that support iscsi offload? Those are typically completely invisible to the OS and just appear as another HBA. On 8/20/19 3:26 PM, Tony Rodriguez wrote: I modified systemd-219 functionality to meet a custom requirement for detecting filesystems/devices that use iscsi without using the _netdev keyword in /etc/fstab. It is a strict requirement regarding not using _netdev that I have no control over. However, I am stuck. From my understanding, src/fstab-generator.c is used to determine LOCAL-FS and REMOTE-FS. I have modified fstab-generator.c to search for iscsi udev devices which works, provided the system is already fully booted (typical multi-user state type of thing). I can also find /dev/disk/ block devices including iscsi devices using my logic within fstab-generator.c. The problem is when the system reboots, it appears systemd using the same fstab-generator logic that I implemented is unable to detect any iscsi devices. After doing some research, it appears normal systemd behavior is to mount / and /usr then invoke fstab-generator.c to determine if a given filesystem in /etc/fstab should be a local or remote target. The problem is I have no way to tell if a given iscsi filesystem/device specified in /etc/fstab (without the _netdev keyord ) is local/remote during the booting state. Mainly because devices are not yet populated in /dev/disk or udev. Is there a way I can determine if a filesystem/mount point/device specified in /etc/fstab is local or remote without the _netdev or fstype (which handles nfs, etc) keywords within fstab-generator.c? If checking udev and /dev/disk are not possible within fstab-generator.c, because such devices are not yet populated, then what should I modify within the systemd source so I can determine such local/remote targets myself and pass it to fstab-generator.c (especially when the system is in that / and /usr mounting state and udev devices are not yet populated? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Help: Local and Remote targets
I modified systemd-219 functionality to meet a custom requirement for detecting filesystems/devices that use iscsi without using the _netdev keyword in /etc/fstab. It is a strict requirement regarding not using _netdev that I have no control over. However, I am stuck. From my understanding, src/fstab-generator.c is used to determine LOCAL-FS and REMOTE-FS. I have modified fstab-generator.c to search for iscsi udev devices which works, provided the system is already fully booted (typical multi-user state type of thing). I can also find /dev/disk/ block devices including iscsi devices using my logic within fstab-generator.c. The problem is when the system reboots, it appears systemd using the same fstab-generator logic that I implemented is unable to detect any iscsi devices. After doing some research, it appears normal systemd behavior is to mount / and /usr then invoke fstab-generator.c to determine if a given filesystem in /etc/fstab should be a local or remote target. The problem is I have no way to tell if a given iscsi filesystem/device specified in /etc/fstab (without the _netdev keyord ) is local/remote during the booting state. Mainly because devices are not yet populated in /dev/disk or udev. Is there a way I can determine if a filesystem/mount point/device specified in /etc/fstab is local or remote without the _netdev or fstype (which handles nfs, etc) keywords within fstab-generator.c? If checking udev and /dev/disk are not possible within fstab-generator.c, because such devices are not yet populated, then what should I modify within the systemd source so I can determine such local/remote targets myself and pass it to fstab-generator.c (especially when the system is in that / and /usr mounting state and udev devices are not yet populated? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Need help detecting local/remote targets
I modified systemd-219 functionality to meet a custom requirement for detecting filesystems/devices that use iscsi without using the _netdev keyword in /etc/fstab. It is a strict requirement regarding not using _netdev that I have no control over. However, I am stuck. From my understanding, src/fstab-generator.c is used to determine LOCAL-FS and REMOTE-FS. I have modified fstab-generator.c to search for iscsi udev devices which works, provided the system is already fully booted (typical multi-user state type of thing). I can also find /dev/disk/ block devices including iscsi devices using my logic within fstab-generator.c. The problem is when the system reboots, it appears systemd using the same fstab-generator logic that I implemented is unable to detect any iscsi devices. After doing some research, it appears normal systemd behavior is to mount / and /usr then invoke fstab-generator.c to determine if a given filesystem in /etc/fstab should be a local or remote target. The problem is I have no way to tell if a given iscsi filesystem/device specified in /etc/fstab (without the _netdev keyord ) is local/remote during the booting state. Mainly because devices are not yet populated in /dev/disk or udev. Is there a way I can determine if a filesystem/mount point/device specified in /etc/fstab is local or remote without the _netdev or fstype (which handles nfs, etc) keywords within fstab-generator.c? If checking udev and /dev/disk are not possible within fstab-generator.c, because such devices are not yet populated, then what should I modify within the systemd source so I can determine such local/remote targets myself and pass it to fstab-generator.c (especially when the system is in that / and /usr mounting state and udev devices are not yet populated? Thanks, Tony ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
