Re: [systemd-devel] Running system services required for certain filesystems
On Mon, 12.01.15 10:37, Colin Guthrie (gm...@colin.guthr.ie) wrote: Hi, On a related note to my previous message (subject systemctl status not showing still running processes in inactive .mount unit cgroups (NFS specifically)), when mount.nfs runs to mount NFS filesystems, it shells out to /usr/sbin/start-statd which in turn calls sytemctl to start rpc.statd service. This feels ugly. I think the right answer for all this complexity is to add proper socket activation to the various daemons NFS needs. Then the ordering problems go away, and the services are automatically started when they are needed, but not before. inetd back in the day had socket activation for RPC services, maybe we should reintroduce that concept back to systemd? (I didn't follow the details of this thread, my NFS-fu is a bit too limited. If there's anything unresolved I am supposed to say something about, please let me know.) Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Running system services required for certain filesystems
Steve Dickson wrote on 13/01/15 16:29: On a related note to my previous message (subject systemctl status not showing still running processes in inactive .mount unit cgroups (NFS specifically)), when mount.nfs runs to mount NFS filesystems, it shells out to /usr/sbin/start-statd which in turn calls sytemctl to start rpc.statd service. This feels ugly. Why? This is why rpc.statd does not need to be started on the client default any more. Yes but it requires the shelling out to bash script to do some modification of a pre-calculate set of transactions and dynamically adjusts the systemd jobs. It feels very un-systemd to use systemctl during the initial transaction of start jobs to modify things. I guess I'm not such a systemd purist ;-) but it feels ok to me! :-) :) Generally speaking you also have to be really, really careful doing such things as they can be called at unexpected times and result in deadlocks (protected by a timeout thankfully) due to ordering cycles. e.g. say something in early boot that has Before=rpc-statd.service is run, that somehow triggers, e.g. an automount, that in turn calls mount.nfs, which in turn calls systemctl start rpc-statd.service, then that systemctl job will block because the job it creates is waiting for the job with Before=rpc-statd.service in it to complete. I see the deadlock... in theory... but that assumes the first Before=rpc-statd.service never completes which is very unlikely. Well, what I was meaning was the the process that is Before=rpc-statd.service is actually the parent of the one that ultimately tries do the mount. Ultimately the deadlock is there, but systemd will timeout the jobs (by default after 1m 30s), so it's not a real deadlock, but it is still something that could technically happen and delay boot. But it did get me thinking about how clean remote-fs-pre.target really is. We do need to make sure rpc.statd is running before any NFS filesystems are mounted and and relying on the blunt instrument of remote-fs-pre.target seems kinda wrong. It should be more on demand e.g. when I start an nfs mount, it should be able to specify that rpc.statd service is a prerequisite. So my question is, is there a cleaner way to have dependencies like this specified for particular FS types? With the goal being that before systemd will try and mount any NFS filesystems it will make sure that nfs-lock.service (or statd.service or nfs-statd.service or whatever it's name really should be) is running? I kinda want a Requires=nfs-lock.service and After=nfs-lock.service definitions to go into all my *.mount units for any nfs filesystem, but it a way that means I don't have to actually specify this manually in my fstab. Why spread out the pain? I think the sync point we have right now mount.nfs calling start-statd works and keeps everything in one place. Shelling out to start-statd definitely isn't a sync point and as I've outlined above, calling systemctl mid-transaction is really something we should avoid. Again, I do see your point. In this particular I'm not sure there is much else we can do. Yeah, that's somewhat my own conclusion so far too sadly... :s Thanks for your input here. Domain knowledge is invaluable! That's how we *all* learn! :-) Indeed :) All the best Col -- Colin Guthrie colin(at)mageia.org http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Running system services required for certain filesystems
On 01/12/2015 05:40 PM, Colin Guthrie wrote: Steve Dickson wrote on 12/01/15 20:31: Hello On 01/12/2015 05:37 AM, Colin Guthrie wrote: Hi, On a related note to my previous message (subject systemctl status not showing still running processes in inactive .mount unit cgroups (NFS specifically)), when mount.nfs runs to mount NFS filesystems, it shells out to /usr/sbin/start-statd which in turn calls sytemctl to start rpc.statd service. This feels ugly. Why? This is why rpc.statd does not need to be started on the client default any more. Yes but it requires the shelling out to bash script to do some modification of a pre-calculate set of transactions and dynamically adjusts the systemd jobs. It feels very un-systemd to use systemctl during the initial transaction of start jobs to modify things. I guess I'm not such a systemd purist ;-) but it feels ok to me! :-) Generally speaking you also have to be really, really careful doing such things as they can be called at unexpected times and result in deadlocks (protected by a timeout thankfully) due to ordering cycles. e.g. say something in early boot that has Before=rpc-statd.service is run, that somehow triggers, e.g. an automount, that in turn calls mount.nfs, which in turn calls systemctl start rpc-statd.service, then that systemctl job will block because the job it creates is waiting for the job with Before=rpc-statd.service in it to complete. I see the deadlock... in theory... but that assumes the first Before=rpc-statd.service never completes which is very unlikely. So calling systemctl during the initial transaction is really something to strongly discourage IMO. Ideally all information would be available after all the generators are run to calculate the initial transaction right at the beginning without any of dynamic modification in the middle. We have a sync point for this in the form of remote-fs-pre.target, but for some reason upstream nfs-utils people still deem that /usr/sbin/start-statd is a required component. I'm not seeing how remote-fs-pre.target is a sync point. Its only used by the nfs-client.target... Well, it's original intention was as as a sync point, but it doesn't seem to be getting used that way now (and there are some good reasons which I'll cover in a reply to Andrei). OK. But it did get me thinking about how clean remote-fs-pre.target really is. We do need to make sure rpc.statd is running before any NFS filesystems are mounted and and relying on the blunt instrument of remote-fs-pre.target seems kinda wrong. It should be more on demand e.g. when I start an nfs mount, it should be able to specify that rpc.statd service is a prerequisite. So my question is, is there a cleaner way to have dependencies like this specified for particular FS types? With the goal being that before systemd will try and mount any NFS filesystems it will make sure that nfs-lock.service (or statd.service or nfs-statd.service or whatever it's name really should be) is running? I kinda want a Requires=nfs-lock.service and After=nfs-lock.service definitions to go into all my *.mount units for any nfs filesystem, but it a way that means I don't have to actually specify this manually in my fstab. Why spread out the pain? I think the sync point we have right now mount.nfs calling start-statd works and keeps everything in one place. Shelling out to start-statd definitely isn't a sync point and as I've outlined above, calling systemctl mid-transaction is really something we should avoid. Again, I do see your point. In this particular I'm not sure there is much else we can do. I do like that it solves the case of calling mount /mountpoint command manually as a sysadmin and it will start the necessary service but I still thing it's ugly if called via systemctl start /mountpoint - we should be able to handle this kind of dep without such shelling out. Something like a pseudo service - systemd-fstype@nfs.service with Type=oneshot+RemainAfterExit=true+Exec=/usr/bin/true that is run by systemd before it does it mounting to act as a sync point (thus allowing nfs-lock.service to just put RequiredBy=systemd-fstype@nfs.service+Before=systemd-fstype@nfs.service and all is well) - there shouldn't really be a strong need for any actual changes to systemd-fstype@.service (or any systemd-fstype@nfs.service.d dropins) here, as it can all be specified the other way around in nfs-lock.service. WOW.. Granted I'm no systemd expert... what did you say?? :-) My apologies but I'm unable to parse the above paragraph at all! In the end, I'm all for making things go smoother but I've never been a fan of fixing something that's not broken... To be fair, I could probably word it better, and (being totally fair) I'm suggesting a similar abuse of a .service unit that the current nfs-utils.service does (which we really shouldn't do!) But ultimately, what the above would
Re: [systemd-devel] Running system services required for certain filesystems
Colin Guthrie wrote on 12/01/15 10:37: So, overall is remote-fs-pre.target sufficient here, or should we look into supporting this in a more hotplug/JIT friendly way? Digging into this further, I actually notice a problem with remote-fs-pre.target, at least on my system. It seems that it's not activated here any longer, despite me having remote-fs.target enabled and, as remote-fs-pre.target is static, I cannot specifically enable it. It's my understanding that it should be automatically started. Also, in an unrelated issue, remote-fs.target seems to be reached *before* my actual remote fs's are mounted. This all seems a little wrong (although correct in the sense that I am using the nofail option which is what triggers this lack of waiting) After a fresh boot here (note specifically how remote-fs-pre.target is not active and that the rhome.mount is reached about 40 seconds after remote-fs.target is reached): [colin@jimmy systemd (master)]$ systemctl status rhome.mount remote-fs.target remote-fs-pre.target network.target nfs-lock.service ● rhome.mount - /rhome Loaded: loaded (/etc/fstab) Active: active (mounted) since Mon 2015-01-12 10:44:24 GMT; 19min ago Where: /rhome What: nfs:/home/ Docs: man:fstab(5) man:systemd-fstab-generator(8) Process: 19065 ExecMount=/bin/mount -n nfs:/home/ /rhome -t nfs -o _netdev,nfsvers=3,nofail,user,tcp,rsize=8192,wsize=8192,soft (code=exited, status=0/SUCCESS) ● remote-fs.target - Remote File Systems Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled) Active: active since Mon 2015-01-12 10:43:44 GMT; 19min ago Docs: man:systemd.special(7) ● remote-fs-pre.target - Remote File Systems (Pre) Loaded: loaded (/usr/lib/systemd/system/remote-fs-pre.target; static) Active: inactive (dead) Docs: man:systemd.special(7) ● network.target - Network Loaded: loaded (/usr/lib/systemd/system/network.target; static) Active: active since Mon 2015-01-12 10:44:24 GMT; 19min ago Docs: man:systemd.special(7) http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget ● nfs-lock.service - NFS file locking service. Loaded: loaded (/usr/lib/systemd/system/nfs-lock.service; static) Active: active (running) since Mon 2015-01-12 10:44:24 GMT; 19min ago Process: 19099 ExecStart=/sbin/rpc.statd $STATDARGS (code=exited, status=0/SUCCESS) Process: 19074 ExecStartPre=/usr/libexec/nfs-utils/scripts/nfs-lock.preconfig (code=exited, status=0/SUCCESS) Main PID: 19101 (rpc.statd) CGroup: /system.slice/nfs-lock.service └─19101 /sbin/rpc.statd On my system I think nfs-lock.service was probably started by the (now fixed) horrible /usr/sbin/start-statd callout script implemented in mount.nfs. I do have nfs-lock unit specified with Before=remote-fs-pre.target, so in the past it was started before any mounts were attempted, but now it doesn't start it might not be activated until later. I'm not sure but is this just a bug in fstab-generator? i.e. Shouldn't it put a Requires+After=remote-fs-pre.target in it's generated .mount units for those mounts that are determined to be remote? (neither of which are present here) Or am I missing something? What else is meant to pull in remote-fs-pre.target? The only mention in the code is in src/core/mount.c but that's just for an After= dep, not a Requires= one. Perhaps the fstab-generator just needs a Requires= dep and mount.c takes care of the After= bit? I don't see any specific commits to fstab-generator since v217 which is what I'm using. This is my generated unit: [colin@jimmy systemd (master)]$ cat /run/systemd/generator/rhome.mount # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) [Mount] What=nfs:/home/ Where=/rhome Type=nfs Options=_netdev,nfsvers=3,nofail,user,tcp,rsize=8192,wsize=8192,soft And the other deps that are added in memory only (not sure why they are not just encoded into the generated .mount unit). [colin@jimmy systemd (master)]$ systemctl show rhome.mount| grep -Ei before|after|want|require Requires=-.mount Wants=network-online.target system.slice WantedBy=remote-fs.target Before=umount.target After=systemd-journald.socket remote-fs-pre.target network.target network-online.target system.slice -.mount RequiresMountsFor=/ Going back to the fact that remote-fs.target starts before the mounts due to my use of nofail, this is the offending line in fstab-generator: if (post !noauto !nofail !automount) fprintf(f, Before=%s\n, post); In an ideal world, I'd still like remote-fs.target (aka post) to wait, but I can see why local-fs.target should not wait (when post contains that value). e.g. I'd like remote-fs.target to be reached eventually, but only after the mount jobs have timed out Would a condition of: if (post !noauto (!nofail || streq(post, SPECIAL_REMOTE_FS_TARGET) !automount)
Re: [systemd-devel] Running system services required for certain filesystems
Steve Dickson wrote on 12/01/15 20:31: Hello On 01/12/2015 05:37 AM, Colin Guthrie wrote: Hi, On a related note to my previous message (subject systemctl status not showing still running processes in inactive .mount unit cgroups (NFS specifically)), when mount.nfs runs to mount NFS filesystems, it shells out to /usr/sbin/start-statd which in turn calls sytemctl to start rpc.statd service. This feels ugly. Why? This is why rpc.statd does not need to be started on the client default any more. Yes but it requires the shelling out to bash script to do some modification of a pre-calculate set of transactions and dynamically adjusts the systemd jobs. It feels very un-systemd to use systemctl during the initial transaction of start jobs to modify things. Generally speaking you also have to be really, really careful doing such things as they can be called at unexpected times and result in deadlocks (protected by a timeout thankfully) due to ordering cycles. e.g. say something in early boot that has Before=rpc-statd.service is run, that somehow triggers, e.g. an automount, that in turn calls mount.nfs, which in turn calls systemctl start rpc-statd.service, then that systemctl job will block because the job it creates is waiting for the job with Before=rpc-statd.service in it to complete. So calling systemctl during the initial transaction is really something to strongly discourage IMO. Ideally all information would be available after all the generators are run to calculate the initial transaction right at the beginning without any of dynamic modification in the middle. We have a sync point for this in the form of remote-fs-pre.target, but for some reason upstream nfs-utils people still deem that /usr/sbin/start-statd is a required component. I'm not seeing how remote-fs-pre.target is a sync point. Its only used by the nfs-client.target... Well, it's original intention was as as a sync point, but it doesn't seem to be getting used that way now (and there are some good reasons which I'll cover in a reply to Andrei). But it did get me thinking about how clean remote-fs-pre.target really is. We do need to make sure rpc.statd is running before any NFS filesystems are mounted and and relying on the blunt instrument of remote-fs-pre.target seems kinda wrong. It should be more on demand e.g. when I start an nfs mount, it should be able to specify that rpc.statd service is a prerequisite. So my question is, is there a cleaner way to have dependencies like this specified for particular FS types? With the goal being that before systemd will try and mount any NFS filesystems it will make sure that nfs-lock.service (or statd.service or nfs-statd.service or whatever it's name really should be) is running? I kinda want a Requires=nfs-lock.service and After=nfs-lock.service definitions to go into all my *.mount units for any nfs filesystem, but it a way that means I don't have to actually specify this manually in my fstab. Why spread out the pain? I think the sync point we have right now mount.nfs calling start-statd works and keeps everything in one place. Shelling out to start-statd definitely isn't a sync point and as I've outlined above, calling systemctl mid-transaction is really something we should avoid. I do like that it solves the case of calling mount /mountpoint command manually as a sysadmin and it will start the necessary service but I still thing it's ugly if called via systemctl start /mountpoint - we should be able to handle this kind of dep without such shelling out. Something like a pseudo service - systemd-fstype@nfs.service with Type=oneshot+RemainAfterExit=true+Exec=/usr/bin/true that is run by systemd before it does it mounting to act as a sync point (thus allowing nfs-lock.service to just put RequiredBy=systemd-fstype@nfs.service+Before=systemd-fstype@nfs.service and all is well) - there shouldn't really be a strong need for any actual changes to systemd-fstype@.service (or any systemd-fstype@nfs.service.d dropins) here, as it can all be specified the other way around in nfs-lock.service. WOW.. Granted I'm no systemd expert... what did you say?? :-) My apologies but I'm unable to parse the above paragraph at all! In the end, I'm all for making things go smoother but I've never been a fan of fixing something that's not broken... To be fair, I could probably word it better, and (being totally fair) I'm suggesting a similar abuse of a .service unit that the current nfs-utils.service does (which we really shouldn't do!) But ultimately, what the above would do is allow all the deps for the initial transaction to be pre-calculated right at the start without the need to shell out to something that calls systemctl start rpc-statd.service. Sadly, we'd still need a way to know that this was happening (i.e. being called from within systemd, not via mount ... directly) as a problem scenario would be that the machine was booted without any NFS
Re: [systemd-devel] Running system services required for certain filesystems
Andrei Borzenkov wrote on 12/01/15 18:31: В Mon, 12 Jan 2015 11:34:37 + Colin Guthrie gm...@colin.guthr.ie пишет: It seems that it's not activated here any longer, despite me having remote-fs.target enabled and, as remote-fs-pre.target is static, I cannot specifically enable it. It's my understanding that it should be automatically started. ... What else is meant to pull in remote-fs-pre.target? The only mention in the code is in src/core/mount.c but that's just for an After= dep, not a Requires= one. remote-fs-pre.target is intended to be pulled in by consumer; as per man page, unit that wants to be ordered before all remote mounts pulls it in via a Wants= type dependency. May be some other service has changed and lost this dependency? Ahh indeed, I must have missed that change back when it was tweaked around a bit a year or two back :s Although there is probably no harm in always starting it. Hmmm, yeah, but I now see a problem. I have fstab entries with nofail. This means that they do not prevent remote-fs.target from starting before they are mounted. This then allows systemd-user-sessions to start before they are mounted (it has After=remote-fs.target). If we enable remote-fs-pre.target, remote-fs.target is delayed until after that (it as After=remote-fs-pre.target). Now, remote-fs-pre.target won't start until the network is up, so the net result is that even with only nofail NFS fstab definitions, I don't get my graphical login until after the network is up! So darn! This breaks a few of my general desired behaviour and requirements! I'm wondering if the systemd-user-sessions After=remote-fs.target directive should be dropped and perhaps moved over to the fstab generator instead? That would probably keep me happy :D One for the discussion board at the sprint on the 30th maybe. Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
