On Nov 20, 2016, at 1:17 AM, Christian Seiler <christian.is...@gmail.com> wrote: > > On 11/19/2016 08:46 PM, The Lee-Man wrote: >> In this wonderful new world of systemd, I have an issue with stopping the >> iscsid service when the daemon has died or been killed. >> >> My setup: >> * I have an iscsid.socket unit file, which is enabled and started >> * I have an iscsid.service unit file, which controls the iscsid daemon. >> This is disabled and not started >> >> Normally, if I run a command like "iscsiadm -m discovery -t st -p >> SOME-TARGET", systemd notices that iscsiadm is trying to talk to iscsid >> through the socket, and it starts up iscsid. This is the cool part (IMHO) >> of systemd socket activation. >> >> When I want to stop iscsid, I can just tell systemctl to do it via >> "systemctl stop iscsid", and it runs the "ExecStop=" command in the service >> unit file, which is "iscsiadm -k 0 2" before terminating the daemon >> process(es). >> >> [NOTE: the "2" here in this command actually does nothing and is ignored, >> but I copied this from someplace else long ago, and the "2" was present >> there.] >> >> It is of importance, in this case, that the ExecStop command actually sends >> an IPC message to the daemon (iscsid) requesting it to cleanly shut itself >> down. Herein lies the rub. >> >> All of this works great until the daemon happens not to be running. You can >> simulate this with "kill -TERM $(pidof iscsid)" when the daemon is running. >> Now you are in a situation where systemd started the service and knows it >> is now not running, so it seems to send the ExecStop command to cleanly >> shut it down. This command hangs! It seems to be stuck in an infinite loop >> trying to send the shutdown command to the daemon, waiting for it to >> timeout, then trying again. The daemon starts up, sees an IPC error, and >> exits. > > The problem here is that iscsiadm tries to connect to the iSCSI socket. > This triggers systemd's socket activation, but systemd doesn't allow > conflicting jobs (the systemd terminology for an operation) to execute > at the same time, so it enqueues the start job due to socket activation > so that it will be executed _after_ the current stop job is complete. > > Unfortunately iscsiadm hangs until it gets a reply from the socket, so > the stop job hangs until the start job has been executed again.
In my case this can become an infinite loop. > >> While this clearly seems like a systemd issue, I have found a workaround >> that looks clean. Well, as clean as the shutdown command is, anyway. This >> involves: >> >> * modifying the "iscsiadm -k" command to require passing in the PID of the >> daemon to be killed >> * modifying the iscsiadm to only call kill_iscsid() if positive PID is >> passed in, and that PID exists. It verifies this by sending a blank signal >> to the process. >> * modifying the iscsid.service systemd unit file so that the ExecStop >> command becomes "iscsiadm -k 0 $MAINPID" > > That seems like a reasonable way to work around this problem. > >> I know other distributions are having been dealing with iscsid service >> shutdown issues with systemd for a while now. Perhaps somebody else has a >> better solution? > > In Debian we just don't use socket activation for iscsid, but just > install a systemd service file alone. I think I had trouble with > socket activation (may well have been the same problem you had, I > don't remember), but when I actually thought about it in more detail, > I didn't see the point in having socket activation for iscsid in the > first place, at least not in Debian, so I just dropped that part. > > The rationale for why we don't make use of socket activation: > > - we unconditionally execute iscsiadm -m node --loginall=automatic > at boot if open-iscsi is installed > > - that will cause iscsid to start at boot even with socket > activation, and stay active for the entirety of the running > system > > => this means that even if there are no configured sessions > we still don't save time at boot > > - since iscsid is required to be kept around for session recovery > it can't auto-exit on idle, because an active session means it > should never be idle (and there's currently no support in the > code for that) > > - the corner case where people have iscsid installed but don't use > it didn't seem that relevant, especially since iscsid uses less > than 4 MiB of RAM on my system if unused > > - but even if we were to make iscsiadm not contact iscsid when > using --loginall=automatic and no sessions are configured, this > still wouldn't help us, because there still could be sessions > configured in the initramfs, for which we'd want to have iscsid > running at boot to perform session recovery > > The only thing I could think of to make this at least somewhat > useful is to have iscsid exit if there are no active sessions and > a certain time has passed => exit on idle. Doesn't save boot time, > but saves a small amount of resources in the corner case where the > user has no active sessions. I use socket activation for SUSE because we want open-iscsi installed by default but not running. There was an effort to reduce the number of default-running services, and it takes an act of congress to add a new default-running service now. The approach now is to give the end-user more choice in which services are running, rather than choosing for them. But socket activation allows iscsid to only be started the first time it’s needed, and that was an acceptable alternative to having the daemon running all the time. > > However, that still leaves a potential hang on shutdown: if iscsid > is killed manually (or even stopped via systemctl stop), and then > the system is rebooted, at least on Debian systems we do iscsiadm > -m node --logoutall=all on shutdown. That tries to contact the > socket, and that will now also hang on shutdown (because systemd > will refuse to start iscsid while a shutdown is in progress), and > after 90s (default job timeout in systemd) the logout job will be > forcefully killed. If you stop iscsid manually without socket > activation being active, it will at least not hang (although the > session logout will still not take place either, I don't have a > good solution for that). We have a separate service for iscsi sessions, layered on top of the daemon service, but I can see this sequence happening for us if the user has the session service enabled but manually kills iscsid. [Have I mentioned I mostly dislike systemd?] > > Personally, I'm still of a mind that socket activation doesn't > really provide a tangible benefit in the case of iscsid, so unless > someone has a convincing argument to the contrary, I think at > least in Debian we'll not use it and just always start iscsid as > a regular service. If you have iscsid running virtually all the time, I agree you don’t need socket activation for it. > > Regards, > Christian > -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
