Re: [systemd-devel] systemd-notify
On 18.04.19 18:17, Felipe Sateler wrote: > On Thu, 18 Apr 2019 14:21:09 +0200, Josef Moellers wrote: > >> Hi, >> >> We're currently working on a bug which afaict is due to a race >> condition: >> >> 1) systemd starts xenstored.service >> 2) /etc/xen/scripts/launch-xenstore does its work (starts /usr/lib/xen/ >> bin/init-xenstore-domain) >> 3) /etc/xen/scripts/launch-xenstore runs "systemd-notify --ready" >> 4) "systemd-notify --ready" sends a UDP-message to systemd 5) >> /etc/xen/scripts/launch-xenstore exits 6) systemd gets SIGCHLD and >> removes the PID from watch_pids[12] >> 7) systemd receives the UDP message, but >>a) the process is gone b) the PID is not in watch_pids[12] any more. >> 8) "Cannot find unit for notify message of PID..." >> 9) No start of depending units. >> >> I see no proper way to get out of this but to make the systemd-notify >> synchronous rather than fire-and-forget and expect it to wait for a >> response from systemd. > > If the launch-xenstore script starts a daemon and then exists: why not > make the unit Type=forking and thus forget about systemd-notify --ready ? The problem is that there is no PID to look after: xenstorage runs in another guest, so in the scope of systemd it simply has no PID. But we think that we found the problem: a patch, which straighted out handling of SIGCHLD and sd-notify messages, hadn't (yet) been included in that version of systemd. Lennart's reply pointed the way. Thanks anyway to all who have spent some time thinking about the issue. Happy Easter, Josef -- SUSE Linux GmbH Maxfeldstrasse 5 90409 Nuernberg Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify
On Thu, 18 Apr 2019 14:21:09 +0200, Josef Moellers wrote: > Hi, > > We're currently working on a bug which afaict is due to a race > condition: > > 1) systemd starts xenstored.service > 2) /etc/xen/scripts/launch-xenstore does its work (starts /usr/lib/xen/ > bin/init-xenstore-domain) > 3) /etc/xen/scripts/launch-xenstore runs "systemd-notify --ready" > 4) "systemd-notify --ready" sends a UDP-message to systemd 5) > /etc/xen/scripts/launch-xenstore exits 6) systemd gets SIGCHLD and > removes the PID from watch_pids[12] > 7) systemd receives the UDP message, but >a) the process is gone b) the PID is not in watch_pids[12] any more. > 8) "Cannot find unit for notify message of PID..." > 9) No start of depending units. > > I see no proper way to get out of this but to make the systemd-notify > synchronous rather than fire-and-forget and expect it to wait for a > response from systemd. If the launch-xenstore script starts a daemon and then exists: why not make the unit Type=forking and thus forget about systemd-notify --ready ? -- Saludos, Felipe Sateler ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify
On 18.04.19 15:35, Lennart Poettering wrote: > On Do, 18.04.19 14:21, Josef Moellers (jmoell...@suse.de) wrote: > >> Hi, >> >> We're currently working on a bug which afaict is due to a race condition: >> >> 1) systemd starts xenstored.service >> 2) /etc/xen/scripts/launch-xenstore does its work (starts >> /usr/lib/xen/bin/init-xenstore-domain) >> 3) /etc/xen/scripts/launch-xenstore runs "systemd-notify --ready" >> 4) "systemd-notify --ready" sends a UDP-message to systemd >> 5) /etc/xen/scripts/launch-xenstore exits >> 6) systemd gets SIGCHLD and removes the PID from watch_pids[12] >> 7) systemd receives the UDP message, but >>a) the process is gone >>b) the PID is not in watch_pids[12] any more. >> 8) "Cannot find unit for notify message of PID..." >> 9) No start of depending units. >> >> I see no proper way to get out of this but to make the systemd-notify >> synchronous rather than fire-and-forget and expect it to wait for a >> response from systemd. > > Hmm, the sd_notify() message handling actually runs at a higher > priority than the SIGCHLD handling, so that when both happen at the > same time we should always process the sd_notify message first, and > the SIGCHLD message second, precisey to avoid this problem. > > Which systemd version is this? Is this reproducible in current > upstream versions? It's SUSE systemd-v234+suse.381.g98de7a236 We're seeing this on one of our openQA SLES15 Virtualization-Milestone testing systems, haven't checked with the most recent upstream version yet. I'm not sure if we can easily switch to a different systemd version. Obviously, I accept your reservation. But what you wrote earlier about the ordering: although I am pretty convinced that the scenario is what happens, I'll re-check whether maybe the current upstream version has modifications that haven't found their way into our version. Thanks, Josef -- SUSE Linux GmbH Maxfeldstrasse 5 90409 Nuernberg Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify
On Do, 18.04.19 14:21, Josef Moellers (jmoell...@suse.de) wrote: > Hi, > > We're currently working on a bug which afaict is due to a race condition: > > 1) systemd starts xenstored.service > 2) /etc/xen/scripts/launch-xenstore does its work (starts > /usr/lib/xen/bin/init-xenstore-domain) > 3) /etc/xen/scripts/launch-xenstore runs "systemd-notify --ready" > 4) "systemd-notify --ready" sends a UDP-message to systemd > 5) /etc/xen/scripts/launch-xenstore exits > 6) systemd gets SIGCHLD and removes the PID from watch_pids[12] > 7) systemd receives the UDP message, but >a) the process is gone >b) the PID is not in watch_pids[12] any more. > 8) "Cannot find unit for notify message of PID..." > 9) No start of depending units. > > I see no proper way to get out of this but to make the systemd-notify > synchronous rather than fire-and-forget and expect it to wait for a > response from systemd. Hmm, the sd_notify() message handling actually runs at a higher priority than the SIGCHLD handling, so that when both happen at the same time we should always process the sd_notify message first, and the SIGCHLD message second, precisey to avoid this problem. Which systemd version is this? Is this reproducible in current upstream versions? Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] systemd-notify
Hi, We're currently working on a bug which afaict is due to a race condition: 1) systemd starts xenstored.service 2) /etc/xen/scripts/launch-xenstore does its work (starts /usr/lib/xen/bin/init-xenstore-domain) 3) /etc/xen/scripts/launch-xenstore runs "systemd-notify --ready" 4) "systemd-notify --ready" sends a UDP-message to systemd 5) /etc/xen/scripts/launch-xenstore exits 6) systemd gets SIGCHLD and removes the PID from watch_pids[12] 7) systemd receives the UDP message, but a) the process is gone b) the PID is not in watch_pids[12] any more. 8) "Cannot find unit for notify message of PID..." 9) No start of depending units. I see no proper way to get out of this but to make the systemd-notify synchronous rather than fire-and-forget and expect it to wait for a response from systemd. Josef -- SUSE Linux GmbH Maxfeldstrasse 5 90409 Nuernberg Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
Eelco Dolstra eelco.dolstra at logicblox.com writes: Hi all, I've noticed that the command systemd-notify --ready does not work reliably to signal that a service is ready. It works sometimes, but most of the time you get a message like: systemd[1]: Cannot find unit for notify message of PID 3137. in the journal, and the service stays in the activating state. The reason is that systemd-notify sends its message asynchronously and exits immediately. So by the time systemd processes the message, systemd-notify has probably already exited, and so systemd cannot gets cgroup. (Note that this affects other systemd-notify messages as well, but for --ready it's particularly bad because it causes services to hang in the activating state.) Any suggestions what to do about this? I can see a few solutions: * Have sd_notify() include its own unit name in the notification message. This would be insecure (though probably fine if the sender is root). However, it could be made secure by having systemd pass some random cookie to services via an environment variable, which sd_notify() could then include in its notification messages to authenticate itself. * Make systemd-notify synchronous, i.e., have it wait for a message back from systemd after it has determined the client's unit. Not entirely trivial given that sd_notify() uses SOCK_DGRAM. * Give each service its own notification socket, rather than using the global /run/systemd/notify. That is, in the service, set $NOTIFY_SOCKET to something like /run/systemd/notify-foo.service, and have systemd listen on that socket. By making the socket private to the service's mount namespace, you would know for sure that any message arriving on the socket comes from the service. * Document that you shouldn't use systemd-notify. Not an ideal solution What do you think? Hi Eelco, Ref: http://permalink.gmane.org/gmane.comp.sysutils.systemd.devel/18649 I am trying implement watchdog feature using sd_notify, could you pls give us more details on setting $NOTIFY_SOCKET. Like how to create socket and assign to environment NOTIFY_SOCKET for this [Unit] After=snmpd.service Before=exxxl.service nxx.service ixx.service lac [Service] Type=notify ExecStart=/data/ABC -d ExecStop=/data/pkill ABC TimeoutStopSec=5 Environment=NOTIFY_SOCKET= WatchdogSec=30s NotifyAccess=all FailureAction==reboot-force [Install] WantedBy=multi-user.target Best regards, Pradip. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
On Mon, 07.07.14 09:51, pradip K (pskoth...@gmail.com) wrote: Hi Eelco, Please dont hijack threads, open your own. I am trying implement watchdog feature using sd_notify, could you pls give us more details on setting $NOTIFY_SOCKET. Like how to create socket and assign to environment NOTIFY_SOCKET for this [Unit] After=snmpd.service Before=exxxl.service nxx.service ixx.service lac ^this is not a valid unit name [Service] Type=notify ExecStart=/data/ABC -d ExecStop=/data/pkill ABC TimeoutStopSec=5 Environment=NOTIFY_SOCKET= Dont do this, NOTIFY_SOCKET is implicitly set because you already set Type=notify. WatchdogSec=30s NotifyAccess=all FailureAction==reboot-force Use a single = only. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
Hi, On 18/05/14 18:54, Lennart Poettering wrote: Instead of sending the datagram with our own PID in the ucred field we could simply try to override it with our parent's PID. This only works if the service is running as root, right? But yeah, that's already an improvement over the current situation :-) -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
On Tue, 20.05.14 10:56, Eelco Dolstra (eelco.dols...@logicblox.com) wrote: Hi, On 18/05/14 18:54, Lennart Poettering wrote: Instead of sending the datagram with our own PID in the ucred field we could simply try to override it with our parent's PID. This only works if the service is running as root, right? But yeah, that's already an improvement over the current situation :-) Yupp, this only works with priviliges, but thankfully this should actually cover most scripts people want to run like this... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
On Thu, 24.04.14 23:51, Zbigniew Jędrzejewski-Szmek (zbys...@in.waw.pl) wrote: On Wed, Apr 23, 2014 at 08:50:34PM +0200, Lennart Poettering wrote: On Wed, 23.04.14 15:15, Eelco Dolstra (eelco.dols...@logicblox.com) wrote: Hi all, I've noticed that the command systemd-notify --ready does not work reliably to signal that a service is ready. It works sometimes, but most of the time you get a message like: systemd[1]: Cannot find unit for notify message of PID 3137. in the journal, and the service stays in the activating state. The reason is that systemd-notify sends its message asynchronously and exits immediately. So by the time systemd processes the message, systemd-notify has probably already exited, and so systemd cannot gets cgroup. (Note that this affects other systemd-notify messages as well, but for --ready it's particularly bad because it causes services to hang in the activating state.) Any suggestions what to do about this? I can see a few solutions: There is ongoing work to fix the kernel to add SCM_CGROUPS for us to messages. With that in place we have a race-free way to get this data for incoming messages. I have some hopes that this will soonishly enter the kernel, but then again, this story has been cookie for the past 5 years to no successs... What about simply waiting in the background for 10s? An ugly workaround, but should fix the issue until we have something better. What about this idea instead+: Instead of sending the datagram with our own PID in the ucred field we could simply try to override it with our parent's PID. This will not fix 100% of the cases, but I am quite sure it will fix most, since the parent process is usually the one that stays around and if you want to send READY=1, then you are likely to stay around for longer, so that the parent PID should be good enough. I am pretty sure we should make this change, regardless whether it fixes all or only a part of the cases, simply because I think it is the right thing to do, after all we also send MAINPID= of the paren PID, instead of our own, for a reason... Does this make sense? Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] systemd-notify --ready is not reliable
Hi all, I've noticed that the command systemd-notify --ready does not work reliably to signal that a service is ready. It works sometimes, but most of the time you get a message like: systemd[1]: Cannot find unit for notify message of PID 3137. in the journal, and the service stays in the activating state. The reason is that systemd-notify sends its message asynchronously and exits immediately. So by the time systemd processes the message, systemd-notify has probably already exited, and so systemd cannot gets cgroup. (Note that this affects other systemd-notify messages as well, but for --ready it's particularly bad because it causes services to hang in the activating state.) Any suggestions what to do about this? I can see a few solutions: * Have sd_notify() include its own unit name in the notification message. This would be insecure (though probably fine if the sender is root). However, it could be made secure by having systemd pass some random cookie to services via an environment variable, which sd_notify() could then include in its notification messages to authenticate itself. * Make systemd-notify synchronous, i.e., have it wait for a message back from systemd after it has determined the client's unit. Not entirely trivial given that sd_notify() uses SOCK_DGRAM. * Give each service its own notification socket, rather than using the global /run/systemd/notify. That is, in the service, set $NOTIFY_SOCKET to something like /run/systemd/notify-foo.service, and have systemd listen on that socket. By making the socket private to the service's mount namespace, you would know for sure that any message arriving on the socket comes from the service. * Document that you shouldn't use systemd-notify. Not an ideal solution :-) What do you think? -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd-notify --ready is not reliable
On Wed, 23.04.14 15:15, Eelco Dolstra (eelco.dols...@logicblox.com) wrote: Hi all, I've noticed that the command systemd-notify --ready does not work reliably to signal that a service is ready. It works sometimes, but most of the time you get a message like: systemd[1]: Cannot find unit for notify message of PID 3137. in the journal, and the service stays in the activating state. The reason is that systemd-notify sends its message asynchronously and exits immediately. So by the time systemd processes the message, systemd-notify has probably already exited, and so systemd cannot gets cgroup. (Note that this affects other systemd-notify messages as well, but for --ready it's particularly bad because it causes services to hang in the activating state.) Any suggestions what to do about this? I can see a few solutions: There is ongoing work to fix the kernel to add SCM_CGROUPS for us to messages. With that in place we have a race-free way to get this data for incoming messages. I have some hopes that this will soonishly enter the kernel, but then again, this story has been cookie for the past 5 years to no successs... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
