Re: [systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount
On Wed, Aug 25, 2021 at 3:44 PM Andrei Borzenkov wrote:
...
> > Here's the udev rule:
> > ```
> > ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*",
> > ENV{ID_FS_USAGE}=="filesystem", TAG+="systemd",
> > ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment@media-$name.service",
> > RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no
> > --options=ro --collect $devnode /media/$name"
> > ```
> >
> > And here's the systemd service:
> > It is templated and gets instantiated with "media-sdb1". It therefore has
> > an "After=media-sdb1.mount". I suspect Systemd-udevd executes the
> > ENV{SYSTEMD_WANTS} part before the RUN{program} part. Hence,
> > "media-sdb1.mount" doesn't yet exist when the service gets started, as it
> > gets created a tad later by systemd-mount.
> >
> > ```
> > [Unit]
> > Description=Start standalone Mender deployment (%i)
> > After=%i.mount
> >
> > [Service]
> > Type=oneshot
> > Restart=no
> > ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
> > ```
> >
...
>
> Hmm ... if systemd-mount --property accepts Wants and Before, your
> mount unit could pull in your service unit. I cannot test right now.
>
Yes, this seems to work, so in principle
RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no
--options=ro --collect --property
Wants=start-standalone-mender-deployment@media-$name.service $devnode
/media/$name"
is possible. Unfortunately this starts unit even if mount fails and
systemd-mount does not accept RequiredBy property". It is still
possible to add Requires to service itself.
[Unit]
Description=Start standalone Mender deployment (%i)
After=%i.mount
Requires=%i.mount
This will fail the service start job if the mount job fails.
Wants on mount unit pulls in service, so we are guaranteed to always
have both start jobs - for mount and for service and dependencies are
observed.
Re: [systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount
Andrei Borzenkov wrote on 25/08/2021 13:44:
On Wed, Aug 25, 2021 at 2:26 PM Manuel Wagesreither wrote:
Hello all,
this is my first post on this mailing list and, first of all, I'd like to thank
you and appreciate your work on systemd in general. I admire the logic, the
completeness of the manpages and in general how beautifully things are
engineered. I'm no unix graybeard and systemd saved me from having to learn all
that legacy stuff systemd replaces. Compared to fstab, /etc/network/interfaces
and init.d, systemd is a piece of art.
---
I'm working on an embedded device which should access and scan connected usb
drives for certain files. I seem to witness a race condition with my current
solution. I would ask for advice on how to implement this functionality in a
better way.
When a device /dev/sdb1 is connected, the udev rule below starts BOTH
* a systemd-service "start-standalone-mender-deployment@media-sdb1.service"
* `systemd-mount --no-block --automount=no --options=ro --collect /dev/sdb1
/media/sdb1`
The service then starts a shell script accessing the usb drive. Occasionally,
it says the directory the usb drive is mounted at is empty. When checking
manually, I see it's not. I strongly suspect the script accessed the directory
before the usb drive got mounted there.
Here's the udev rule:
```
ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*", ENV{ID_FS_USAGE}=="filesystem",
TAG+="systemd", ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment@media-$name.service", RUN{program}+="/usr/bin/systemd-mount
--no-block --automount=no --options=ro --collect $devnode /media/$name"
```
And here's the systemd service:
It is templated and gets instantiated with "media-sdb1". It therefore has an
"After=media-sdb1.mount". I suspect Systemd-udevd executes the ENV{SYSTEMD_WANTS} part before the
RUN{program} part. Hence, "media-sdb1.mount" doesn't yet exist when the service gets started, as it
gets created a tad later by systemd-mount.
```
[Unit]
Description=Start standalone Mender deployment (%i)
After=%i.mount
[Service]
Type=oneshot
Restart=no
ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
```
Can you confirm my theory?
This sounds reasonable. Yet again the same confusion - dependencies
are between jobs. "After=%i.mount" actually means "wait until start
job for %i.mount completes before selecting start job for your unit".
If the start job for your unit happens to be selected for execution
before the start job for %i.mount was queued (it is quite possible as
systemd-mount invocation takes time), there is no start job to wait
for and After becomes noop.
The only alternative I see is to invoke systemd-mount without --no-block from
the shell script itself. Instead of communicating the mount point (media-sdb1)
via unit template parameter, I would communicate the device path (/dev/sdb1) to
the template unit and pass it on to the shell script, which would determine
mount point based on that.
Yes, there is no nice solution. Systemd was not designed for dynamic
dependencies like in your case. While it is possible to add
Wants=%i.mount to your unit, there is no guarantee that this unit
definition will be present (due to the same race condition).
Sometimes in this case a unit definition file is created and
"systemctl daemon-reload" is invoked. This has its own can of worms so
can hardly be recommended.
So your approach is probably the most practical one.
Hmm ... if systemd-mount --property accepts Wants and Before, your
mount unit could pull in your service unit. I cannot test right now.
Not sure how well it works when mount points are involved but would a
.path unit work here?
e.g. move the start of the
start-standalone-mender-deployment@media-sdb1.service out of the udev
rule and instead worry about the mount path, not the device name?
So the udev rule just worries about mounting, then the path unit
triggers the start of the mender deployment?
Just a thought and not 100% how well it works with mount points as I
said as I've really not played much with them.
Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/
Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/
Re: [systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount
On Wed, Aug 25, 2021 at 2:26 PM Manuel Wagesreither wrote:
>
> Hello all,
>
> this is my first post on this mailing list and, first of all, I'd like to
> thank you and appreciate your work on systemd in general. I admire the logic,
> the completeness of the manpages and in general how beautifully things are
> engineered. I'm no unix graybeard and systemd saved me from having to learn
> all that legacy stuff systemd replaces. Compared to fstab,
> /etc/network/interfaces and init.d, systemd is a piece of art.
>
> ---
>
> I'm working on an embedded device which should access and scan connected usb
> drives for certain files. I seem to witness a race condition with my current
> solution. I would ask for advice on how to implement this functionality in a
> better way.
>
> When a device /dev/sdb1 is connected, the udev rule below starts BOTH
> * a systemd-service "start-standalone-mender-deployment@media-sdb1.service"
> * `systemd-mount --no-block --automount=no --options=ro --collect /dev/sdb1
> /media/sdb1`
>
> The service then starts a shell script accessing the usb drive. Occasionally,
> it says the directory the usb drive is mounted at is empty. When checking
> manually, I see it's not. I strongly suspect the script accessed the
> directory before the usb drive got mounted there.
>
> Here's the udev rule:
> ```
> ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*",
> ENV{ID_FS_USAGE}=="filesystem", TAG+="systemd",
> ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment@media-$name.service",
> RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no --options=ro
> --collect $devnode /media/$name"
> ```
>
> And here's the systemd service:
> It is templated and gets instantiated with "media-sdb1". It therefore has an
> "After=media-sdb1.mount". I suspect Systemd-udevd executes the
> ENV{SYSTEMD_WANTS} part before the RUN{program} part. Hence,
> "media-sdb1.mount" doesn't yet exist when the service gets started, as it
> gets created a tad later by systemd-mount.
>
> ```
> [Unit]
> Description=Start standalone Mender deployment (%i)
> After=%i.mount
>
> [Service]
> Type=oneshot
> Restart=no
> ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
> ```
>
> Can you confirm my theory?
>
This sounds reasonable. Yet again the same confusion - dependencies
are between jobs. "After=%i.mount" actually means "wait until start
job for %i.mount completes before selecting start job for your unit".
If the start job for your unit happens to be selected for execution
before the start job for %i.mount was queued (it is quite possible as
systemd-mount invocation takes time), there is no start job to wait
for and After becomes noop.
> The only alternative I see is to invoke systemd-mount without --no-block from
> the shell script itself. Instead of communicating the mount point
> (media-sdb1) via unit template parameter, I would communicate the device path
> (/dev/sdb1) to the template unit and pass it on to the shell script, which
> would determine mount point based on that.
>
Yes, there is no nice solution. Systemd was not designed for dynamic
dependencies like in your case. While it is possible to add
Wants=%i.mount to your unit, there is no guarantee that this unit
definition will be present (due to the same race condition).
Sometimes in this case a unit definition file is created and
"systemctl daemon-reload" is invoked. This has its own can of worms so
can hardly be recommended.
So your approach is probably the most practical one.
Hmm ... if systemd-mount --property accepts Wants and Before, your
mount unit could pull in your service unit. I cannot test right now.
> I'm all ears if you have comments or advice on that. I guess I'm not the
> first one implementing something like this.
>
>
> Regards,
> Manuel
>
> P.S.: I won't be able to respond until Sunday Aug 29th.
[systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount
Hello all,
this is my first post on this mailing list and, first of all, I'd like to thank
you and appreciate your work on systemd in general. I admire the logic, the
completeness of the manpages and in general how beautifully things are
engineered. I'm no unix graybeard and systemd saved me from having to learn all
that legacy stuff systemd replaces. Compared to fstab, /etc/network/interfaces
and init.d, systemd is a piece of art.
---
I'm working on an embedded device which should access and scan connected usb
drives for certain files. I seem to witness a race condition with my current
solution. I would ask for advice on how to implement this functionality in a
better way.
When a device /dev/sdb1 is connected, the udev rule below starts BOTH
* a systemd-service "start-standalone-mender-deployment@media-sdb1.service"
* `systemd-mount --no-block --automount=no --options=ro --collect /dev/sdb1
/media/sdb1`
The service then starts a shell script accessing the usb drive. Occasionally,
it says the directory the usb drive is mounted at is empty. When checking
manually, I see it's not. I strongly suspect the script accessed the directory
before the usb drive got mounted there.
Here's the udev rule:
```
ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*",
ENV{ID_FS_USAGE}=="filesystem", TAG+="systemd",
ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment@media-$name.service",
RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no --options=ro
--collect $devnode /media/$name"
```
And here's the systemd service:
It is templated and gets instantiated with "media-sdb1". It therefore has an
"After=media-sdb1.mount". I suspect Systemd-udevd executes the
ENV{SYSTEMD_WANTS} part before the RUN{program} part. Hence, "media-sdb1.mount"
doesn't yet exist when the service gets started, as it gets created a tad later
by systemd-mount.
```
[Unit]
Description=Start standalone Mender deployment (%i)
After=%i.mount
[Service]
Type=oneshot
Restart=no
ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
```
Can you confirm my theory?
The only alternative I see is to invoke systemd-mount without --no-block from
the shell script itself. Instead of communicating the mount point (media-sdb1)
via unit template parameter, I would communicate the device path (/dev/sdb1) to
the template unit and pass it on to the shell script, which would determine
mount point based on that.
I'm all ears if you have comments or advice on that. I guess I'm not the first
one implementing something like this.
Regards,
Manuel
P.S.: I won't be able to respond until Sunday Aug 29th.
Re: [systemd-devel] How does journald talks to other services?
Got it. What do we mean by "indexing" of logs for faster lookup? , Journlad does this I have seen this word in docs but no clear explanation, is it related to addition of timestamp, process_name, log_priority (info,debug...) to the log messages ? On Tue, 24 Aug 2021 at 11:54, Lennart Poettering wrote: > On Di, 24.08.21 09:11, Nishant Nayan (nayan.nishant2...@gmail.com) wrote: > > > So what are the cases where syslog forwards logs to journal? > > Is there a case where both journal and syslog end up sending same logs to > > each other ( like a cycle ) resulting in duplicate logs? > > systemd does not pick up messages from another syslog service, only from > syslog clients. Thus, there is no loop. > > Lennart > > -- > Lennart Poettering, Berlin >
[systemd-devel] required free space for reloading
Hello everyone, we use systemd on an embedded platform featuring the SAMA5D27C-D5M SoC [1], which has has 64 MiB of RAM. The software is based on DistroKit [2] and ptxdist [3]. tmpfs are mounted through /etc/fstab and are allowed to take 20% of the memory, which I think is quite much already. (The kernel itself reserves 17M for itself, with tmpfs mounted and only systemd (including dbus, udev, timesyncd) and NetworkManager running, there's less than 30M of free memory left for applications.) We run into this on first boot: [ 26.481404] systemd-rc-once[178]: Failed to reload daemon: Refusing to reexecute, not enough space available on /run/systemd. Currently, 9.2M are free, but a safety buffer of 16.0M is enforced. I digged in systemd code and issues and found #5219 [4]. In the discussion it is stated the current requirement of 16 MiB of free memory in /run/systemd was chosen quite arbitrary, without serious investigation of actual demand? I fully understand that requiring some amount of free memory is a solution to #5016 [5]. However the current limit does not seem to work well on systems with "low" memory like embedded systems. I patched systemd and changed the limit to 8 MiB which works fine for our usecase. Would such a patch have a chance to be accepted? Or are regressions for #5016 to be expected on other systems? What about trying to find a better solution than a rather high set fixed limit to be on the safe side? Is there any way to determine or estimate the amount of memory needed for reloading in advance at runtime? Thanks for reading Greets Alex [1] https://www.microchip.com/en-us/product/ATSAMA5D27C-D5M [2] https://www.pengutronix.de/en/software/distrokit.html [3] https://www.ptxdist.org/ [4] https://github.com/systemd/systemd/pull/5219 [5] https://github.com/systemd/systemd/issues/5016
Re: [systemd-devel] how to know what triggered a serivce start
Hi How to know what triggered a service start ? For eg, service X may Require service Y, so starting service-Y will start service-X. Can I get this info from 'systemd-analyze' ? If yes, which argument to use for systemd-analyze? I am interested to see what actually triggered and not just list out the dependency of a service Thanks
