Re: [systemd-devel] early mounts in systemd

2021-05-03 Thread Lennart Poettering
On Fr, 30.04.21 15:14, Kenneth Porter (sh...@sewingwitch.com) wrote:

> --On Friday, April 30, 2021 11:39 AM -0400 Rick Winscot
>  wrote:
>
> > Early in the project it was decided to make the rootfs read-only... in an
> > effort to improve durability in environments where power fluctuations
> > might cause problems on the eMMC. At the same time, making logging (e.g.
> > /var) persistent for debugging was added to requirements. Persistent
> > storage would be achieved by mounting /var to a separate partition that is
> > read-write.
>
> Does /etc need to be read-only? On my last server I decided to make /usr
> read-only but root is writable and /var is part of that. I put /home on its
> own partition.

I think making /usr read-only makes a ton of sense.

The way I see it, besides the traditional Linux scheme where the whole
fs is writable the following two scenarios make the most sense, and
are what I personally intend to support in systemd very well:

1. root fs writable, /var/ part of it, but /usr/ separate and
   read-only/immutable.

2. rootfs read-only/immutable, /usr/ part of it, but /var/ separate
   and writable.

The main difference I that in the second case the configuration is
immutable too, while the firt case allows it to be changed locally.

Lennart

--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-03 Thread Lennart Poettering
On Fr, 30.04.21 18:23, Silvio Knizek (killermoe...@gmx.net) wrote:

> Am Freitag, dem 30.04.2021 um 10:39 -0400 schrieb Rick Winscot:
> > My question for anyone on the list, is the method outlined below a
> > reasonable solution to mounting /var early in the start-up cycle?
> >
> > Or... is there a better way? Some trimming
> >
>
> Hi Rick,
>
> by definition if you need to mount /var (or /usr for this argument),
> you need an initrd [1] which actually set up everything as you
> requires. Anything else has a tendency to break in unpleasant ways due
> to race conditions and such. You don't need much, just enough to set up
> everything required for the root and API file systems.

>From systemd's side we actually explicitly support environments where
/var is mounted after the initrd transition. From our side everything
should just work, we should have all the necessary deps in place to
make /var being mounted post-initrd but still during early boot just
work. I'd consider a bug in systemd if something of systemd#s own
components can't deal with /var/ being mounted after the transition.

(I mean, there have been discussions on whether we shouldn't require
/var to be mounted from initrd, but so far we didn't decide that this
was necessary, given the political effort this would take to require)

Lennart

--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-03 Thread Norbert Lange
I believe you can have a hard time if you dont do (the last step
atleast) with a var.mount file.
The mount files are special as systemd will pull those up implicitly
if they are needed.

Maybe you could rework your solution, so that one service mounts your
first or second choice in
a separate directory (/run/mymount) then you add a var.mount unit that
depends ("BindsTo") on said service
and bind-mounts /run/mymount to var.
That way you will hook into the existing systemd machinery.

I did something similar with an overlayfs once. Tested to work correctly,
even if you have further mounts like /var/tmp (systemd should
automatically add dependency to var.mount).

Norbert

# file rootfs-bindmount-var.service
[Unit]
Description=Bind-mount variable storage (/var)
Documentation=man:file-hierarchy(7)
ConditionPathIsSymbolicLink=!/var
# ConditionPathIsReadWrite=!/var
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=local-fs-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=-/bin/mkdir /run/varoverlay
ExecStartPre=/bin/mount --make-private -n -t tmpfs tmpfs_root_ovl
/run/varoverlay
ExecStartPre=/bin/mkdir /run/varoverlay/lower /run/varoverlay/upper
/run/varoverlay/work
ExecStart=/bin/mount --make-private -n --bind /var /run/varoverlay/lower
ExecStop=/bin/umount -n /run/varoverlay/lower
ExecStopPost=/bin/umount -n /run/varoverlay
ExecStopPost=/bin/rmdir /run/varoverlay


# file var.mount
[Unit]
Description=variable storage (/var)
Documentation=man:file-hierarchy(7)
ConditionPathIsSymbolicLink=!/var
ConditionPathIsReadWrite=!/var
After=rootfs-bindmount-var.service
BindsTo=rootfs-bindmount-var.service

[Mount]
What=overlay_var
Where=/var
Type=overlay
Options=lowerdir=/run/varoverlay/lower,upperdir=/run/varoverlay/upper,workdir=/run/varoverlay/work,redirect_dir=on,index=on,xino=on
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-02 Thread Rick Winscot
Luca, I have a similar setup where /var/log is not mounted in initramfs for
$reasons. I can’t believe I forgot to mention this!

Overriding with the tips you all have mentioned have things on the right
track!

Many thanks to the gurus of systemd!



On Sat, May 1, 2021 at 2:44 AM Luca Boccassi 
wrote:

> On Fri, 30 Apr 2021 at 22:15, Michael Biebl  wrote:
> >
> > Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot
> > :
> >
> > > At this point, flush is attempting to re-route /run/log/journal to
> /var/log/journal ... and the /var partition is not yet mounted. Units
> generated for fstab in /run/systemd/generator that manage the mount have an
> After=local-fs-pre.target which is too late.
> > >
> > > Other dependency errors appear in the log; all with the same root
> cause. By the time [ a specified service ] that uses /var is ready, the
> partition has not yet been mounted. My solution resolves the /var mount as
> soon as the block device is seen by udev - which made all the dependency
> errors go away.
> >
> > Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
> > correctly started after /var has been mounted.
> > In case you are interested, I attached a journalctl dump, /etc/fstab
> > and systemd-analyze dump as well.
> > As you can see, systemd-journal-flush.service has a proper
> > After=var.mount ordering.
> >
> > I wonder if you have a dependency loop somewhere and systemd resolves
> > this by removing that ordering.
>
> As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service -
> I had the same issue, because my /var/log partition is also not mounted in
> the initramfs for $reasons, so it appears late at boot.
> I'd suggest again to try and override it.
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Dave Howorth
On Sat, 1 May 2021 18:08:55 +0200
Michael Biebl  wrote:
> Am Sa., 1. Mai 2021 um 18:07 Uhr schrieb Michael Biebl
> :
> >
> > Am Sa., 1. Mai 2021 um 17:46 Uhr schrieb Dave Howorth
> > :  
> > > If systemd removes (i.e. doesn't obey) a directive, I'd expect it
> > > to be polite and log that fact somewhere. No?  
> >
> > It should, yes.  
> 
> To be precise, it should log a message like here
> https://sources.debian.org/src/systemd/247.3-5/src/core/transaction.c/?hl=414#L414

Many thanks :) Rick and/or Luca should be able to find that if it is the
source of their problems!

PS however you sent your mail, it meant that my MUA didn't
automatically send a copy to the list when I hit reply. I'd much prefer
to keep everything on list.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Michael Biebl
Am Sa., 1. Mai 2021 um 18:07 Uhr schrieb Michael Biebl :
>
> Am Sa., 1. Mai 2021 um 17:46 Uhr schrieb Dave Howorth 
> :
> > If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
> > polite and log that fact somewhere. No?
>
> It should, yes.

To be precise, it should log a message like here
https://sources.debian.org/src/systemd/247.3-5/src/core/transaction.c/?hl=414#L414
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Michael Biebl
Am Sa., 1. Mai 2021 um 17:46 Uhr schrieb Dave Howorth :
> If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
> polite and log that fact somewhere. No?

It should, yes.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Dave Howorth
On Fri, 30 Apr 2021 22:08:17 +0200
Michael Biebl  wrote:
> Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot
> :
> 
> > At this point, flush is attempting to re-route /run/log/journal
> > to /var/log/journal ... and the /var partition is not yet mounted.
> > Units generated for fstab in /run/systemd/generator that manage the
> > mount have an After=local-fs-pre.target which is too late.
> >
> > Other dependency errors appear in the log; all with the same root
> > cause. By the time [ a specified service ] that uses /var is ready,
> > the partition has not yet been mounted. My solution resolves
> > the /var mount as soon as the block device is seen by udev - which
> > made all the dependency errors go away.  
> 
> Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
> correctly started after /var has been mounted.
> In case you are interested, I attached a journalctl dump, /etc/fstab
> and systemd-analyze dump as well.
> As you can see, systemd-journal-flush.service has a proper
> After=var.mount ordering.
> 
> I wonder if you have a dependency loop somewhere and systemd resolves
> this by removing that ordering.

If systemd removes (i.e. doesn't obey) a directive, I'd expect it to be
polite and log that fact somewhere. No?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Michael Biebl
Am Sa., 1. Mai 2021 um 08:44 Uhr schrieb Luca Boccassi
:
>
> On Fri, 30 Apr 2021 at 22:15, Michael Biebl  wrote:
> >
> > I wonder if you have a dependency loop somewhere and systemd resolves
> > this by removing that ordering.
>
> As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service - I 
> had the same issue, because my /var/log partition is also not mounted in the 
> initramfs for $reasons, so it appears late at boot.
> I'd suggest again to try and override it.

I guess my point is, that before trying to suggest solutions, we
should understand what the underlying problem is first.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-05-01 Thread Luca Boccassi
On Fri, 30 Apr 2021 at 22:15, Michael Biebl  wrote:
>
> Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot
> :
>
> > At this point, flush is attempting to re-route /run/log/journal to
/var/log/journal ... and the /var partition is not yet mounted. Units
generated for fstab in /run/systemd/generator that manage the mount have an
After=local-fs-pre.target which is too late.
> >
> > Other dependency errors appear in the log; all with the same root
cause. By the time [ a specified service ] that uses /var is ready, the
partition has not yet been mounted. My solution resolves the /var mount as
soon as the block device is seen by udev - which made all the dependency
errors go away.
>
> Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
> correctly started after /var has been mounted.
> In case you are interested, I attached a journalctl dump, /etc/fstab
> and systemd-analyze dump as well.
> As you can see, systemd-journal-flush.service has a proper
> After=var.mount ordering.
>
> I wonder if you have a dependency loop somewhere and systemd resolves
> this by removing that ordering.

As mentioned earlier, I strongly suspect systemd-tmpfiles-setup.service - I
had the same issue, because my /var/log partition is also not mounted in
the initramfs for $reasons, so it appears late at boot.
I'd suggest again to try and override it.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Kenneth Porter
--On Friday, April 30, 2021 11:39 AM -0400 Rick Winscot 
 wrote:



Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations
might cause problems on the eMMC. At the same time, making logging (e.g.
/var) persistent for debugging was added to requirements. Persistent
storage would be achieved by mounting /var to a separate partition that is
read-write.


Does /etc need to be read-only? On my last server I decided to make /usr 
read-only but root is writable and /var is part of that. I put /home on its 
own partition.




___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Michael Biebl
Am Fr., 30. Apr. 2021 um 20:27 Uhr schrieb Rick Winscot
:

> At this point, flush is attempting to re-route /run/log/journal to 
> /var/log/journal ... and the /var partition is not yet mounted. Units 
> generated for fstab in /run/systemd/generator that manage the mount have an 
> After=local-fs-pre.target which is too late.
>
> Other dependency errors appear in the log; all with the same root cause. By 
> the time [ a specified service ] that uses /var is ready, the partition has 
> not yet been mounted. My solution resolves the /var mount as soon as the 
> block device is seen by udev - which made all the dependency errors go away.

Fwiw, I can't reproduce the problem. systemd-journal-flush.service is
correctly started after /var has been mounted.
In case you are interested, I attached a journalctl dump, /etc/fstab
and systemd-analyze dump as well.
As you can see, systemd-journal-flush.service has a proper
After=var.mount ordering.

I wonder if you have a dependency loop somewhere and systemd resolves
this by removing that ordering.


fstab
Description: Binary data


journal.txt.gz
Description: application/gzip


systemd-analyze.dump.txt.gz
Description: application/gzip
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Rick Winscot
15:31:18 localhost systemd[1]: Dependency failed for Flush Journal to
Persistent Storage.
░░ Subject: A start job for unit systemd-journal-flush.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit systemd-journal-flush.service has finished with a
failure.
░░
░░ The job identifier is 38 and the job result is dependency.

At this point, flush is attempting to re-route /run/log/journal to
/var/log/journal ... and the /var partition is not yet mounted. Units
generated for fstab in /run/systemd/generator that manage the mount have an
After=local-fs-pre.target which is too late.

Other dependency errors appear in the log; all with the same root cause. By
the time [ a specified service ] that uses /var is ready, the partition has
not yet been mounted. My solution resolves the /var mount as soon as the
block device is seen by udev - which made all the dependency errors go away.



On Fri, Apr 30, 2021 at 1:46 PM Michael Biebl  wrote:

> Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot
> :
> >
> > systemd 247
>
> Ok, thanks
>
> > /etc/systemd/journald.conf storage is persistent,
> systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.
> >
> > Mounting /var on a separate read-write partition handles the persistent
> log requirement as well as offloading other read-write operations that can
> no longer live on the rootfs... being read-only.
>
> Sure, but what is the actual problem? I do have systems with a
> separate /var and don't remember any ordering issues / race
> conditions.
> Do you have any error messages / journal logs which show the issue?
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Michael Biebl
Am Fr., 30. Apr. 2021 um 19:42 Uhr schrieb Rick Winscot
:
>
> systemd 247

Ok, thanks

> /etc/systemd/journald.conf storage is persistent, 
> systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.
>
> Mounting /var on a separate read-write partition handles the persistent log 
> requirement as well as offloading other read-write operations that can no 
> longer live on the rootfs... being read-only.

Sure, but what is the actual problem? I do have systems with a
separate /var and don't remember any ordering issues / race
conditions.
Do you have any error messages / journal logs which show the issue?
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Rick Winscot
systemd 247

/etc/systemd/journald.conf storage is persistent,
systemd-journal-flush.service has RequiresMountsFor=/var/log/journal.

Mounting /var on a separate read-write partition handles the persistent log
requirement as well as offloading other read-write operations that can no
longer live on the rootfs... being read-only.




On Fri, Apr 30, 2021 at 11:45 AM Michael Biebl  wrote:

> What is the actual problem you have with a separate /var and
> systemd-journald?
> For completeness sake, which systemd version do you have?
>
> Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot
> :
> >
> > We have an embedded product that uses a minimal Linux distribution
> generated via Buildroot.
> >
> > Early in the project it was decided to make the rootfs read-only... in
> an effort to improve durability in environments where power fluctuations
> might cause problems on the eMMC. At the same time, making logging (e.g.
> /var) persistent for debugging was added to requirements. Persistent
> storage would be achieved by mounting /var to a separate partition that is
> read-write.
> >
> > Several-hundred hours later... with many systemd-analyze reports and
> various configurations tested, we have determined that managing the /var
> mount with stadard services is not going to work due to tightly coupled and
> precisely timed dependencies. Attempts with /etc/fstab and the systemd
> generator are also unstable.
> >
> > Getting /var mounted in proximity to the initialization of
> systemd-journald.service seemed illusive.
> >
> > Several days ago I found a post on Stackoverflow that tied into udev
> triggers that seemed promising; resulting in the method outlined below.
> Initial testing shows proper timing with all dependencies satisfied.
> However, the solution feels... hackey.
> >
> > My question for anyone on the list, is the method outlined below a
> reasonable solution to mounting /var early in the start-up cycle?
> >
> > Or... is there a better way? Some trimming
> >
> >
> > Step One: Create a systemd service that mounts /var to the specified
> partition
> > Service:  /etc/systemd/system/var.service
> >
> > [Unit]
> > Description=service for mounting /var
> > DefaultDependencies=no
> >
> > [Service]
> > Type=oneshot
> > RemainAfterExit=yes
> > ExecStart=/bin/mount /dev/mmcblk0p6
> >
> >
> > Step Two: Add a nofail mount
> > fstab:/etc/fstab
> >
> > /dev/root / auto rw 0 1
> > /dev/mmcblk0p6 /var ext4 rw,nofail 0 0
> >
> >
> > Step Three: Add a wants dependency on the mount in udev triggers (some
> lines deleted for brevity)
> > Service:/usr/lib/systemd/system/systemd-udev-trigger.service
> >
> > [Unit]
> > ...
> > Wants=systemd-udevd.service var.service
> >
> > [Service]
> > ...
> > ExecStart=udevadm trigger --type=subsystems --action=add ;
> /usr/bin/udevadm trigger
> >
> >
> > Finally, systemd-analyze plot shows that the mount works as desired.
> >
> > systemd-udev-trigger.service
> > var.service
> > dev-mmcblk0p6.device
> > var.mount
> > 
> > systemd-remount-fs.service
> > systemd-journal-flush.service
> > local-fs-pre.target
> > 
> > ___
> > systemd-devel mailing list
> > systemd-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Silvio Knizek
Am Freitag, dem 30.04.2021 um 10:39 -0400 schrieb Rick Winscot:
> My question for anyone on the list, is the method outlined below a
> reasonable solution to mounting /var early in the start-up cycle?
>
> Or... is there a better way? Some trimming
>

Hi Rick,

by definition if you need to mount /var (or /usr for this argument),
you need an initrd [1] which actually set up everything as you
requires. Anything else has a tendency to break in unpleasant ways due
to race conditions and such. You don't need much, just enough to set up
everything required for the root and API file systems.

BR
Silvio

[1] https://systemd.io/INITRD_INTERFACE/

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Luca Boccassi
On Fri, 30 Apr 2021 at 16:45, Rick Winscot  wrote:
>
> We have an embedded product that uses a minimal Linux distribution
generated via Buildroot.
>
> Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations might
cause problems on the eMMC. At the same time, making logging (e.g. /var)
persistent for debugging was added to requirements. Persistent storage
would be achieved by mounting /var to a separate partition that is
read-write.
>
> Several-hundred hours later... with many systemd-analyze reports and
various configurations tested, we have determined that managing the /var
mount with stadard services is not going to work due to tightly coupled and
precisely timed dependencies. Attempts with /etc/fstab and the systemd
generator are also unstable.
>
> Getting /var mounted in proximity to the initialization of
systemd-journald.service seemed illusive.
>
> Several days ago I found a post on Stackoverflow that tied into udev
triggers that seemed promising; resulting in the method outlined below.
Initial testing shows proper timing with all dependencies satisfied.
However, the solution feels... hackey.
>
> My question for anyone on the list, is the method outlined below a
reasonable solution to mounting /var early in the start-up cycle?
>
> Or... is there a better way? Some trimming
>
>
> Step One: Create a systemd service that mounts /var to the specified
partition
> Service:  /etc/systemd/system/var.service
>
> [Unit]
> Description=service for mounting /var
> DefaultDependencies=no
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/bin/mount /dev/mmcblk0p6
>
>
> Step Two: Add a nofail mount
> fstab:/etc/fstab
>
> /dev/root / auto rw 0 1
> /dev/mmcblk0p6 /var ext4 rw,nofail 0 0
>
>
> Step Three: Add a wants dependency on the mount in udev triggers (some
lines deleted for brevity)
> Service:/usr/lib/systemd/system/systemd-udev-trigger.service
>
> [Unit]
> ...
> Wants=systemd-udevd.service var.service
>
> [Service]
> ...
> ExecStart=udevadm trigger --type=subsystems --action=add ;
/usr/bin/udevadm trigger
>
>
> Finally, systemd-analyze plot shows that the mount works as desired.
>
> systemd-udev-trigger.service
> var.service
> dev-mmcblk0p6.device
> var.mount
> 
> systemd-remount-fs.service
> systemd-journal-flush.service
> local-fs-pre.target
> 
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel

You don't need to synchronize with the actual journald unit - you need to
synchronize with systemd-journal-flush.service which is what moves the
journal from /run/log to /var/log.
The default flush unit is ordered with systemd-tmpfiles-setup which is also
somewhat early-boot - what I do is mask systemd-journal-flush.service, and
provide my own without that ordering constraint.

If you mount your /var via mount units/fstab generator, then
RequiresMountsFor=/var/log/journal will take care of the ordering
automagically.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] early mounts in systemd

2021-04-30 Thread Michael Biebl
What is the actual problem you have with a separate /var and systemd-journald?
For completeness sake, which systemd version do you have?

Am Fr., 30. Apr. 2021 um 16:39 Uhr schrieb Rick Winscot
:
>
> We have an embedded product that uses a minimal Linux distribution generated 
> via Buildroot.
>
> Early in the project it was decided to make the rootfs read-only... in an 
> effort to improve durability in environments where power fluctuations might 
> cause problems on the eMMC. At the same time, making logging (e.g. /var) 
> persistent for debugging was added to requirements. Persistent storage would 
> be achieved by mounting /var to a separate partition that is read-write.
>
> Several-hundred hours later... with many systemd-analyze reports and various 
> configurations tested, we have determined that managing the /var mount with 
> stadard services is not going to work due to tightly coupled and precisely 
> timed dependencies. Attempts with /etc/fstab and the systemd generator are 
> also unstable.
>
> Getting /var mounted in proximity to the initialization of 
> systemd-journald.service seemed illusive.
>
> Several days ago I found a post on Stackoverflow that tied into udev triggers 
> that seemed promising; resulting in the method outlined below. Initial 
> testing shows proper timing with all dependencies satisfied. However, the 
> solution feels... hackey.
>
> My question for anyone on the list, is the method outlined below a reasonable 
> solution to mounting /var early in the start-up cycle?
>
> Or... is there a better way? Some trimming
>
>
> Step One: Create a systemd service that mounts /var to the specified partition
> Service:  /etc/systemd/system/var.service
>
> [Unit]
> Description=service for mounting /var
> DefaultDependencies=no
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/bin/mount /dev/mmcblk0p6
>
>
> Step Two: Add a nofail mount
> fstab:/etc/fstab
>
> /dev/root / auto rw 0 1
> /dev/mmcblk0p6 /var ext4 rw,nofail 0 0
>
>
> Step Three: Add a wants dependency on the mount in udev triggers (some lines 
> deleted for brevity)
> Service:/usr/lib/systemd/system/systemd-udev-trigger.service
>
> [Unit]
> ...
> Wants=systemd-udevd.service var.service
>
> [Service]
> ...
> ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm 
> trigger
>
>
> Finally, systemd-analyze plot shows that the mount works as desired.
>
> systemd-udev-trigger.service
> var.service
> dev-mmcblk0p6.device
> var.mount
> 
> systemd-remount-fs.service
> systemd-journal-flush.service
> local-fs-pre.target
> 
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


[systemd-devel] early mounts in systemd

2021-04-30 Thread Rick Winscot
We have an embedded product that uses a minimal Linux distribution
generated via Buildroot.

Early in the project it was decided to make the rootfs read-only... in an
effort to improve durability in environments where power fluctuations might
cause problems on the eMMC. At the same time, making logging (e.g. /var)
persistent for debugging was added to requirements. Persistent storage
would be achieved by mounting /var to a separate partition that is
read-write.

Several-hundred hours later... with many systemd-analyze reports and
various configurations tested, we have determined that managing the /var
mount with stadard services is not going to work due to tightly coupled and
precisely timed dependencies. Attempts with /etc/fstab and the systemd
generator are also unstable.

Getting /var mounted in proximity to the initialization of
systemd-journald.service seemed illusive.

Several days ago I found a post on Stackoverflow that tied into udev
triggers that seemed promising; resulting in the method outlined below.
Initial testing shows proper timing with all dependencies satisfied.
However, the solution feels... hackey.

My question for anyone on the list, is the method outlined below a
reasonable solution to mounting /var early in the start-up cycle?

Or... is there a better way? Some trimming


Step One: Create a systemd service that mounts /var to the specified
partition
Service:  /etc/systemd/system/var.service

[Unit]
Description=service for mounting /var
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p6


Step Two: Add a nofail mount
fstab:/etc/fstab

/dev/root / auto rw 0 1
/dev/mmcblk0p6 /var ext4 rw,nofail 0 0


Step Three: Add a wants dependency on the mount in udev triggers (some
lines deleted for brevity)
Service:/usr/lib/systemd/system/systemd-udev-trigger.service

[Unit]
...
Wants=systemd-udevd.service var.service

[Service]
...
ExecStart=udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm
trigger


Finally, systemd-analyze plot shows that the mount works as desired.

systemd-udev-trigger.service
var.service
dev-mmcblk0p6.device
var.mount

systemd-remount-fs.service
systemd-journal-flush.service
local-fs-pre.target

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel