Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Dimitri John Ledkov]

On Wed, 26 Feb 2020 at 09:59, Andreas Kempe  wrote:
>
> Hello everyone,
>
> I'm working in a project with an embedded Linux system based on
> Openembedded using Systemd version 241 as our init process. We're
> using a read-only /etc. To facilitate development, we want to use a
> writeable overlay on /etc, but we ran into an issue.
>
> When we start, Systemd detects that there is no machine-id file
> present in /etc so it generates and mounts a /etc/machine-id. When our
> mount unit then applies the overlay on /etc, it hides the mounted
> file. Journald later fails to start because /etc/machine-id isn't
> visible through the overlay.
>

I would expect the /etc/machine-id to exist, and be an empty file on
the RO underlay, then systemd should setup machine-id in /run, and
then after the overlay of /etc is setup to use and is RW, fire the
systemd-machine-id-commit.service unit which will transfer the
machine-id from /run into the RW /etc overlay, after that everything
else should operate "normal".

We do this in Ubuntu live installer images, which use overlayfs across
all of / on top of read-only squashfs rootfs.


> At this point we're considering a number of workarounds, but I thought
> it worthwhile asking the experts before we go patching Systemd or
> similar.

I think didrocks or pitti introduced above for all the cases we had in
ubuntu where we have RO rootfs with a writable overlay which "appears"
later.

No idea if above is suitable for you at all, and/or need tweaking.
I.e. self-transfer machine-id from /run to /etc with like adding
wants=/before= systemd-machine-id-commit.service or some such?

-- 
Regards,

Dimitri.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Tue, Mar 31, 2020 at 03:52:44PM +0200, Lennart Poettering wrote:
> On Mi, 26.02.20 09:44, Andreas Kempe (andreas.ke...@actia.se) wrote:
> 
> > Hello everyone,
> >
> > I'm working in a project with an embedded Linux system based on
> > Openembedded using Systemd version 241 as our init process. We're
> > using a read-only /etc. To facilitate development, we want to use a
> > writeable overlay on /etc, but we ran into an issue.
> >
> > When we start, Systemd detects that there is no machine-id file
> > present in /etc so it generates and mounts a /etc/machine-id. When our
> > mount unit then applies the overlay on /etc, it hides the mounted
> > file. Journald later fails to start because /etc/machine-id isn't
> > visible through the overlay.
> 
> So when you swap out /etc you really should so so atomically, and
> ensure that at no point in time stuff (i.e. mounts) go missing while
> you shift things around. In particular not the machine-id.
> 

Yes, this is sound advice and of course the reasonable way to handling
moving file systems around. We weren't counting on having anything
extra mounted in /etc so it took us a bit by surprise.

> Hence, the code that replaces /etc with an overlayfs, why doesn't it
> bind mount /etc/machine-id into the new overlayfs before moving it
> into place? i.e. first mount the overlayfs to some tmpdir, bind mount
> /etc/machine-id into it, then use mount --move to move the fully
> prepped overlayfs along with all submounts to /etc. That way noone
> will notice anything changing.
> 

The simple answer for not doing the above is that we didn't think of
the idea. Personally, I don't have very much experience with overlays
and I am used to simply mounting things at boot where they should be
one thing at a time.

One learns something new every day. Thank you for the advice! :)

Cordially,
Andreas Kempe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Lennart Poettering]

On Mi, 26.02.20 09:44, Andreas Kempe (andreas.ke...@actia.se) wrote:

> Hello everyone,
>
> I'm working in a project with an embedded Linux system based on
> Openembedded using Systemd version 241 as our init process. We're
> using a read-only /etc. To facilitate development, we want to use a
> writeable overlay on /etc, but we ran into an issue.
>
> When we start, Systemd detects that there is no machine-id file
> present in /etc so it generates and mounts a /etc/machine-id. When our
> mount unit then applies the overlay on /etc, it hides the mounted
> file. Journald later fails to start because /etc/machine-id isn't
> visible through the overlay.

So when you swap out /etc you really should so so atomically, and
ensure that at no point in time stuff (i.e. mounts) go missing while
you shift things around. In particular not the machine-id.

Hence, the code that replaces /etc with an overlayfs, why doesn't it
bind mount /etc/machine-id into the new overlayfs before moving it
into place? i.e. first mount the overlayfs to some tmpdir, bind mount
/etc/machine-id into it, then use mount --move to move the fully
prepped overlayfs along with all submounts to /etc. That way noone
will notice anything changing.

Lennart

--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Thu, Feb 27, 2020 at 05:39:27PM +0200, Mantas Mikulėnas wrote:
> On Wed, Feb 26, 2020 at 11:59 AM Andreas Kempe 
> wrote:
> 
> > Hello everyone,
> >
> > I'm working in a project with an embedded Linux system based on
> > Openembedded using Systemd version 241 as our init process. We're
> > using a read-only /etc. To facilitate development, we want to use a
> > writeable overlay on /etc, but we ran into an issue.
> >
> > When we start, Systemd detects that there is no machine-id file
> > present in /etc so it generates and mounts a /etc/machine-id. When our
> > mount unit then applies the overlay on /etc, it hides the mounted
> > file. Journald later fails to start because /etc/machine-id isn't
> > visible through the overlay.
> >
> > At this point we're considering a number of workarounds, but I thought
> > it worthwhile asking the experts before we go patching Systemd or
> > similar.
> >
> > My gut feeling is that using overlays on /etc can't be that uncommon
> > and it is likely PEBKAC on our end. Is there some canonical way of
> > doing overlays with Systemd and we're screwing things up?
> >
> 
> If you have an initramfs, consider setting up the /etc overlay there
> instead.
> 

For the development assisting part of it, that is unfortunately not an
option for various reasons. We will have some overlays in the final
released product as well, but there we will mount them in the
initramfs so hopefully it won't be a problem then.

Cordially,
Andreas Kempe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Thu, Feb 27, 2020 at 11:53:45PM +0100, Jérémy ROSEN wrote:
> Le jeu. 27 févr. 2020 à 16:30, Andreas Kempe  a
> écrit :
> > I was contemplating whether it could be acceptable having the same
> > static machine-id file pre-generated for all systems. I'm not 100% sure
> > what it's used for, TBH; would it be a really bad idea?
> >
> 
> As long as two machines with the same machine-id are never in contact you
> should be fine...
> 
> Theoretically the machine-id should never cross the network, but you never
> know what individual apps might do
> 
> The only place where that could be problematic is the journal : if you mix
> the logs of multiple machines with the
> same machine-id, you won't be able to tell them appart and that might have
> other side-effects I wouldn't know about...
> 

AFAIK, we don't use the ID for anything in our own software and mixing
logs up shouldn't really be an issue, I think. I'll keep it in mind as
a potential workaround if nothing else.

Thank you for the help!
Cordially,
Andreas Kempe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Jérémy ROSEN]

Le jeu. 27 févr. 2020 à 16:30, Andreas Kempe  a
écrit :

> On Thu, Feb 27, 2020 at 10:04:37AM +0100, Jérémy ROSEN wrote:
>
> It is somewhat comforting knowing that others are seeing similar
> issues. :)
>
>
And not to far... you're a customer of ours :P
(well... actia in Toulouse is...)


> > I did a complete analysis of what's going on, with a patch that improves
> > the situation here : https://github.com/systemd/systemd/pull/14135
> > I am not sure how to deal with it in your specific case.
> > the simplest approch would be to mount your overlay in a initrd (or in a
> > small script shell that is run before systemd and exec systemd as its
> last
> > step)
> >
>
> I was contemplating whether it could be acceptable having the same
> static machine-id file pre-generated for all systems. I'm not 100% sure
> what it's used for, TBH; would it be a really bad idea?
>

As long as two machines with the same machine-id are never in contact you
should be fine...

Theoretically the machine-id should never cross the network, but you never
know what individual apps might do

The only place where that could be problematic is the journal : if you mix
the logs of multiple machines with the
same machine-id, you won't be able to tell them appart and that might have
other side-effects I wouldn't know about...


>
> > My patch wouldn't really help in your case, but maybe you can "cheat" by
> > having the underlying /etc/machine-id bein a symlink to the overlay
> > directory... that could work.
> >
>
> I had a look at your patch and as you said, it doesn't really solve
> our use case. At the moment, we decided to remove the overlay from the
> affected parts and simply require a new system image if one wants to
> change /etc.
>
> We were planning on having signed read-only overlays for configuration
> in the future so I guess we'll have to investigate this further at a
> later date.
>
> Thank you for taking the time to respond!
> Cordially,
> Andreas Kempe



-- 
[image: SMILE]  

20 rue des Jardins
92600 Asnières-sur-Seine
*Jérémy ROSEN*
Architecte technique

[image: email] jeremy.ro...@smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter]  [image: Facebook]
 [image: LinkedIn]
 [image: Github]


[image: Découvrez l’univers Smile, rendez-vous sur smile.eu]

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Mantas Mikulėnas]

On Wed, Feb 26, 2020 at 11:59 AM Andreas Kempe 
wrote:

> Hello everyone,
>
> I'm working in a project with an embedded Linux system based on
> Openembedded using Systemd version 241 as our init process. We're
> using a read-only /etc. To facilitate development, we want to use a
> writeable overlay on /etc, but we ran into an issue.
>
> When we start, Systemd detects that there is no machine-id file
> present in /etc so it generates and mounts a /etc/machine-id. When our
> mount unit then applies the overlay on /etc, it hides the mounted
> file. Journald later fails to start because /etc/machine-id isn't
> visible through the overlay.
>
> At this point we're considering a number of workarounds, but I thought
> it worthwhile asking the experts before we go patching Systemd or
> similar.
>
> My gut feeling is that using overlays on /etc can't be that uncommon
> and it is likely PEBKAC on our end. Is there some canonical way of
> doing overlays with Systemd and we're screwing things up?
>

If you have an initramfs, consider setting up the /etc overlay there
instead.

-- 
Mantas Mikulėnas
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Thu, Feb 27, 2020 at 10:04:37AM +0100, Jérémy ROSEN wrote:
> Le mer. 26 févr. 2020 à 10:59, Andreas Kempe  a
> écrit :
> 
> > Hello everyone,
> >
> > I'm working in a project with an embedded Linux system based on
> > Openembedded using Systemd version 241 as our init process. We're
> > using a read-only /etc. To facilitate development, we want to use a
> > writeable overlay on /etc, but we ran into an issue.
> >
> > When we start, Systemd detects that there is no machine-id file
> > present in /etc so it generates and mounts a /etc/machine-id. When our
> > mount unit then applies the overlay on /etc, it hides the mounted
> > file. Journald later fails to start because /etc/machine-id isn't
> > visible through the overlay.
> >
> > At this point we're considering a number of workarounds, but I thought
> > it worthwhile asking the experts before we go patching Systemd or
> > similar.
> >
> > My gut feeling is that using overlays on /etc can't be that uncommon
> > and it is likely PEBKAC on our end. Is there some canonical way of
> > doing overlays with Systemd and we're screwing things up?
> >
> > Thank you in advance for any help!
> > Cordially,
> > Andreas Kempe
>
> I had similar problems with a case of booting with the rootfs read-only and
> then becoming read-write later...
> 
> Basically systemd only checks for machine-id very early (before reading any
> config file) and does not deal well with /etc changing status...
> 

It is somewhat comforting knowing that others are seeing similar
issues. :)

> I did a complete analysis of what's going on, with a patch that improves
> the situation here : https://github.com/systemd/systemd/pull/14135
> I am not sure how to deal with it in your specific case.
> the simplest approch would be to mount your overlay in a initrd (or in a
> small script shell that is run before systemd and exec systemd as its last
> step)
> 

I was contemplating whether it could be acceptable having the same
static machine-id file pre-generated for all systems. I'm not 100% sure
what it's used for, TBH; would it be a really bad idea?

> My patch wouldn't really help in your case, but maybe you can "cheat" by
> having the underlying /etc/machine-id bein a symlink to the overlay
> directory... that could work.
> 

I had a look at your patch and as you said, it doesn't really solve
our use case. At the moment, we decided to remove the overlay from the
affected parts and simply require a new system image if one wants to
change /etc.

We were planning on having signed read-only overlays for configuration
in the future so I guess we'll have to investigate this further at a
later date.

Thank you for taking the time to respond!
Cordially,
Andreas Kempe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Jérémy ROSEN]

I had similar problems with a case of booting with the rootfs read-only and
then becoming read-write later...

Basically systemd only checks for machine-id very early (before reading any
config file) and does not deal well with /etc changing status...

I did a complete analysis of what's going on, with a patch that improves
the situation here : https://github.com/systemd/systemd/pull/14135
I am not sure how to deal with it in your specific case.
the simplest approch would be to mount your overlay in a initrd (or in a
small script shell that is run before systemd and exec systemd as its last
step)

My patch wouldn't really help in your case, but maybe you can "cheat" by
having the underlying /etc/machine-id bein a symlink to the overlay
directory... that could work.

Regards
Jeremy

Le mer. 26 févr. 2020 à 10:59, Andreas Kempe  a
écrit :

> Hello everyone,
>
> I'm working in a project with an embedded Linux system based on
> Openembedded using Systemd version 241 as our init process. We're
> using a read-only /etc. To facilitate development, we want to use a
> writeable overlay on /etc, but we ran into an issue.
>
> When we start, Systemd detects that there is no machine-id file
> present in /etc so it generates and mounts a /etc/machine-id. When our
> mount unit then applies the overlay on /etc, it hides the mounted
> file. Journald later fails to start because /etc/machine-id isn't
> visible through the overlay.
>
> At this point we're considering a number of workarounds, but I thought
> it worthwhile asking the experts before we go patching Systemd or
> similar.
>
> My gut feeling is that using overlays on /etc can't be that uncommon
> and it is likely PEBKAC on our end. Is there some canonical way of
> doing overlays with Systemd and we're screwing things up?
>
> Thank you in advance for any help!
> Cordially,
> Andreas Kempe
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>


-- 
[image: SMILE]  

20 rue des Jardins
92600 Asnières-sur-Seine
*Jérémy ROSEN*
Architecte technique

[image: email] jeremy.ro...@smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter]  [image: Facebook]
 [image: LinkedIn]
 [image: Github]


[image: Découvrez l’univers Smile, rendez-vous sur smile.eu]

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

Hello everyone,

I'm working in a project with an embedded Linux system based on
Openembedded using Systemd version 241 as our init process. We're
using a read-only /etc. To facilitate development, we want to use a
writeable overlay on /etc, but we ran into an issue.

When we start, Systemd detects that there is no machine-id file
present in /etc so it generates and mounts a /etc/machine-id. When our
mount unit then applies the overlay on /etc, it hides the mounted
file. Journald later fails to start because /etc/machine-id isn't
visible through the overlay.

At this point we're considering a number of workarounds, but I thought
it worthwhile asking the experts before we go patching Systemd or
similar.

My gut feeling is that using overlays on /etc can't be that uncommon
and it is likely PEBKAC on our end. Is there some canonical way of
doing overlays with Systemd and we're screwing things up?

Thank you in advance for any help!
Cordially,
Andreas Kempe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel