networkd: learned DNS server not propagated to rdnss option in RA

[Michael Heimpold]

Dear all,

I'm trying to setup an embedded system as IPv6 router using IPv6
prefix delegation. The system has two network interfaces, eth0 and
eth1. eth0 is the upstream interface, but it is part of a bridge interface,
but I think this does not play an important role here.
The upstream interfaces acquires an IPv6 via RA and also learns
the DNS server from the upstream router. IPv6 and DNS resolving
works properly on the system.
(note: the upstream interface is dual-stack, IPv4 is also working as usual).

The goal is to pass routable IPv6 addesses down to a system connected
via eth1 ("client"). In my understanding, this is what IPv6 prefix delegation
is for and it is working so far, that the client acquires an IPv6 and I can
ping IPv6 addresses on the Internet etc.
However, when I look at the IPv6 RA sent out on eth1 interface, the rdnss 
option is missing, so that the client does not learn any DNS server over the
link.

Please find below my configuration files. My understanding of the
documentation is, that this should work out-of-the-box since all related
config options have reasonable defaults (EmitDNS, UseDNS...), so that
learned DNS server from upstream interfaces should propagate to the RAs
of the downstream interface. But this is not the case.
If I manually configure DNS=... option in the "IPv6SendRA" section, then a
rdnss field is part of the RA and then the client learns the DNS server and
I also have DNS on the client working. So this is not an issue on the
client itself.

On my system, I'm using systemd-networkd v250.5 from Yocto kirkstone,
but I also tested v254.4 with no difference in behavior.

My question is whether I have a wrong understanding or might this be a bug?
Any hints are appreciated.

Thanks,
Michael


--snip: /lib/systemd/network/br0.network--
[Match]
Name=br0

[Network]
DHCP=yes
IPv6AcceptRA=yes

[DHCPv4]
ClientIdentifier=mac
UseHostname=no

[DHCPv6]
UseHostname=no
--snap--

--snip: /etc/systemd/network/eth1.network--
[Match]
Name=eth1

[Network]
DHCP=no
IPv6AcceptRA=no
IPForward=ipv6
IPv6SendRA=yes
DHCPPrefixDelegation=yes
--snap--

--snip: output of resolvectl on the embedded system (shortened)--
...

Link 5 (br0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/
unsupported
   DNS Servers: 192.168.8.254 fd00::c614:6dff:fed6:c2e9
--snap--

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Mantas Mikulėnas]

On Tue, Dec 12, 2023 at 6:15 PM Christopher Wong 
wrote:

> Hi Mantas,
>
>
>
> After user@1001.service failed, it trigger the stopping process and
> become inactive.
>

Ah yeah, that makes sense, user-runtime-dir@ has StopWhenUnneeded=yes – so
of course after user@1001 crashes you're not going to see anything mounted
anymore.

Could you try temporarily removing that option / setting it to 'no', just
to see what changes?


>
>
> ○ user-runtime-dir@1001.service - User Runtime Directory /run/user/1001
>
>  Loaded: loaded (/etc/systemd/system/user-runtime-dir@.service;
> static)
>
> Drop-In: /usr/lib/systemd/system/service.d
>
>  └─10-axis.conf, 20-axis-sandbox.conf
>
>  Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago
>
>Duration: 315ms
>
>Docs: man:user@.service(5)
>
> Process: 16325 ExecStartPre=ls -la /run/user (code=exited,
> status=0/SUCCESS)
>
> Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS)
>
> Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir
> start 1001 (code=exited, status=0/SUCCESS)
>
> Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS)
>
> Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited,
> status=0/SUCCESS)
>
> Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS)
>
> Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir
> stop 1001 (code=exited, status=0/SUCCESS)
>
>Main PID: 16329 (code=exited, status=0/SUCCESS)
>
> CPU: 48ms
>
>
>
> /etc/fstab don’t include anything on /run/user/1001 and there is no mount
> unit for run-user-1001.mount either.
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas 
> *Date: *Tuesday, 12 December 2023 at 17:05
> *To: *Christopher Wong 
> *Cc: *Systemd 
> *Subject: *Re: [systemd-devel] Manual start of user@.service failed
> with permission denied
>
> That sounds like it's getting immediately unmounted (or maybe not being
> mounted at all despite the program doing so).
>
>
>
> Does the user-runtime-dir service continue to show as "active" after this,
> or does it return to "inactive"?
>
>
>
> Does your /etc/fstab have any mentions of /run/user/1001? Or more
> generally, are there any run-user-1001.mount units? (If you 'systemctl
> status' this unit, does the status include a source path?)
>
>
>
> On Tue, Dec 12, 2023, 17:34 Christopher Wong 
> wrote:
>
> Hi Mantas,
>
>
>
> I currently have the following flow:
>
>
>
>1. No /run/user/1001 directory
>2. systemctl start user@1001.service
>3. systemd start user-runtime-dir@1001.service which ends successfully.
>4. The directory /run/user/1001 exists now, but is empty, owned by
>root with mode 0700
>5. I don’t have findmnt on my system, so I used mount, but
>/run/user/1001 is not listed.
>6. systemd start user@1001.service which fails due to permission
>denied.
>
>
>
> I can’t explain why the /run/user/1001 is owned by root after
> user-runtime-dir@1001.service successfully exited. I added some personal
> print in systemd code to ensure that the mount command returned success
> (r=0). Although, the mount was successful the command “mount” didn’t list
> it. In the list of mounts starting with /run I could only find these
> entries:
>
>
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs
> (rw,nosuid,nodev,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs
> (ro,nosuid,nodev,noexec,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type
> tmpfs (ro,nosuid,nodev,mode=755)
>
>
>
> If I do a chown of the directory in user@1001.service then it works
>
>
>
> root@host:/run/user# ls -la 1001
>
> drwx--3 ida  root80 Dec 12 16:19 .
>
> drwxr-xr-x3 root root60 Dec 12 16:19 ..
>
> srw-rw-rw-1 ida  ssh-user 0 Dec 12 16:19 bus
>
> drwxr-xr-x5 ida  ssh-user   140 Dec 12 16:19 systemd
>
>
>
> The ”mount” command don’t list /run/user/1001 for the successful case
> either.
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas 
> *Date: *Monday, 11 December 2023 at 17:56
> *To: *Christopher Wong 
> *Cc: *Systemd 
> *Subject: *Re: [systemd-devel] Manual start of user@.service failed
> with permission denied
>
> On Mon, Dec 11, 2023, 17:28 Christopher Wong 
> wrote:
>
> Hi Mantas,
>
>
>
> I have added ExecStartPre to user@.service to run “id” and “ls -la”:
>
>
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount
> /run/user/1001 owned by 1001:118
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs
> (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV
> "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
>
> Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory
> /run/user/1001.
>
> Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
>
> Dec 11 

RE: networkd RetransmitSec - how to make it work on a host?

[Muggeridge, Matt]

> -Original Message-
> From: Lennart Poettering 
> Sent: Wednesday, December 13, 2023 7:32 AM
> To: Muggeridge, Matt 
> Cc: systemd-devel@lists.freedesktop.org
> Subject: Re: networkd RetransmitSec - how to make it work on a host?
> 
> On Mo, 11.12.23 02:49, Muggeridge, Matt (matt.muggerid...@hpe.com)
> wrote:
> 
> > The RetransmitSec option was introduced in systemd-v255, but I cannot
> > get it to work for Neighbor Solicitations from a Host. Instead, I
> > observe that the NS are always transmitted at 1 second intervals,
> > regardless of whether it was changed by:
> 
> Please file this as git issue. It sounds like a bug report, which should 
> really go to
> github.
> 
> Lennart
> 
> --
> Lennart Poettering, Berlin

There's probably something about it that I've misunderstood, which is why I 
asked here.

Ok, I will raise a github issue.

Cheers,
Matt.

RE: IPv6 Compliance for networkd

[Muggeridge, Matt]

> -Original Message-
> From: Lennart Poettering 
> Sent: Wednesday, December 13, 2023 4:22 AM
> To: Muggeridge, Matt 
> Cc: systemd-devel@lists.freedesktop.org
> Subject: Re: IPv6 Compliance for networkd
> 
> On Mo, 11.12.23 19:14, Muggeridge, Matt (matt.muggerid...@hpe.com)
> wrote:
> 
> > Hello, networkd developer community,
> >
> > I am hoping to rally support for making networkd IPv6 compliant and
> > I'm will to help, but cannot do it alone. Is there any interest in
> > making systemd-networkd IPv6 compliant?
> 
> Well, interest is relative. I think for most people IPv6 already works well
> enough, they don't really care about compliance programs on this so much.
> But as long as the requirements are reasonable they also wouldn't mind (and
> prefer) if networkd passes those qualifications.
> 
> > There are many organizations (especially US Government) that mandate
> > IPv6 compliance (USGv6).  Products that are dependent on networkd
> > cannot be bid to these customers.
> 
> For the people currently involved with networkd upstream this is not a top
> priority. If this is important to you however, that's great, we are happy to
> review/merge patches.
> 
> > How do I engage with the right people in the developer community?
> 
> Send PRs via github.
> 
> > Thanks,
> > Matt.
> >
> > PS: Mailing list topics go unanswered and github issues get lost in
> > the noise, so I'm hoping there's a more efficient way to collaborate.
> 
> It's an Open Source project: if something matters a lot to you, then please 
> file
> PRs to get the work merged. We generally try to review PRs sooner or later,
> but we are swamped with work, so it might take a while. Just filing issues
> (while also appreciated) will usually not magically make somebody work on
> this for you though. It's kinda the same with most open source projects btw.
> 
> If this is something you'd like to see addressed soon, I'd recommend maybe
> paying some consultancy (we have worked with codethink on some projects,
> they should be willing to work on this, are capable and now hot get stuff in
> systemd done).
> 
> If you don't have the cash for that, it might work to get funding from this 
> from
> organizations such as the German STF and things like that. I am pretty sure
> that the US has something similar?
> 
> Anyway, judging by your email address I understand you work for HPE, so I'd
> assume your company actually has the funds to payroll this though, if this
> matters to you.
> 
> Lennart
> 
> --
> Lennart Poettering, Berlin

Thanks Lennart.  That all makes sense.

Cheers,
Matt.
 

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Nils Kattenbeck]

On Tue, Dec 12, 2023 at 10:02 PM Lennart Poettering
 wrote:
>
> If you have 7 cpio initrds then the kernel will allocate a tmpfs and
> unpack them all into it, one after the other, on top of each other,
> and then jumps into the result.
>
> if you have an erofs and 7 cpio initds, what are you going to do? You
> cannot extract into an erofs, it's immutable. You'd need something
> like overlayfs, but that would require (at least for now) an
> additional step in userspace, which is something to avoid.
>
> Alternatively (and preferred by me) would support a mode where it
> would unpack any cpios it gets into a tmpfs, and then pass an fsopen()
> fd to that to the executable it then invokes from the erofs. the
> executable could then mount that somewhere if it wants. But this would
> require a kenrel patch.

Such a kernel patch would likely be the more advanced method.
I also saw that they now wrote to the LKML to potentially discuss
something like this.
The method with an overlaysfs would likely be easier for init systems
to use but also less customizable.

> > Even if everything is the same there are codes paths which might not
> > be taken during usual operation. An example would be services similar
> > to the new systemd-bsod which are only triggered in emergencies.
> > Having these in the cpio means that they will always be read and
> > decompressed.
>
> systemd-bsod is tiny though, less than 8K compressed here. Not sure it
> is a good example.

Yes that is right though it is the first and most universal thing
which came to mind.
A better example would be something like a fleet management SDK (in
Java or a similar language with a runtime) which phones to a
management server indicating a boot failure and publishing crash logs.

> > Using sysexts also has the drawback that each and every one of them
> > has to be decompressed. I might be mistaken but I expect that this
> > will be the case even if the extension-release in the sysext results
> > in it being discarded which is obviously another big drawback.
>
> sysexts are erofs or squashfs file systems with verity backing. Only
> the sectors you access are decompressed.

Okay I forgot that they were erofs based and mentioned cpio archives
so I assumed they would be one.
Do they need to be fully read from disk to generate the cpio archive?

> Lennart
>
> --
> Lennart Poettering, Berlin

Re: networkd RetransmitSec - how to make it work on a host?

[Lennart Poettering]

On Mo, 11.12.23 02:49, Muggeridge, Matt (matt.muggerid...@hpe.com) wrote:

> The RetransmitSec option was introduced in systemd-v255, but I
> cannot get it to work for Neighbor Solicitations from a
> Host. Instead, I observe that the NS are always transmitted at 1
> second intervals, regardless of whether it was changed by:

Please file this as git issue. It sounds like a bug report, which
should really go to github.

Lennart

--
Lennart Poettering, Berlin

Re: systemd units disabled when calling systemctl daemon-reload

[Lennart Poettering]

On Di, 12.12.23 19:06, Etienne Cordonnier (ecordonn...@snap.com) wrote:

> Hello,
> I am debugging some embedded system running systemd. The behavior I am
> observing is that many systemd targets such as multi-user.target are
> disabled after I run systemctl daemon-reload (as shown by systemctl
> list-units --type target --all). This causes many systemd units to be
> disabled, and forces me to reboot the system.

What do you mean by "disabled"?

in systemd targets can be active and inactive, and that's what
"systemctl list-unit" shows. They can also be enabled/disabled, but
that's what "systemctl list-unit-files" shows. But targets such as
multi-user.target cannot be enabled nor disabled, they are considered
"static", i.e. always enabled if you so will. Which "systemctl
list-unit-file" should actually show.

Hence, I don#t really grok what you are trying to say here...

> Is there a way to debug this systemd target transition? I already
> enabled systemctl
> log-level debug, but I still don't understand why the systemd target is
> changing when I call systemctl daemon-reload on this particular system.

Please state OS, systemd version and provide relevant logs. Otherwise
this is not actionable.

Lennart

--
Lennart Poettering, Berlin

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Lennart Poettering]

On Di, 12.12.23 21:34, Nils Kattenbeck (nilskem...@gmail.com) wrote:

> Hi, while I have been following this thread passively for now I also
> wanted to chime in.
>
> > (The main reason why sd-stub doesn't actually support erofs-initrds,
> > is that sd-stub also generates initrd cpios on the fly, to pass
> > credentials and system extension images to the kernel, and you can't
> > really mix erofs and cpio initrds into one)
>
> What prevents one from mixing the two (especially given that the
> hypothetical erofs initrd support does not yet exist)?
> Or are you talking about mixing this with your memmap+root=/dev/pmem
> suggestion?

If you have 7 cpio initrds then the kernel will allocate a tmpfs and
unpack them all into it, one after the other, on top of each other,
and then jumps into the result.

if you have an erofs and 7 cpio initds, what are you going to do? You
cannot extract into an erofs, it's immutable. You'd need something
like overlayfs, but that would require (at least for now) an
additional step in userspace, which is something to avoid.

Alternatively (and preferred by me) would support a mode where it
would unpack any cpios it gets into a tmpfs, and then pass an fsopen()
fd to that to the executable it then invokes from the erofs. the
executable could then mount that somewhere if it wants. But this would
require a kenrel patch.

> Even if everything is the same there are codes paths which might not
> be taken during usual operation. An example would be services similar
> to the new systemd-bsod which are only triggered in emergencies.
> Having these in the cpio means that they will always be read and
> decompressed.

systemd-bsod is tiny though, less than 8K compressed here. Not sure it
is a good example.

> Using sysexts also has the drawback that each and every one of them
> has to be decompressed. I might be mistaken but I expect that this
> will be the case even if the extension-release in the sysext results
> in it being discarded which is obviously another big drawback.

sysexts are erofs or squashfs file systems with verity backing. Only
the sectors you access are decompressed.

Lennart

--
Lennart Poettering, Berlin

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Eric Curtin]

On Tue, 12 Dec 2023 at 20:35, Nils Kattenbeck  wrote:
>
> Hi, while I have been following this thread passively for now I also
> wanted to chime in.
>
> > (The main reason why sd-stub doesn't actually support erofs-initrds,
> > is that sd-stub also generates initrd cpios on the fly, to pass
> > credentials and system extension images to the kernel, and you can't
> > really mix erofs and cpio initrds into one)
>
> What prevents one from mixing the two (especially given that the
> hypothetical erofs initrd support does not yet exist)?
> Or are you talking about mixing this with your memmap+root=/dev/pmem 
> suggestion?
>
> > The try to optimize the initrd a bit by making it an erofs/memmap
> > thing and so on. And make sure the initrd only contains stuff you
> > always need, so that reading it all into memory is necessary anyway,
> > and hence any approach that tries to run even the initrd off a disk
> > image won't be necessary becuase you need to read everything anyway.
>
> Having to ensure that the initrd is as small as possible is definitely
> no easy task.
> Furthermore unless one has total control over the devices, or even if
> there are only a few hardware revisions, parts of the initrd might not
> be used.
> Even if everything is the same there are codes paths which might not
> be taken during usual operation. An example would be services similar
> to the new systemd-bsod which are only triggered in emergencies.
> Having these in the cpio means that they will always be read and
> decompressed.
> Using sysexts also has the drawback that each and every one of them
> has to be decompressed. I might be mistaken but I expect that this
> will be the case even if the extension-release in the sysext results
> in it being discarded which is obviously another big drawback.
>
> Regardless, even if every single file within the cpio archive (and
> potential sysexts) is used, erofs still has a distinct advantage over
> cpio!
> With cpio everything has to be decompressed and read up front. With
> erofs this is not the case.
> Only the fs header has to be read at first as files are decompressed on 
> demand.
> This means that critical stuff can be started earlier as it does not
> have to wait for decompression of stuff only needed later on.
> For example an initrd-only (i.e. not pivolint root), graphical system
> could start all background services long before the UI starts and
> accesses large asset files.
>
> I agree that this splitting up into another micro-initrd just for some
> storage stuff etc (which I still have not groked completely) does not
> seem to offer any advantages to what we have today. *However*, I
> certainly think that standardizing and supporting some kind of erofs
> based initrd would gain some advantages.

Are we sure? A bunch of stuff in modern initrd's today have nothing to
do with mounting storage. I've proved there's benefit to that with the
data on the initoverlayfs page, you save ~300ms on systemd start time
on a Raspberry Pi 4 with an sd card, if you use an NVMe drive over USB
on a Raspberry Pi 4 it's even more... ~500ms. I wouldn't say that's
insignificant. You still get all the functionality of the fully
fledged initramfs when systemd starts but you save between 300ms and
500ms.

>
> On the other hand this feels like going back to an old ramdisk again.
> This goes beyond my knowledge but based on the kernel docs most
> drawbacks of ramdisks would not apply to an approach with erofs. Also
> maybe the more flexible loopback devices could be used(?) which might
> alleviate some problems.

For the record, this is what we are doing for initoverlayfs at the
moment, mounting "/boot" partition and then loopback. There are
significant advantages as there are few bytes read until you start
using initoverlayfs.

/boot/initramfs-6.5.12-200.fc38.x86_64.img
/boot/initoverlayfs-6.5.12-200.fc38.x86_64.img

>
> -- This block device was of fixed size, so the filesystem mounted on
> it was of fixed size.
>-> Should not be of concern as it is readonly anyhow.
> -- Using a ram disk also required unnecessarily copying memory from
> the fake block device into the page cache (and copying changes back
> out), as well as creating and destroying dentries.
>-> (?) This one I am actually not too sure about and supersedes my
> knowledge on tmpfs, vfs (and its cache layers), erofs caching, and
> loopback devices).
> -- Plus it needed a filesystem driver (such as ext2) to format and
> interpret this data.
>-> erofs is already included in most initrds (and is not too big if
> it is not)
>
> Regards, Nils
>

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Nils Kattenbeck]

Hi, while I have been following this thread passively for now I also
wanted to chime in.

> (The main reason why sd-stub doesn't actually support erofs-initrds,
> is that sd-stub also generates initrd cpios on the fly, to pass
> credentials and system extension images to the kernel, and you can't
> really mix erofs and cpio initrds into one)

What prevents one from mixing the two (especially given that the
hypothetical erofs initrd support does not yet exist)?
Or are you talking about mixing this with your memmap+root=/dev/pmem suggestion?

> The try to optimize the initrd a bit by making it an erofs/memmap
> thing and so on. And make sure the initrd only contains stuff you
> always need, so that reading it all into memory is necessary anyway,
> and hence any approach that tries to run even the initrd off a disk
> image won't be necessary becuase you need to read everything anyway.

Having to ensure that the initrd is as small as possible is definitely
no easy task.
Furthermore unless one has total control over the devices, or even if
there are only a few hardware revisions, parts of the initrd might not
be used.
Even if everything is the same there are codes paths which might not
be taken during usual operation. An example would be services similar
to the new systemd-bsod which are only triggered in emergencies.
Having these in the cpio means that they will always be read and
decompressed.
Using sysexts also has the drawback that each and every one of them
has to be decompressed. I might be mistaken but I expect that this
will be the case even if the extension-release in the sysext results
in it being discarded which is obviously another big drawback.

Regardless, even if every single file within the cpio archive (and
potential sysexts) is used, erofs still has a distinct advantage over
cpio!
With cpio everything has to be decompressed and read up front. With
erofs this is not the case.
Only the fs header has to be read at first as files are decompressed on demand.
This means that critical stuff can be started earlier as it does not
have to wait for decompression of stuff only needed later on.
For example an initrd-only (i.e. not pivolint root), graphical system
could start all background services long before the UI starts and
accesses large asset files.

I agree that this splitting up into another micro-initrd just for some
storage stuff etc (which I still have not groked completely) does not
seem to offer any advantages to what we have today. *However*, I
certainly think that standardizing and supporting some kind of erofs
based initrd would gain some advantages.

On the other hand this feels like going back to an old ramdisk again.
This goes beyond my knowledge but based on the kernel docs most
drawbacks of ramdisks would not apply to an approach with erofs. Also
maybe the more flexible loopback devices could be used(?) which might
alleviate some problems.

-- This block device was of fixed size, so the filesystem mounted on
it was of fixed size.
   -> Should not be of concern as it is readonly anyhow.
-- Using a ram disk also required unnecessarily copying memory from
the fake block device into the page cache (and copying changes back
out), as well as creating and destroying dentries.
   -> (?) This one I am actually not too sure about and supersedes my
knowledge on tmpfs, vfs (and its cache layers), erofs caching, and
loopback devices).
-- Plus it needed a filesystem driver (such as ext2) to format and
interpret this data.
   -> erofs is already included in most initrds (and is not too big if
it is not)

Regards, Nils

Re: systemd units disabled when calling systemctl daemon-reload

[Vito Caputo]

On Tue, Dec 12, 2023 at 07:06:22PM +0100, Etienne Cordonnier wrote:
> Hello,
> I am debugging some embedded system running systemd. The behavior I am
> observing is that many systemd targets such as multi-user.target are
> disabled after I run systemctl daemon-reload (as shown by systemctl
> list-units --type target --all). This causes many systemd units to be
> disabled, and forces me to reboot the system.
> 
> Is there a way to debug this systemd target transition? I already
> enabled systemctl
> log-level debug, but I still don't understand why the systemd target is
> changing when I call systemctl daemon-reload on this particular system.
> 

I haven't noticed this behavior myself, and attempted a repro here on
v253 (Arch x86_64) just now.

Others here on systemd-devel are likely to wonder what systemd version
you're using, and if you've attempted reproduction using a newer version
if not the latest.

If the version is old and a newer one fixes it, I'd be inclined to try
bisecting for identifying which changes fixed it.  But that assumes a
consistent ability to repro, and good vs. bad versions to work from.

Regards,
Vito Caputo

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Demi Marie Obenour]

On Tue, Dec 12, 2023 at 06:40:32PM +0100, Lennart Poettering wrote:
> On Mo, 11.12.23 12:48, Eric Curtin (ecur...@redhat.com) wrote:
> 
> > Although the nice thing about a storage-init like approach is there's
> > basically zero copies up front. What storage-init is trying to be, is
> > a tool to just call systemd storage things, without also inheriting
> > all the systemd stack.
> 
> Just to make this clear: using things like systemd-cryptsetup outside
> of the systemd stack is not going to work once you leave trivial
> setups. i.e. the TPM hookup involves multiple services these days, and
> it's not going to get any simpler. i.e. systemd-tpm2-setup,
> systemd-pcrextend, systemd-pcrlock and so on. I am sorry, but doing
> reasonable disk encryption with TPM involved means you either buy into
> the whole systemd offer (i.e. with the service manager) or you have to
> rewrite your own systemd.
> 
> But maybe I am misunderstanding what you are saying here.

I think a key factor here is that the initial suggestion was for
automotive use cases.  One can have a vastly simpler system if one is
willing to deliver hardware-specific images, rather than trying to have
a single image that supports many different hardware models.  Automotive
and other embedded systemd understandably do not want to pay for
complexity that they do not need, and which is present to support
features (such as supporting arbitrary hardware) they will never use.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab


signature.asc
Description: PGP signature

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Stephen Smoogen]

On Tue, 12 Dec 2023 at 12:38, Lennart Poettering 
wrote:

> On Mo, 11.12.23 12:48, Eric Curtin (ecur...@redhat.com) wrote:
>
> > Sort of yes, but preferably using that __initramfs_start /
> > initrd_start buffer as is without copying any bytes anywhere else and
> > without teaching the bootloaders to do things.
> >
> > The "memmap=" approach you suggested sounds like what we are thinking,
> > but do you think we could do this without teaching bootloaders to do
> > new things?
>
> Well, in a standard UEFI world it would suffice to teach the memmap=
> logic to the stub that is glued in front of the kernel. For example,
> make sd-stub find the erofs initrd in the UKI, then trivially
> synthesize a memmap= switch and append it to the kernel command line.
>
> but of course, you don't believe in UEFI or good boot loaders, so you
> kinda dug your own grave here...
>
>
To clarify here.. it is not that we don't believe in UEFI or good boot
loaders, it is more that the various hardware being tasked to these
scenarios does not come with it. This is more of trying to make the best
with the ingredients we have, and realizing what we end up with will not be
as palatable as we wished. We all know that having UEFI or coreboot would
make this so much easier and better, but it would have taken the board
designers to have realized that nearly a decade ago since that is when
initial board designs seem to have been chosen. Even if they realized it at
this moment.. we would still be dealing with this for a while.

At this point it isn't that we are trying to dig this grave any deeper, but
are trying to come up with ways to dig out of it :). Some of the proposed
solutions may not do that, but it is what is being tried.



> (The main reason why sd-stub doesn't actually support erofs-initrds,
> is that sd-stub also generates initrd cpios on the fly, to pass
> credentials and system extension images to the kernel, and you can't
> really mix erofs and cpio initrds into one)
>
> Lennart
>
> --
> Lennart Poettering, Berlin
>
>

-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren

Re: IPv6 Compliance for networkd

[Lennart Poettering]

On Mo, 11.12.23 19:14, Muggeridge, Matt (matt.muggerid...@hpe.com) wrote:

> Hello, networkd developer community,
>
> I am hoping to rally support for making networkd IPv6 compliant and
> I'm will to help, but cannot do it alone. Is there any interest in
> making systemd-networkd IPv6 compliant?

Well, interest is relative. I think for most people IPv6 already works
well enough, they don't really care about compliance programs on this
so much. But as long as the requirements are reasonable they also
wouldn't mind (and prefer) if networkd passes those qualifications.

> There are many organizations (especially US Government) that mandate
> IPv6 compliance (USGv6).  Products that are dependent on networkd
> cannot be bid to these customers.

For the people currently involved with networkd upstream this is not a
top priority. If this is important to you however, that's great, we
are happy to review/merge patches.

> How do I engage with the right people in the developer community?

Send PRs via github.

> Thanks,
> Matt.
>
> PS: Mailing list topics go unanswered and github issues get lost in
> the noise, so I'm hoping there's a more efficient way to
> collaborate.

It's an Open Source project: if something matters a lot to you, then
please file PRs to get the work merged. We generally try to review PRs
sooner or later, but we are swamped with work, so it might take a
while. Just filing issues (while also appreciated) will usually
not magically make somebody work on this for you though. It's kinda
the same with most open source projects btw.

If this is something you'd like to see addressed soon, I'd recommend
maybe paying some consultancy (we have worked with codethink on some
projects, they should be willing to work on this, are capable and now
hot get stuff in systemd done).

If you don't have the cash for that, it might work to get funding from
this from organizations such as the German STF and things like
that. I am pretty sure that the US has something similar?

Anyway, judging by your email address I understand you work for HPE,
so I'd assume your company actually has the funds to payroll this
though, if this matters to you.

Lennart

--
Lennart Poettering, Berlin

Re: systemd units disabled when calling systemctl daemon-reload

[Etienne Cordonnier]

More specifically, basic.target, local-fs.target and multi-user.target are
disabled after I run systemctl daemon-reload.

On Tue, Dec 12, 2023 at 7:06 PM Etienne Cordonnier 
wrote:

> Hello,
> I am debugging some embedded system running systemd. The behavior I am
> observing is that many systemd targets such as multi-user.target are
> disabled after I run systemctl daemon-reload (as shown by systemctl
> list-units --type target --all). This causes many systemd units to be
> disabled, and forces me to reboot the system.
>
> Is there a way to debug this systemd target transition? I already enabled 
> systemctl
> log-level debug, but I still don't understand why the systemd target is
> changing when I call systemctl daemon-reload on this particular system.
>
> Thanks,
> Etienne
>

systemd units disabled when calling systemctl daemon-reload

[Etienne Cordonnier]

Hello,
I am debugging some embedded system running systemd. The behavior I am
observing is that many systemd targets such as multi-user.target are
disabled after I run systemctl daemon-reload (as shown by systemctl
list-units --type target --all). This causes many systemd units to be
disabled, and forces me to reboot the system.

Is there a way to debug this systemd target transition? I already
enabled systemctl
log-level debug, but I still don't understand why the systemd target is
changing when I call systemctl daemon-reload on this particular system.

Thanks,
Etienne

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Lennart Poettering]

On Mo, 11.12.23 17:03, Eric Curtin (ecur...@redhat.com) wrote:

> A generic approach is hard, I think it's worth discussing which type of boots
> you should actually care about milliseconds of performance for. It would be 
> nice
> if we had an init system that contained the binary data to do the minimum for
> standard Fedora, Debian installs and everything else was an extension whether
> that's sysexts, dlopen, a new binary to execute etc.
>
> If the network is ingrained in your boot stack like this, I'm
> guessing you probably don't care about boot performance.

Uh, I am not sure that's really true. People boot up VMs on demand,
based on network traffic. They sure care about latency and boot
times. I mean people care about firecracker and these things precisely
because it brings the of off-to-IP to a minimum.

> Automotive has an expectation for really fast boots, like 2 seconds, in 
> standard
> desktops installs there's some expectation as you interface directly
> with a human,
> but for other installs how much expectation is there?

AFAIR in particular in cars there's quite som functionality you
probaly want to move very early in boot. Which yells to me that you
want a service manager super early. Which again suggests to me that
the first initrd that runs should probably already cover that.

If I were you I'd probably focus on a design like this: ship a basic
systemd in an initrd. Complete enough to find the harddisk, and to run
the other services that are absolutely necessary this early. Then,
once you found the disk, look for sysext images on it, and apply them
all on top of the initrd's root fs you are already running with. Never
transition anywhere else.

The try to optimize the initrd a bit by making it an erofs/memmap
thing and so on. And make sure the initrd only contains stuff you
always need, so that reading it all into memory is necessary anyway,
and hence any approach that tries to run even the initrd off a disk
image won't be necessary becuase you need to read everything anyway.

Lennart

--
Lennart Poettering, Berlin

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Lennart Poettering]

On Mo, 11.12.23 11:28, Demi Marie Obenour (d...@invisiblethingslab.com) wrote:

> I don't think this is "a pretty specific solution to one set of devices"
> _at all_.  To the contrary, it is _exactly_ what I want to see desktop
> systems moving to in the future.
>
> It solves the problem of large firmware images.  It solves the problem
> of device-specific configuration, because one can use a file on the EFI
> system partition that is read by userspace and either treated as
> untrusted or TPM-signed.  It means that one have a complete set of
> recovery tools in the event of a problem, rather than being limited to
> whatever one can squeese into an initramfs.  One can even include a full
> GUI stack (with accessibility support!), rather than just Plymouth.  For
> Qubes OS, one can include enough of the Xen and Qubes toolstack to even
> launch virtual machines, allowing the use of USB devices and networking
> for recovery purposes.  It even means that one can use a FIDO2 token to
> unlock the hard drive without a USB stack on the host.  And because the
> initramfs _only_ needs to load the boot extension volume, it can be
> very, _very_ small, which works great with using Linux as a coreboot
> payload.

systemd's "system extension" concept ("sysexts") already allow you to
do all that. The stuff I was fantasizing about would only change one
thing: instead of sd-stub from uefi mode already putting the sysexts
you installed into memory for the initrd to consume, it would be some
proto-initrd that would do so. This does not really change what you
can do with this, but mostly is just an optimization, reducing iops
and memory use a bit, and thus boot time latency.

> The only problem I can see that this does not solve is network boot, but
> that is very much a niche use case when compared to the millions of
> Fedora or Debian desktop installs, or even the tens of thousands of
> Qubes OS installs.  Furthermore, I would _much_ rather network boot be
> handled by userspace and kexec, rather than the closed source UEFI network
> stack.

Well, somebody's niche is somebody else's common case. In VM/cloud/server
scenarios network booting is not that "niche" as it might be on the desktop.

> It does require some care when upgrading, as the dm-verity image and the
> UKI cannot both be updated atomically, but one can solve that by first
> writing the new dm-verity image to a separate location.  The UKI will
> try both both the old and new locations for the dm-verity image and
> rename the new image over the old one on success.  The wrong image will
> simply fail to mount as its root hash will be wrong.

systemd-sysext already covers this just fine: you can encode in their
"extension-release" file to which base images they match up, and
systemd-syext will then find the right one to apply, and ignore the
others. Thus just make sure you drop in the sysexts fist, and the UKI
last and things should be perfectly robust.

Lennart

--
Lennart Poettering, Berlin

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Lennart Poettering]

On Mo, 11.12.23 12:48, Eric Curtin (ecur...@redhat.com) wrote:

> Although the nice thing about a storage-init like approach is there's
> basically zero copies up front. What storage-init is trying to be, is
> a tool to just call systemd storage things, without also inheriting
> all the systemd stack.

Just to make this clear: using things like systemd-cryptsetup outside
of the systemd stack is not going to work once you leave trivial
setups. i.e. the TPM hookup involves multiple services these days, and
it's not going to get any simpler. i.e. systemd-tpm2-setup,
systemd-pcrextend, systemd-pcrlock and so on. I am sorry, but doing
reasonable disk encryption with TPM involved means you either buy into
the whole systemd offer (i.e. with the service manager) or you have to
rewrite your own systemd.

But maybe I am misunderstanding what you are saying here.

Lennart

--
Lennart Poettering, Berlin

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Lennart Poettering]

On Mo, 11.12.23 12:48, Eric Curtin (ecur...@redhat.com) wrote:

> Sort of yes, but preferably using that __initramfs_start /
> initrd_start buffer as is without copying any bytes anywhere else and
> without teaching the bootloaders to do things.
>
> The "memmap=" approach you suggested sounds like what we are thinking,
> but do you think we could do this without teaching bootloaders to do
> new things?

Well, in a standard UEFI world it would suffice to teach the memmap=
logic to the stub that is glued in front of the kernel. For example,
make sd-stub find the erofs initrd in the UKI, then trivially
synthesize a memmap= switch and append it to the kernel command line.

but of course, you don't believe in UEFI or good boot loaders, so you
kinda dug your own grave here...

(The main reason why sd-stub doesn't actually support erofs-initrds,
is that sd-stub also generates initrd cpios on the fly, to pass
credentials and system extension images to the kernel, and you can't
really mix erofs and cpio initrds into one)

Lennart

--
Lennart Poettering, Berlin

Re: IPv6 Compliance for networkd

[Stephen Hemminger]

On Tue, 12 Dec 2023 03:51:37 +
"Muggeridge, Matt"  wrote:

> > 
> > If you need these problems fixed so that you can use systemd-networkd in
> > your commercial products, I recommend getting your company to pay
> > developers to fix systemd-networkd.
> > --
> > Sincerely,
> > Demi Marie Obenour (she/her/hers)
> > Invisible Things Lab  
> 
> I appreciate that point, too. It seems prudent to request collaboration from 
> others in the community, in case there is overlapping interest.
> 
> Thanks,
> Matt.

Also, most of the IPv6 compliance tests require expensive special
purpose hardware and software. This is not something the average open source
contributor will have available.

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Christopher Wong]

Hi Mantas,

After user@1001.service failed, it trigger the 
stopping process and become inactive.


○ user-runtime-dir@1001.service - User 
Runtime Directory /run/user/1001

 Loaded: loaded 
(/etc/systemd/system/user-runtime-dir@.service;
 static)

Drop-In: /usr/lib/systemd/system/service.d

 └─10-axis.conf, 20-axis-sandbox.conf

 Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago

   Duration: 315ms

   Docs: man:user@.service(5)

Process: 16325 ExecStartPre=ls -la /run/user (code=exited, status=0/SUCCESS)

Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS)

Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start 
1001 (code=exited, status=0/SUCCESS)

Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS)

Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited, 
status=0/SUCCESS)

Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS)

Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir stop 1001 
(code=exited, status=0/SUCCESS)

   Main PID: 16329 (code=exited, status=0/SUCCESS)

CPU: 48ms

/etc/fstab don’t include anything on /run/user/1001 and there is no mount unit 
for run-user-1001.mount either.

Best regards,
Christopher Wong


From: Mantas Mikulėnas 
Date: Tuesday, 12 December 2023 at 17:05
To: Christopher Wong 
Cc: Systemd 
Subject: Re: [systemd-devel] Manual start of user@.service failed with 
permission denied
That sounds like it's getting immediately unmounted (or maybe not being mounted 
at all despite the program doing so).

Does the user-runtime-dir service continue to show as "active" after this, or 
does it return to "inactive"?

Does your /etc/fstab have any mentions of /run/user/1001? Or more generally, 
are there any run-user-1001.mount units? (If you 'systemctl status' this unit, 
does the status include a source path?)

On Tue, Dec 12, 2023, 17:34 Christopher Wong 
mailto:christopher.w...@axis.com>> wrote:
Hi Mantas,

I currently have the following flow:


  1.  No /run/user/1001 directory
  2.  systemctl start user@1001.service
  3.  systemd start 
user-runtime-dir@1001.service which ends 
successfully.
  4.  The directory /run/user/1001 exists now, but is empty, owned by root with 
mode 0700
  5.  I don’t have findmnt on my system, so I used mount, but /run/user/1001 is 
not listed.
  6.  systemd start user@1001.service which fails due 
to permission denied.

I can’t explain why the /run/user/1001 is owned by root after 
user-runtime-dir@1001.service 
successfully exited. I added some personal print in systemd code to ensure that 
the mount command returned success (r=0). Although, the mount was successful 
the command “mount” didn’t list it. In the list of mounts starting with /run I 
could only find these entries:


Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs 
(rw,nosuid,nodev,mode=755)

Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs 
(ro,nosuid,nodev,noexec,mode=755)

Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type tmpfs 
(ro,nosuid,nodev,mode=755)

If I do a chown of the directory in user@1001.service 
then it works

root@host:/run/user# ls -la 1001
drwx--3 ida  root80 Dec 12 16:19 .
drwxr-xr-x3 root root60 Dec 12 16:19 ..
srw-rw-rw-1 ida  ssh-user 0 Dec 12 16:19 bus
drwxr-xr-x5 ida  ssh-user   140 Dec 12 16:19 systemd

The ”mount” command don’t list /run/user/1001 for the successful case either.

Best regards,
Christopher Wong


From: Mantas Mikulėnas mailto:graw...@gmail.com>>
Date: Monday, 11 December 2023 at 17:56
To: Christopher Wong 
mailto:christopher.w...@axis.com>>
Cc: Systemd 
mailto:systemd-devel@lists.freedesktop.org>>
Subject: Re: [systemd-devel] Manual start of user@.service failed with 
permission denied
On Mon, Dec 11, 2023, 17:28 Christopher Wong 
mailto:christopher.w...@axis.com>> wrote:
Hi Mantas,

I have added ExecStartPre to user@.service to run “id” 
and “ls -la”:

Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount /run/user/1001 
owned by 1001:118
Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) on 
/run/user/1001 (MS_NOSUID|MS_NODEV 
"mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory /run/user/1001.
Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) 
groups=118(ssh-users),236(systemd-journal)
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x3 root root60 Dec 

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Mantas Mikulėnas]

That sounds like it's getting immediately unmounted (or maybe not being
mounted at all despite the program doing so).

Does the user-runtime-dir service continue to show as "active" after this,
or does it return to "inactive"?

Does your /etc/fstab have any mentions of /run/user/1001? Or more
generally, are there any run-user-1001.mount units? (If you 'systemctl
status' this unit, does the status include a source path?)

On Tue, Dec 12, 2023, 17:34 Christopher Wong 
wrote:

> Hi Mantas,
>
>
>
> I currently have the following flow:
>
>
>
>1. No /run/user/1001 directory
>2. systemctl start user@1001.service
>3. systemd start user-runtime-dir@1001.service which ends successfully.
>4. The directory /run/user/1001 exists now, but is empty, owned by
>root with mode 0700
>5. I don’t have findmnt on my system, so I used mount, but
>/run/user/1001 is not listed.
>6. systemd start user@1001.service which fails due to permission
>denied.
>
>
>
> I can’t explain why the /run/user/1001 is owned by root after
> user-runtime-dir@1001.service successfully exited. I added some personal
> print in systemd code to ensure that the mount command returned success
> (r=0). Although, the mount was successful the command “mount” didn’t list
> it. In the list of mounts starting with /run I could only find these
> entries:
>
>
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs
> (rw,nosuid,nodev,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs
> (ro,nosuid,nodev,noexec,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type
> tmpfs (ro,nosuid,nodev,mode=755)
>
>
>
> If I do a chown of the directory in user@1001.service then it works
>
>
>
> root@host:/run/user# ls -la 1001
>
> drwx--3 ida  root80 Dec 12 16:19 .
>
> drwxr-xr-x3 root root60 Dec 12 16:19 ..
>
> srw-rw-rw-1 ida  ssh-user 0 Dec 12 16:19 bus
>
> drwxr-xr-x5 ida  ssh-user   140 Dec 12 16:19 systemd
>
>
>
> The ”mount” command don’t list /run/user/1001 for the successful case
> either.
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas 
> *Date: *Monday, 11 December 2023 at 17:56
> *To: *Christopher Wong 
> *Cc: *Systemd 
> *Subject: *Re: [systemd-devel] Manual start of user@.service failed
> with permission denied
>
> On Mon, Dec 11, 2023, 17:28 Christopher Wong 
> wrote:
>
> Hi Mantas,
>
>
>
> I have added ExecStartPre to user@.service to run “id” and “ls -la”:
>
>
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount
> /run/user/1001 owned by 1001:118
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs
> (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV
> "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
>
> Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory
> /run/user/1001.
>
> Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
>
> Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users)
> groups=118(ssh-users),236(systemd-journal)
>
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x3 root root
> 60 Dec 11 15:50 .
>
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x   98 root root
> 2120 Dec 11 15:30 ..
>
> Dec 11 15:50:34 host ls[40293]: drwx--2 root root
> 40 Dec 11 15:50 1001
>
> Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in
> user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK
> +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2
> -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT
> -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP
> -SYSVINIT default-hierarchy=unified)
>
>
>
> The /run/user/1001 belongs to root with mode 0700. Should this belong to
> root?
>
> No, it should be owned by UID 1001 (though still mode 0700).
>
> Is it because I manually start user@1001.service as root?
>
> Which user started the .service is usually not important, all services get
> a "fresh" environment that's fully described by the unit file.
>
>
>
> So even if you did 'systemctl start' as root, the unit has User=%i so the
> instance parameter tells it which UID to run as, so will be running as UID
> 1001. Likewise user-runtime-dir@1001 will get the UID for the mount from
> its instance name (you can see that the "Mounting tmpfs" message has the
> correct information).
>
> However, after user-runtime-dir@1001.service has finished it startup,
> the user@1001.service is started as uid=1001 and therefore can’t create
> any directories under /run/user/1001. Resulting in user@1001.service
> failed to start.
>
>
>
> If I add “ExecStartPre=+chown %i /run/user/%i” to user@.service then it
> works! But I am unsure if this is really the way fix this.
>
>
>
> So far, it sounds like the directory is being created *by something else*
> before user-runtime-dir@ is even 

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Christopher Wong]

Hi Andrei,

The systemd logs tells me that /run/user/1001 is mounted as uid=1001, but when 
I list the path /run/user/1001 it is empty and is owned by root. I can’t find 
the path when I run the “mount” command. However, even for the successful case 
the path is not listed with the “mount” command.

Best regards,
Christopher Wong


From: Andrei Borzenkov 
Date: Monday, 11 December 2023 at 19:34
To: Christopher Wong , Mantas Mikulėnas 

Cc: Systemd 
Subject: Re: [systemd-devel] Manual start of user@.service failed with 
permission denied
On 11.12.2023 18:28, Christopher Wong wrote:
> Hi Mantas,
>
> I have added ExecStartPre to user@.service to run “id” 
> and “ls -la”:
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount 
> /run/user/1001 owned by 1001:118
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) 
> on /run/user/1001 (MS_NOSUID|MS_NODEV 
> "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
> Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory 
> /run/user/1001.
> Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
> Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) 
> groups=118(ssh-users),236(systemd-journal)
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x3 root root60 
> Dec 11 15:50 .
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x   98 root root  2120 
> Dec 11 15:30 ..
> Dec 11 15:50:34 host ls[40293]: drwx--2 root root40 
> Dec 11 15:50 1001
> Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in user 
> mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP 
> +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC 
> +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 
> +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT 
> default-hierarchy=unified)
>
> The /run/user/1001 belongs to root with mode 0700. Should this belong to root?

No.

> Is it because I manually start user@1001.service as 
> root?

No.

> However, after 
> user-runtime-dir@1001.service has 
> finished it startup,  the user@1001.service is 
> started as uid=1001 and therefore can’t create any directories under 
> /run/user/1001. Resulting in user@1001.service 
> failed to start.
>
> If I add “ExecStartPre=+chown %i /run/user/%i” to 
> user@.service then it works! But I am unsure if this is 
> really the way fix this.

As clearly seen from logs, systemd-user-runtime-dir mounts tmpfs with
option uid=1001 over /run/user/1001. Is it still a mounted filesystem
when you check it? It sounds like you see mount point which indeed has
permissions 700 and owner root, not mounted filesystem.

>
> Regarding the testing, I have done both restart of everything and manual, but 
> the result is the same. Now that I have the 
> “Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do “systemctl 
> set-environment …”
>
> Thank you for taking your time!
>
> Best regards,
> Christopher Wong
>
>
> From: Mantas Mikulėnas 
> Date: Friday, 8 December 2023 at 21:53
> To: Christopher Wong 
> Cc: Systemd 
> Subject: Re: [systemd-devel] Manual start of user@.service failed with 
> permission denied
> On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong 
> mailto:christopher.w...@axis.com>> wrote:
> Hi Mantas,
>
> I have from your suggestion done the following:
>
> Putting the below in user@.service
>
> [Service]
> ...
> Environment=XDG_RUNTIME_DIR=/run/user/%i
> Environment=SYSTEMD_LOG_LEVEL=debug
>
> Putting the below in user-runtime-dir@.service
>
> [Service]
> ...
> Environment=SYSTEMD_LOG_LEVEL=debug
>
> Then I have disabled the global set-log-level debug (if this is also 
> required, please let me know).
>
> Unlike set-environment that's not global, it only affects pid1.
>
>
> What I can see from the logs is that 
> user-runtime-dir@1001.service seems to 
> be started and mount /run/user/1001, but addition creation of directory under 
> this mount is getting permission denied.
>
> Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001.
> Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory 
> /run/user/1001...
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
> UNSET -> OPENING
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus by 
> connecting to /run/dbus/system_bus_socket...
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
> OPENING -> AUTHENTICATING
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
> AUTHENTICATING -> HELLO
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message 
> type=method_call sender=n/a 

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Christopher Wong]

Hi Belanger,

Thank you for your suggestion! I tried putting RuntimeDirectory= into the 
user@.service. I suppose I shouldn’t need to do this 
since the runtime directory should be fixed by the service 
user-runtime-dir@.service. Result is that it 
didn’t work.

Best regards,
Christopher Wong


From: Belanger, Martin 
Date: Monday, 11 December 2023 at 17:43
To: Christopher Wong , Mantas Mikulėnas 

Cc: Systemd 
Subject: RE: [systemd-devel] Manual start of user@.service failed with 
permission denied
Have you tried using "RuntimeDirectory=" (and optionally 
"RuntimeDirectoryPreserve=")

Ref: 
https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#RuntimeDirectory=

Regards,
Martin

From: systemd-devel  On Behalf Of 
Christopher Wong
Sent: Monday, December 11, 2023 10:28 AM
To: Mantas Mikulėnas 
Cc: Systemd 
Subject: Re: [systemd-devel] Manual start of user@.service failed with 
permission denied

[EXTERNAL EMAIL]
Hi Mantas,

I have added ExecStartPre to mailto:user@.service to run “id” and “ls -la”:

Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount /run/user/1001 
owned by 1001:118
Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) on 
/run/user/1001 (MS_NOSUID|MS_NODEV 
"mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory /run/user/1001.
Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) 
groups=118(ssh-users),236(systemd-journal)
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x3 root root60 Dec 
11 15:50 .
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x   98 root root  2120 Dec 
11 15:30 ..
Dec 11 15:50:34 host ls[40293]: drwx--2 root root40 Dec 
11 15:50 1001
Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in user 
mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP 
+GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC 
+KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 
-LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT 
default-hierarchy=unified)

The /run/user/1001 belongs to root with mode 0700. Should this belong to root? 
Is it because I manually start mailto:user@1001.service as root?
However, after mailto:user-runtime-dir@1001.service has finished it startup,  
the mailto:user@1001.service is started as uid=1001 and therefore can’t create 
any directories under /run/user/1001. Resulting in mailto:user@1001.service 
failed to start.

If I add “ExecStartPre=+chown %i /run/user/%i” to mailto:user@.service then it 
works! But I am unsure if this is really the way fix this.

Regarding the testing, I have done both restart of everything and manual, but 
the result is the same. Now that I have the 
“Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do “systemctl 
set-environment …”

Thank you for taking your time!

Best regards,
Christopher Wong


From: Mantas Mikulėnas 
Date: Friday, 8 December 2023 at 21:53
To: Christopher Wong 
Cc: Systemd 
Subject: Re: [systemd-devel] Manual start of mailto:user@.service failed 
with permission denied
On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong 
 wrote:
Hi Mantas,

I have from your suggestion done the following:

Putting the below in mailto:user@.service

[Service]
...
Environment=XDG_RUNTIME_DIR=/run/user/%i
Environment=SYSTEMD_LOG_LEVEL=debug

Putting the below in mailto:user-runtime-dir@.service

[Service]
...
Environment=SYSTEMD_LOG_LEVEL=debug

Then I have disabled the global set-log-level debug (if this is also required, 
please let me know).

Unlike set-environment that's not global, it only affects pid1.


What I can see from the logs is that mailto:user-runtime-dir@1001.service seems 
to be started and mount /run/user/1001, but addition creation of directory 
under this mount is getting permission denied.

Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001.
Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory 
/run/user/1001...
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
UNSET -> OPENING
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus by 
connecting to /run/dbus/system_bus_socket...
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
OPENING -> AUTHENTICATING
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state 
AUTHENTICATING -> HELLO
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message 
type=method_call sender=n/a destination=org.freedesktop.DBus 
path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello 

Re: [systemd-devel] Manual start of user@.service failed with permission denied

[Christopher Wong]

Hi Mantas,

I currently have the following flow:


  1.  No /run/user/1001 directory
  2.  systemctl start user@1001.service
  3.  systemd start 
user-runtime-dir@1001.service which ends 
successfully.
  4.  The directory /run/user/1001 exists now, but is empty, owned by root with 
mode 0700
  5.  I don’t have findmnt on my system, so I used mount, but /run/user/1001 is 
not listed.
  6.  systemd start user@1001.service which fails due 
to permission denied.

I can’t explain why the /run/user/1001 is owned by root after 
user-runtime-dir@1001.service 
successfully exited. I added some personal print in systemd code to ensure that 
the mount command returned success (r=0). Although, the mount was successful 
the command “mount” didn’t list it. In the list of mounts starting with /run I 
could only find these entries:


Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs 
(rw,nosuid,nodev,mode=755)

Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs 
(ro,nosuid,nodev,noexec,mode=755)

Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type tmpfs 
(ro,nosuid,nodev,mode=755)

If I do a chown of the directory in user@1001.service 
then it works

root@host:/run/user# ls -la 1001
drwx--3 ida  root80 Dec 12 16:19 .
drwxr-xr-x3 root root60 Dec 12 16:19 ..
srw-rw-rw-1 ida  ssh-user 0 Dec 12 16:19 bus
drwxr-xr-x5 ida  ssh-user   140 Dec 12 16:19 systemd

The ”mount” command don’t list /run/user/1001 for the successful case either.

Best regards,
Christopher Wong


From: Mantas Mikulėnas 
Date: Monday, 11 December 2023 at 17:56
To: Christopher Wong 
Cc: Systemd 
Subject: Re: [systemd-devel] Manual start of user@.service failed with 
permission denied
On Mon, Dec 11, 2023, 17:28 Christopher Wong 
mailto:christopher.w...@axis.com>> wrote:
Hi Mantas,

I have added ExecStartPre to user@.service to run “id” 
and “ls -la”:

Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount /run/user/1001 
owned by 1001:118
Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) on 
/run/user/1001 (MS_NOSUID|MS_NODEV 
"mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory /run/user/1001.
Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) 
groups=118(ssh-users),236(systemd-journal)
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x3 root root60 Dec 
11 15:50 .
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x   98 root root  2120 Dec 
11 15:30 ..
Dec 11 15:50:34 host ls[40293]: drwx--2 root root40 Dec 
11 15:50 1001
Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in user 
mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP 
+GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC 
+KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 
-LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT 
default-hierarchy=unified)

The /run/user/1001 belongs to root with mode 0700. Should this belong to root?
No, it should be owned by UID 1001 (though still mode 0700).
Is it because I manually start user@1001.service as 
root?
Which user started the .service is usually not important, all services get a 
"fresh" environment that's fully described by the unit file.

So even if you did 'systemctl start' as root, the unit has User=%i so the 
instance parameter tells it which UID to run as, so will be running as UID 
1001. Likewise user-runtime-dir@1001 will get the UID for the mount from its 
instance name (you can see that the "Mounting tmpfs" message has the correct 
information).
However, after 
user-runtime-dir@1001.service has 
finished it startup,  the user@1001.service is 
started as uid=1001 and therefore can’t create any directories under 
/run/user/1001. Resulting in user@1001.service failed 
to start.

If I add “ExecStartPre=+chown %i /run/user/%i” to 
user@.service then it works! But I am unsure if this is 
really the way fix this.

So far, it sounds like the directory is being created *by something else* 
before user-runtime-dir@ is even invoked.

Try adding the same "-/bin/ls -lad /run/user/%i" as both ExecStartPre and 
ExecStartPost of user-runtime-dir@ (and maybe even a findmnt). If the directory 
already exists during ExecStartPre, start looking for other services or 
cronjobs, or tmpfiles.d configs, or 'su' invocations, which may cause it to be 
created.

There might also be something that 

Re: [RFC] initoverlayfs - a scalable initial filesystem

[Paul Menzel]

[Sorry for the spam to the people in Cc. Now the real address.]

Dear Luca,


Am 11.12.23 um 22:45 schrieb Luca Boccassi:
> On Mon, 11 Dec 2023 at 21:20, Demi Marie Obenour wrote:
>>
>> On Mon, Dec 11, 2023 at 08:58:58PM +, Luca Boccassi wrote:
>>> On Mon, 11 Dec 2023 at 20:43, Demi Marie Obenour wrote:

 On Mon, Dec 11, 2023 at 08:15:27PM +, Luca Boccassi wrote:
> On Mon, 11 Dec 2023 at 17:30, Demi Marie Obenour

>> On Mon, Dec 11, 2023 at 10:57:58AM +0100, Lennart Poettering wrote:
>>> On Fr, 08.12.23 17:59, Eric Curtin (ecur...@redhat.com) wrote:

[…]

> And for ancient, legacy platforms that do not support modern APIs, the
> old ways will still be there, and can be used. Nobody is going to take
> away grub and dracut from the internet, if you got some special corner
> case where you want to use it it will still be there, but the fact
> that such corner cases exist cannot stop the rest of the ecosystem
> that is targeted to modern hardware from evolving into something
> better, more maintainable and more straightforward.

 The problem is not that UEFI is not usable in automotive systems.  The
 problem is that U-Boot (or any other UEFI implementation) is an extra
 stage in the boot process, slows things down, and has more attack
 surface.
>>>
>>> Whatever firmware you use will have an attack surface, the interface
>>> it provides - whether legacy bios or uefi-based - is irrelevant for
>>> that. Skipping or reimplementing all the verity, tpm, etc logic also
>>> increases the attack surface, as does adding initrd-only code that is
>>> never tested and exercised outside of that limited context. If you are
>>> running with legacy bios on ancient hardware you also will likely lack
>>> tpm, secure boot, and so on, so it's all moot, any security argument
>>> goes out of the window. If anybody cares about platform security, then
>>> a tpm-capable and secureboot-capable firmware with a modern, usable
>>> interface like uefi, running the same code in initrd and full system,
>>> using dm-verity everywhere, is pretty much the best one can do.
>>
>> Neither Chrome OS devices nor Macs with Apple silicon use UEFI, and both
>> have better platform security than any UEFI-based device on the market I
>> am aware of.
> 
> We are talking about Linux distributions here. If one wants to use
> proprietary systems, sure, there are better things out there, but
> that's off topic.

In what way is ChromeOS more proprietary than the other GNU/Linux 
distributions, that allow to install the Chrome browser?


Kind regards,

Paul