Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
On Tue, Mar 31, 2020 at 1:10 AM Lennart Poettering wrote: > On Fr, 27.03.20 10:17, Preston L. Bannister (pres...@bannister.us) wrote: > > > Had the (possibly) clever notion of using an overlayfs as the root mount, > > with a tmpfs as the upper, and the usual persistent volume as the > > lower. > > Current systemd versions support this natively, just boot with > "systemd.volatile=overlay" on the kernel cmdline. Doubt this is in > centos8 yet, though. > I can confirm that adding "systemd.volatile=yes" on the kernel cmdline with Centos 8 results in failed boot. Some obscure error related to initramfs that I have not tried to typify. Also Centos 8 has systemd 232, and it looks like "systemd.volatile=overlay" handed later in systemd. Note that overlayfs is a weird fs, it has strange, non-posixy > semantics (inode nrs a fucked). It generally doesn't work as well as > people want it to work, and while you might get away with using it for > small, well-defined scenarios it's not suitable for complex, general > purpose systems to run as root fs. > > "systemd.volatile=overlay" is a nice tool for testing and development, > and maybe some very specific setups, but it's not really something i'd > want to deploy in production in big scale. > OK - had not hear that before. Thanks. Did see some odd behavior, but assumed (and still) the fault was mine. > Look what src/volatile-root/volatile-root.c in current systemd > versions is doing. It's relatively straight-forward. You can do > something similar with a shell script. > Looking. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
On Fr, 27.03.20 10:17, Preston L. Bannister (pres...@bannister.us) wrote:
> Looking for a sanity check from the folk how know more of systemd than do
> I. Not looking for someone else to solve my problem, but could use a clue.
>
> Trying to figure out how to get an overlayfs root mounted early in boot.
>
> Building an embedded system that must not store any sensitive data when
> powered down. Found this was referred to on this list more recently as a
> "stateless" system. (And Tobias Hunger seems to have the most to say. :) )
> We are using Centos8, with systemd 239. Starting point is a vanilla Centos8
> server installation to a small (8 or 32GB) flash volume.
>
> Had the (possibly) clever notion of using an overlayfs as the root mount,
> with a tmpfs as the upper, and the usual persistent volume as the
> lower.
Current systemd versions support this natively, just boot with
"systemd.volatile=overlay" on the kernel cmdline. Doubt this is in
centos8 yet, though.
Note that overlayfs is a weird fs, it has strange, non-posixy
semantics (inode nrs a fucked). It generally doesn't work as well as
people want it to work, and while you might get away with using it for
small, well-defined scenarios it's not suitable for complex, general
purpose systems to run as root fs.
"systemd.volatile=overlay" is a nice tool for testing and development,
and maybe some very specific setups, but it's not really something i'd
want to deploy in production in big scale.
> The initial round of configuration and test would just be against a stock
> Centos8 install. Once fully configured and tested, would add a default boot
> menu item to boot with root mounted as an overlayfs, with the fully
> configured root volume as the read-only lower.
>
> Updates would be accomplished by booting from the original boot menu entry.
> (This is slightly complicated by the fact the target systems' computers do
> not have a console - but figure I can script altering the default
> boot.)
I doubt overlayfs is good enough to make this workable. In particular
not in the old version included in rhel8...
> BASE=/run/overlay_root
> OVERLAY=$BASE/merged
> ROOT_MOUNT=$OVERLAY/mnt/root
> mount -t tmpfs root-base $BASE ; mkdir $BASE/{lower,upper,work,merged}
> mount --bind / $BASE/lower
> mount -t overlayfs root-overlay $OVERLAY
> -olowerdir=$BASE/lower,upperdir=$BASE/upper,workdir=$BASE/work
> for d in boot dev proc run sys ; do mount --bind /$d $OVERLAY/$d ; done
> mkdir $ROOT_MOUNT ; mount --bind / $ROOT_MOUNT
systemd/pid1 will mount proc/run/sys/dev/… on its own after "systemctl
switch-root", no need to do that yourself.
>
> This all seems to work. The overlayfs prevents writes to persistent media
> in usual places. Have a path to write to persistent media. The special
> directories (that do not work from mounts to lower on an overlayfs) work as
> expected.
>
> What I do not have as yet is a means to get the overlayfs root mounted
> early in boot.
> 1. Does not look like /etc/fstab can create the overlayfs.
> 2. Tried the "systemd.volatile=yes" kernel command line, w/o luck.
> Substantial commits between 239 and 245 around this.
> 3. See systemd "pivot_root" and "switch_root", but not yet puzzled out
> usage.
Look what src/volatile-root/volatile-root.c in current systemd
versions is doing. It's relatively straight-forward. You can do
something similar with a shell script.
Lennart
--
Lennart Poettering, Berlin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
Hi, You said no data is to be stored when powered down. What do you need the overlay for then? Just mount to a tmpfs as '/' and centos as '/usr', which is something that systemd supports out of the box. You will need at least some files in /etc for this to work, and I just have the initrd untar some file that I store on the /usr partition into / to get everything into place. You can also set up the root filesystem as an overlayfs in the initrd if you want to go down that route. https://gitlab.com/cleanroom/cleanroom/-/blob/master/cleanroom/commands/create_initrd.py has the code I use to create an initrd for my systems (using mkinitcpio from arch linux, but I am sure similar things are possible with dracut). Check lines 111 and following for a systemd unit that extracts a tarball in the initrd before switching over to the real root. I use simple filesystem images (containing an EFI, a squashfs for root and a dm-verity partition for the root) in my setup. The contents of the EFI partition is copied to the system EFI partition for this to work. The other two partitions can be used in several forms: * dd-ing them all onto separate partitions (works out of the box;-) * dd-ing them into separate LVM volumes (lines 132 following in the script linked above) * dropping the image file with a certain name into a folder (lines 152 following and 168 following) I am pretty sure you can hook into the same places as I do for the image file magic to make your overlayfs work... but then Fedora is probably using dracut, so things might be a bit different. But then dracut apparently also uses systemd in the back, so the boot up process should be somewhat similar. Best Regards, Tobias PS: Feel free to drop me a mail if you have questions about this topic that you think are not appropriate for the wider audience on the mailing list. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
To be clear, I am trying to introduce the minimal change possible to Centos
8 (due to outside constraints). Also past initial/lab development, the
embedded system goes on a trailer, and operates in (network) isolation for
months or years. Not so much a CI sort of setup. Did read the 0pointer
"stateless" post, before first post here.
Mucking through the systemd startup. switching root just Before local-fs
seems to cause trouble (allowing for possible other error on my part). To
my read, perhaps I need to hack systemd-remount-fs so as to preserve the
surrounding logic. Insert the overlay *after* the root volume is picked up
from /etc/fstab, and *before* the special /{proc,dev,sys} are mounted.
At least that is my present guess.
On Mon, Mar 30, 2020 at 4:18 AM Simon McVittie wrote:
> On Fri, 27 Mar 2020 at 10:17:37 -0700, Preston L. Bannister wrote:
> > Updates would be accomplished by booting from the original boot menu
> entry.
> > (This is slightly complicated by the fact the target systems' computers
> do not
> > have a console - but figure I can script altering the default boot.)
>
> I think you might be better off with something based on the idea of
> building an immutable "golden image" elsewhere, and deploying it to the
> stateless machines using a filesystem-image-based "A/B" update strategy
> (like casync and Android) or a filesystem-tree-based atomic update
> strategy (like libostree, as used in Fedora Silverblue and Endless).
> Either way, the big ideas are:
>
> - build the "golden image" in CI infrastructure somewhere else
> - have enough space for two complete OSs
> - whichever one you are booted into (let's say it's "A"), overwrite the
> other one (let's say "B") with the new OS
> - reboot into "B", but just once (have a clever bootloader)
> - if "B" fails to boot, recover by going back to "A"
> - you hope that a later upgrade will fix the regression and overwrite
> "B" again, this time with a better version
> - if "B" boots OK, keep using it (tell the bootloader to boot it every
> time from now on); next time you upgrade, swap the roles of "A" and "B"
>
> Most people who do that will be using an overlay-like arrangement to
> store whatever state they want to transfer between A and B, such as /etc
> and /home - but in your case you don't want to store any state at all.
>
> smcv
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
On Fri, 27 Mar 2020 at 10:17:37 -0700, Preston L. Bannister wrote: > Updates would be accomplished by booting from the original boot menu entry. > (This is slightly complicated by the fact the target systems' computers do not > have a console - but figure I can script altering the default boot.) I think you might be better off with something based on the idea of building an immutable "golden image" elsewhere, and deploying it to the stateless machines using a filesystem-image-based "A/B" update strategy (like casync and Android) or a filesystem-tree-based atomic update strategy (like libostree, as used in Fedora Silverblue and Endless). Either way, the big ideas are: - build the "golden image" in CI infrastructure somewhere else - have enough space for two complete OSs - whichever one you are booted into (let's say it's "A"), overwrite the other one (let's say "B") with the new OS - reboot into "B", but just once (have a clever bootloader) - if "B" fails to boot, recover by going back to "A" - you hope that a later upgrade will fix the regression and overwrite "B" again, this time with a better version - if "B" boots OK, keep using it (tell the bootloader to boot it every time from now on); next time you upgrade, swap the roles of "A" and "B" Most people who do that will be using an overlay-like arrangement to store whatever state they want to transfer between A and B, such as /etc and /home - but in your case you don't want to store any state at all. smcv ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Stateless system - Centos8 and overlayfs on root
Am Freitag, den 27.03.2020, 10:17 -0700 schrieb Preston L. Bannister:
> Looking for a sanity check from the folk how know more of systemd than do I.
> Not looking for someone else to solve my problem, but could use a clue.
>
> Trying to figure out how to get an overlayfs root mounted early in boot.
>
> Building an embedded system that must not store any sensitive data when
> powered down. Found this was referred to on this list more recently as a
> "stateless" system. (And Tobias Hunger seems to have the most to say. :) ) We
> are using Centos8, with systemd 239. Starting point is a vanilla Centos8
> server installation to a small (8 or 32GB) flash volume.
>
> Had the (possibly) clever notion of using an overlayfs as the root mount,
> with a tmpfs as the upper, and the usual persistent volume as the lower. The
> initial round of configuration and test would just be against a stock Centos8
> install. Once fully configured and tested, would add a default boot menu item
> to boot with root mounted as an overlayfs, with the fully configured root
> volume as the read-only lower.
>
> Updates would be accomplished by booting from the original boot menu entry.
> (This is slightly complicated by the fact the target systems' computers do
> not have a console - but figure I can script altering the default boot.)
>
> Have what I think is a properly configured overlayfs root. Currently hung up
> on getting that root mounted early enough(?) in boot. Working down a list of
> possibilities. :)
>
> Have a script to repeatably build/rebuild the overlayfs root. Relevant
> commands (w/o supporting logic):
>
> BASE=/run/overlay_root
> OVERLAY=$BASE/merged
> ROOT_MOUNT=$OVERLAY/mnt/root
> mount -t tmpfs root-base $BASE ; mkdir $BASE/{lower,upper,work,merged}
> mount --bind / $BASE/lower
> mount -t overlayfs root-overlay $OVERLAY
> -olowerdir=$BASE/lower,upperdir=$BASE/upper,workdir=$BASE/work
> for d in boot dev proc run sys ; do mount --bind /$d $OVERLAY/$d ; done
> mkdir $ROOT_MOUNT ; mount --bind / $ROOT_MOUNT
>
> This all seems to work. The overlayfs prevents writes to persistent media in
> usual places. Have a path to write to persistent media. The special
> directories (that do not work from mounts to lower on an overlayfs) work as
> expected.
>
> What I do not have as yet is a means to get the overlayfs root mounted early
> in boot.
> 1. Does not look like /etc/fstab can create the overlayfs.
> 2. Tried the "systemd.volatile=yes" kernel command line, w/o luck.
> Substantial commits between 239 and 245 around this.
> 3. See systemd "pivot_root" and "switch_root", but not yet puzzled out usage.
>
> Looking for where/how to swap in the overlayfs root early in boot. Or tell me
> I'm an idiot, and this will not work. :/
Hi Preston,
So, you need a initramfs doing this for you and where you do all the
mounting. Or you use tmpfiles.d snippets and /etc as tmpfs. See
http://0pointer.net/blog/projects/stateless.html for more information.
BR
Silvio
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Stateless system - Centos8 and overlayfs on root
Looking for a sanity check from the folk how know more of systemd than do
I. Not looking for someone else to solve my problem, but could use a clue.
Trying to figure out how to get an overlayfs root mounted early in boot.
Building an embedded system that must not store any sensitive data when
powered down. Found this was referred to on this list more recently as a
"stateless" system. (And Tobias Hunger seems to have the most to say. :) )
We are using Centos8, with systemd 239. Starting point is a vanilla Centos8
server installation to a small (8 or 32GB) flash volume.
Had the (possibly) clever notion of using an overlayfs as the root mount,
with a tmpfs as the upper, and the usual persistent volume as the lower.
The initial round of configuration and test would just be against a stock
Centos8 install. Once fully configured and tested, would add a default boot
menu item to boot with root mounted as an overlayfs, with the fully
configured root volume as the read-only lower.
Updates would be accomplished by booting from the original boot menu entry.
(This is slightly complicated by the fact the target systems' computers do
not have a console - but figure I can script altering the default boot.)
Have what I think is a properly configured overlayfs root. Currently hung
up on getting that root mounted early enough(?) in boot. Working down a
list of possibilities. :)
Have a script to repeatably build/rebuild the overlayfs root. Relevant
commands (w/o supporting logic):
BASE=/run/overlay_root
OVERLAY=$BASE/merged
ROOT_MOUNT=$OVERLAY/mnt/root
mount -t tmpfs root-base $BASE ; mkdir $BASE/{lower,upper,work,merged}
mount --bind / $BASE/lower
mount -t overlayfs root-overlay $OVERLAY
-olowerdir=$BASE/lower,upperdir=$BASE/upper,workdir=$BASE/work
for d in boot dev proc run sys ; do mount --bind /$d $OVERLAY/$d ; done
mkdir $ROOT_MOUNT ; mount --bind / $ROOT_MOUNT
This all seems to work. The overlayfs prevents writes to persistent media
in usual places. Have a path to write to persistent media. The special
directories (that do not work from mounts to lower on an overlayfs) work as
expected.
What I do not have as yet is a means to get the overlayfs root mounted
early in boot.
1. Does not look like /etc/fstab can create the overlayfs.
2. Tried the "systemd.volatile=yes" kernel command line, w/o luck.
Substantial commits between 239 and 245 around this.
3. See systemd "pivot_root" and "switch_root", but not yet puzzled out
usage.
Looking for where/how to swap in the overlayfs root early in boot. Or tell
me I'm an idiot, and this will not work. :/
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
