Re: booting successfully with read-only file system
On Fri, Jul 03, 2020 at 09:56:03AM -0400, Colin Walters wrote: > > > On Fri, Jul 3, 2020, at 9:32 AM, Zbigniew Jędrzejewski-Szmek wrote: > > On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote: > > > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > > > > > It would be great if we could fairly reliably boot with a read-only > > > > root file system, > > > > > > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a > > > tmpfs). > > > > I see that this thread is one massive communication failure on my part :( > > > > I wrote about "booting successfully with a read-only file system", but I > > see that I didn't say "... when the disk cannot be mounted rw because of > > file system errors". I thought it'd be clear from the context, but it's > > clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose, > > It's really unfortunate how much confusion there is on the "read only" > topic... Yeah ;( > I know there's a fair amount of subtley here but I would hope at least > a few more people in the Fedora community take the time to actually > dive in to the ostree model and understand things. > > What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely > fully read only (or phased more usefully), does not support persistence > at all because physical ISOs don't - same as any other "Live" system > from Anaconda to all the others. > > But that's a totally distinct thing from merely having /usr mounted ro > by default, while still supporting persistence for /etc and /var (i.e. the > ostree model). Right, so IIUC, if /var is read-only, ostree will have the same problems. > > I was writing about traditional systems in a read-only-by-accident scenario, > > i.e. about the system behaving gracefully when the disk is > > ***unexpectedly*** > > read-only. > > That is an important detail indeed =) > > To be clear I agree with the effort! I think it's going to get really messy > to take it very far...and that's what I was getting at in arguing for > using overlayfs backed by tmpfs basically. This would be another approach. E.g. we systemd could detect that root cannot be remounted rw and insert the overlay. But I don't think this as useful as it sounds, because it would create a fake impression that the storage is rw. The patch for systemd to deal with ro /var/tmp for PrivateTmp=yes is careful to set things up so that the service actually gets a real EROFS error from the kernel when it writes to /var/tmp. It also gets the real EROFS when it uses StateDirectory=foo and /var/foo is ro. I think such real errors are good — they allow services which can do a graceful fallback to do it, and services which should fail to fail. So systemd-update-utmp.service may just write to utmp and warn about wtmp being inaccessible, but postgresql.service should fail to start and not accept any transactions that would then be lost upon reboot. An overlay would also make it hard to go back to normal operation. Maybe that's not so important since one can always reboot, but I think the current property of being able to fix the fs and remount it rw is quite nice. (Also, less risky then trying to reboot and realizing that there are more errors which have caused the fs to be mounted ro again.) > Or maybe it should be more like a "recovery shell" - rather than trying > to log in as your regular user and watch e.g. Firefox fail because it can't > write to /home and wonder just how much of the next years of your > life is going to involve patching software to make this work ;) > get enough to get the a desktop launched for a separate ephemeral > "live" user with sudo rights or so. I'm not sure if this is such a remote goal. Programs *should* already be prepared for reasonable read-only operation, because this can already happen for a few different reasons: apart from the fs being mounted read-only, the disk may be full or the user may exceed quota. For an application the effect is similar. And those conditions can happen at any point at runtime. Disks may also be remounted ro at runtime if errors are detected. So any applications which falls flat on its face if it cannot write to the disk is problematic already. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Fri, Jul 3, 2020 at 9:56 AM Colin Walters wrote: > > What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely > fully read only (or phased more usefully), does not support persistence > at all because physical ISOs don't - same as any other "Live" system > from Anaconda to all the others. > This is actually not true. All Fedora live ISOs are supposed to support persistence, and can be configured that way on a USB stick using livecd-iso-to-disk from the livecd-iso-to-mediums package. However, this is not yet exposed in Fedora Media Writer. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Fri, Jul 3, 2020, at 9:32 AM, Zbigniew Jędrzejewski-Szmek wrote: > On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote: > > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > > > It would be great if we could fairly reliably boot with a read-only > > > root file system, > > > > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a > > tmpfs). > > I see that this thread is one massive communication failure on my part :( > > I wrote about "booting successfully with a read-only file system", but I > see that I didn't say "... when the disk cannot be mounted rw because of > file system errors". I thought it'd be clear from the context, but it's > clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose, It's really unfortunate how much confusion there is on the "read only" topic... I know there's a fair amount of subtley here but I would hope at least a few more people in the Fedora community take the time to actually dive in to the ostree model and understand things. What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely fully read only (or phased more usefully), does not support persistence at all because physical ISOs don't - same as any other "Live" system from Anaconda to all the others. But that's a totally distinct thing from merely having /usr mounted ro by default, while still supporting persistence for /etc and /var (i.e. the ostree model). > I was writing about traditional systems in a read-only-by-accident scenario, > i.e. about the system behaving gracefully when the disk is ***unexpectedly*** > read-only. That is an important detail indeed =) To be clear I agree with the effort! I think it's going to get really messy to take it very far...and that's what I was getting at in arguing for using overlayfs backed by tmpfs basically. Or maybe it should be more like a "recovery shell" - rather than trying to log in as your regular user and watch e.g. Firefox fail because it can't write to /home and wonder just how much of the next years of your life is going to involve patching software to make this work ;) get enough to get the a desktop launched for a separate ephemeral "live" user with sudo rights or so. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Fri, Jul 03, 2020 at 01:32:12PM +, Zbigniew Jędrzejewski-Szmek wrote: > On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote: > > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > > > It would be great if we could fairly reliably boot with a read-only > > > root file system, > > > > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a > > tmpfs). > > I see that this thread is one massive communication failure on my part :( > > I wrote about "booting successfully with a read-only file system", but I > see that I didn't say "... when the disk cannot be mounted rw because of > file system errors". ...but Colin did not say that. Neither tmpfs nor overlayfs (backed by tmpfs) require any existing disk filesystems to be mounted read-write. TBH, that was my first thought when I read your original e-mail: "why would units fail if they only want to write non-persistent stuff to an area that may perfectly well be mounted as tmpfs, just as /run usually is?" > I thought it'd be clear from the context, but it's > clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose, > I was writing about traditional systems in a read-only-by-accident scenario, > i.e. about the system behaving gracefully when the disk is ***unexpectedly*** > read-only. > > Zbyszek > > PS. OK, I know I wrote about making it read-only on purpose using a > kernel commandline option, so really we're just pretending it was > unexpected for testing purposes, but you get what I mean I hope. G'luck, Peter -- Peter Pentchev r...@ringlet.net r...@debian.org p...@storpool.com PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115 C354 651E EFB0 2527 DF13 signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
Hi, On 03/07/2020 14:18, Colin Walters wrote: On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: It would be great if we could fairly reliably boot with a read-only root file system, Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs). That's what we do for Fedora CoreOS based live images, see https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator It works. (Which we recently switched to involve a loopback-mounted xfs on tmpfs because SELinux, but that is mostly only necessary because we want to support Ignition which does system provisioning in the initramfs, which is not true on non-Ignition based systems) If there is additional support required in overlayfs, then please do file a bug and request it, Steve. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote: > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > It would be great if we could fairly reliably boot with a read-only > > root file system, > > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a > tmpfs). I see that this thread is one massive communication failure on my part :( I wrote about "booting successfully with a read-only file system", but I see that I didn't say "... when the disk cannot be mounted rw because of file system errors". I thought it'd be clear from the context, but it's clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose, I was writing about traditional systems in a read-only-by-accident scenario, i.e. about the system behaving gracefully when the disk is ***unexpectedly*** read-only. Zbyszek PS. OK, I know I wrote about making it read-only on purpose using a kernel commandline option, so really we're just pretending it was unexpected for testing purposes, but you get what I mean I hope. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > It would be great if we could fairly reliably boot with a read-only > root file system, Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs). That's what we do for Fedora CoreOS based live images, see https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/20live/live-generator It works. (Which we recently switched to involve a loopback-mounted xfs on tmpfs because SELinux, but that is mostly only necessary because we want to support Ignition which does system provisioning in the initramfs, which is not true on non-Ignition based systems) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 2, 2020, at 12:05 PM, Daniel P. Berrangé wrote: > I presume you're referring to regular Fedora here, but this description > feels like it is approx asking for what Fedora Silverblue has delivered, > only with the writable area for apps being just a ram disk with no > persistence. No no no! That's *not at all* what it is. I am actually kind of at war with the "immutable" terminology because it's more isleading than it is accurate. https://blog.verbum.org/2019/12/23/starting-from-open-and-foss/ is a bit related to this. /etc and /var are fully persistent! It's just an "image based" update system, and /usr is read-only by default, but you can still layer/override packages, and in the future I'd like to make it easier to use ostree to persist non-rpm changes. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 03.07.2020 13:42, Neal Gompa wrote: > Ubuntu's MOTD requires network connectivity to function, I think ours > would not. That would eliminate the network resource contention issues > you were seeing. The good MOTD is an empty MOTD. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 03.07.2020 08:07, Zbigniew Jędrzejewski-Szmek wrote: > Ubuntu's MOTD are well known and people seem to like them a > lot. Fedora hasn't been making that much use of them. But I think we > should in general. Ubuntu MOTD contains ads. Most of Ubuntu users completely disable it right after the installation. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Fri, Jul 3, 2020 at 6:50 AM Roberto Ragusa wrote: > > On 2020-07-03 08:07, Zbigniew Jędrzejewski-Szmek wrote: > > > Ubuntu's MOTD are well known and people seem to like them a > > lot. Fedora hasn't been making that much use of them. But I think we > > should in general. > > Do not follow Ubuntu on this too much. > I've had a case of a machine with many short-lived incoming ssh connections > where the performance bottleneck was the constant regeneration of > the motd on each login. Sabotaging that stuff brutally was the only way > to get back to good performance. > Ubuntu's MOTD requires network connectivity to function, I think ours would not. That would eliminate the network resource contention issues you were seeing. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 2020-07-03 08:07, Zbigniew Jędrzejewski-Szmek wrote: Ubuntu's MOTD are well known and people seem to like them a lot. Fedora hasn't been making that much use of them. But I think we should in general. Do not follow Ubuntu on this too much. I've had a case of a machine with many short-lived incoming ssh connections where the performance bottleneck was the constant regeneration of the motd on each login. Sabotaging that stuff brutally was the only way to get back to good performance. Regards. -- Roberto Ragusamail at robertoragusa.it ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 02, 2020 at 03:17:58PM -0500, Brandon Nielsen wrote: > On 7/2/20 3:10 PM, Christopher Engelhard wrote: > >On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote: > >>It would be great if we could fairly reliably boot with a read-only > >>root file system, all the way to the graphical environment. Obviously, > >>such a machine will not be fully functional, but for users, debugging a > >>disk problem when they have the normal environment with windows, > >>tabbed terminals, graphical editors, and internet is vastly easier. > >> > >>It also creates an image of robustness. Imagine that instead of being > >>rudely dropped to a terminal prompt, the user is instead able to log in > >>as usual and see a popup like > >>>Your home directory is read-only. Do this and that. See https://... > > > >That would be fantastic, and would be miles ahead from any UX I had on > >any computer, ever. > > > >>I hope we can all cooperate to make read-only boots nicely robust and > >>functional. Please play with this and report bugs. I'll try to solve any > >>that relate to systemd. The systemd version with udev.blockdev-read-only > >>is not released yet, but is available from koji ci builds [11]. > > > >Thanks for working on this, I will definitely give it a try myself. > > > >Christopher > > This sounds excellent! > > Could we somehow provide a list of links that may be helpful to > recover from certain common failures? Maybe in a MOTD? Ubuntu's MOTD are well known and people seem to like them a lot. Fedora hasn't been making that much use of them. But I think we should in general. If we switch to btrfs by default, we'll certainly need to beef up the docs about diagnosis and recovery. If MOTD poped up a link to them if there are any fs-related issues in the initial boot would be pretty nifty. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 7/2/20 3:10 PM, Christopher Engelhard wrote: On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote: It would be great if we could fairly reliably boot with a read-only root file system, all the way to the graphical environment. Obviously, such a machine will not be fully functional, but for users, debugging a disk problem when they have the normal environment with windows, tabbed terminals, graphical editors, and internet is vastly easier. It also creates an image of robustness. Imagine that instead of being rudely dropped to a terminal prompt, the user is instead able to log in as usual and see a popup like Your home directory is read-only. Do this and that. See https://... That would be fantastic, and would be miles ahead from any UX I had on any computer, ever. I hope we can all cooperate to make read-only boots nicely robust and functional. Please play with this and report bugs. I'll try to solve any that relate to systemd. The systemd version with udev.blockdev-read-only is not released yet, but is available from koji ci builds [11]. Thanks for working on this, I will definitely give it a try myself. Christopher This sounds excellent! Could we somehow provide a list of links that may be helpful to recover from certain common failures? Maybe in a MOTD? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote: > It would be great if we could fairly reliably boot with a read-only > root file system, all the way to the graphical environment. Obviously, > such a machine will not be fully functional, but for users, debugging a > disk problem when they have the normal environment with windows, > tabbed terminals, graphical editors, and internet is vastly easier. > > It also creates an image of robustness. Imagine that instead of being > rudely dropped to a terminal prompt, the user is instead able to log in > as usual and see a popup like >> Your home directory is read-only. Do this and that. See https://... That would be fantastic, and would be miles ahead from any UX I had on any computer, ever. > I hope we can all cooperate to make read-only boots nicely robust and > functional. Please play with this and report bugs. I'll try to solve any > that relate to systemd. The systemd version with udev.blockdev-read-only > is not released yet, but is available from koji ci builds [11]. Thanks for working on this, I will definitely give it a try myself. Christopher ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 02, 2020 at 06:27:44PM +0200, Vitaly Zaitsev via devel wrote: > On 02.07.2020 17:53, Zbigniew Jędrzejewski-Szmek wrote: > > It would be great if we could fairly reliably boot with a read-only > > root file system, all the way to the graphical environment. > > Already implemented - Silverblue. If you read past the first sentence, you'd notice that I'm talking about a system in failure conditions. If you looked in at earlier replies to my mail, you'd also notice that your idea was already discussed upthread. But instead of just replying to the subject, you actually got as far as the first sentence, so ... there is hope ;) Zbyszek PS. I wrote quite seriously about a real problem. I don't need quick replies that don't answer anything. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On 02.07.2020 17:53, Zbigniew Jędrzejewski-Szmek wrote: > It would be great if we could fairly reliably boot with a read-only > root file system, all the way to the graphical environment. Already implemented - Silverblue. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 02, 2020 at 05:05:09PM +0100, Daniel P. Berrangé wrote: > On Thu, Jul 02, 2020 at 03:53:26PM +, Zbigniew Jędrzejewski-Szmek wrote: > > Hi, > > > > this is partially an outgrowth of the discussion about btrfs as > > default, but makes sense independently too... > > > > It would be great if we could fairly reliably boot with a read-only > > root file system, all the way to the graphical environment. Obviously, > > such a machine will not be fully functional, but for users, debugging a > > disk problem when they have the normal environment with windows, > > tabbed terminals, graphical editors, and internet is vastly easier. > > > > It also creates an image of robustness. Imagine that instead of being > > rudely dropped to a terminal prompt, the user is instead able to log in > > as usual and see a popup like > > > Your home directory is read-only. Do this and that. See https://... > > > > Is the goal to have *everything* working? No. Some services will and > > should fail. If I have a database or anything else which only makes > > sense with permanent storage, failing early and loudly is appropriate. > > But services which need writable storage only tangentially or not at > > all should be robust and not fail. Journald behaves in a fashion where > > it stores logs to /run during early boot and them flushes them to /var/log > > when that becomes available. If /var/log never become available, we > > have a functional logs, with journalctl showing previous and current boot > > just fine. The only caveat is that logs for current boot will be lost > > upon reboot. Such graceful failure should be the norm. > > I presume you're referring to regular Fedora here, but this description > feels like it is approx asking for what Fedora Silverblue has delivered, > only with the writable area for apps being just a ram disk with no > persistence. No, quite the opposite. I am talking about making the best out of a failure situation on a normal Fedora installation. I don't want to boot routinely in such a mode. I want to boot ro, look for help on the internet and in man pages, do some fsck operations on the read-only disk, and finally remount the disk rw and carry on with my day. Zbyszek > https://docs.fedoraproject.org/en-US/fedora-silverblue/ > > "Silverblue is a variant of Fedora Workstation. It looks, >feels and behaves like a regular desktop operating system, >and the experience is similar to what you find with using >a standard Fedora Workstation. > >However, unlike other operating systems, Silverblue is >immutable. This means that every installation is identical >to every other installation of the same version. The operating >system that is on disk is exactly the same from one machine to >the next, and it never changes as it is used. > >Silverblue’s immutable design is intended to make it more >stable, less prone to bugs, and easier to test and develop. >Finally, Silverblue’s immutable design also makes it an >excellent platform for containerized applications as well >as container-based software development. In each case, >applications (apps) and containers are kept separate from >the host system, improving stability and reliability." > > > Regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o-https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 2, 2020 at 12:05 PM Daniel P. Berrangé wrote: > > On Thu, Jul 02, 2020 at 03:53:26PM +, Zbigniew Jędrzejewski-Szmek wrote: > > Hi, > > > > this is partially an outgrowth of the discussion about btrfs as > > default, but makes sense independently too... > > > > It would be great if we could fairly reliably boot with a read-only > > root file system, all the way to the graphical environment. Obviously, > > such a machine will not be fully functional, but for users, debugging a > > disk problem when they have the normal environment with windows, > > tabbed terminals, graphical editors, and internet is vastly easier. > > > > It also creates an image of robustness. Imagine that instead of being > > rudely dropped to a terminal prompt, the user is instead able to log in > > as usual and see a popup like > > > Your home directory is read-only. Do this and that. See https://... > > > > Is the goal to have *everything* working? No. Some services will and > > should fail. If I have a database or anything else which only makes > > sense with permanent storage, failing early and loudly is appropriate. > > But services which need writable storage only tangentially or not at > > all should be robust and not fail. Journald behaves in a fashion where > > it stores logs to /run during early boot and them flushes them to /var/log > > when that becomes available. If /var/log never become available, we > > have a functional logs, with journalctl showing previous and current boot > > just fine. The only caveat is that logs for current boot will be lost > > upon reboot. Such graceful failure should be the norm. > > I presume you're referring to regular Fedora here, but this description > feels like it is approx asking for what Fedora Silverblue has delivered, > only with the writable area for apps being just a ram disk with no > persistence. > Silverblue fails when the disk can't mount as read-write either. So this benefits Silverblue by making it possible for those failure modes to work properly. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 02, 2020 at 03:53:26PM +, Zbigniew Jędrzejewski-Szmek wrote: > Hi, > > this is partially an outgrowth of the discussion about btrfs as > default, but makes sense independently too... > > It would be great if we could fairly reliably boot with a read-only > root file system, all the way to the graphical environment. Obviously, > such a machine will not be fully functional, but for users, debugging a > disk problem when they have the normal environment with windows, > tabbed terminals, graphical editors, and internet is vastly easier. > > It also creates an image of robustness. Imagine that instead of being > rudely dropped to a terminal prompt, the user is instead able to log in > as usual and see a popup like > > Your home directory is read-only. Do this and that. See https://... > > Is the goal to have *everything* working? No. Some services will and > should fail. If I have a database or anything else which only makes > sense with permanent storage, failing early and loudly is appropriate. > But services which need writable storage only tangentially or not at > all should be robust and not fail. Journald behaves in a fashion where > it stores logs to /run during early boot and them flushes them to /var/log > when that becomes available. If /var/log never become available, we > have a functional logs, with journalctl showing previous and current boot > just fine. The only caveat is that logs for current boot will be lost > upon reboot. Such graceful failure should be the norm. I presume you're referring to regular Fedora here, but this description feels like it is approx asking for what Fedora Silverblue has delivered, only with the writable area for apps being just a ram disk with no persistence. https://docs.fedoraproject.org/en-US/fedora-silverblue/ "Silverblue is a variant of Fedora Workstation. It looks, feels and behaves like a regular desktop operating system, and the experience is similar to what you find with using a standard Fedora Workstation. However, unlike other operating systems, Silverblue is immutable. This means that every installation is identical to every other installation of the same version. The operating system that is on disk is exactly the same from one machine to the next, and it never changes as it is used. Silverblue’s immutable design is intended to make it more stable, less prone to bugs, and easier to test and develop. Finally, Silverblue’s immutable design also makes it an excellent platform for containerized applications as well as container-based software development. In each case, applications (apps) and containers are kept separate from the host system, improving stability and reliability." Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: booting successfully with read-only file system
On Thu, Jul 2, 2020 at 11:54 AM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: > Hi, > > this is partially an outgrowth of the discussion about btrfs as > default, but makes sense independently too... > > It would be great if we could fairly reliably boot with a read-only > root file system, all the way to the graphical environment. Obviously, > such a machine will not be fully functional, but for users, debugging a > disk problem when they have the normal environment with windows, > tabbed terminals, graphical editors, and internet is vastly easier. > > It also creates an image of robustness. Imagine that instead of being > rudely dropped to a terminal prompt, the user is instead able to log in > as usual and see a popup like > > Your home directory is read-only. Do this and that. See https://... > > Is the goal to have *everything* working? No. Some services will and > should fail. If I have a database or anything else which only makes > sense with permanent storage, failing early and loudly is appropriate. > But services which need writable storage only tangentially or not at > all should be robust and not fail. Journald behaves in a fashion where > it stores logs to /run during early boot and them flushes them to /var/log > when that becomes available. If /var/log never become available, we > have a functional logs, with journalctl showing previous and current boot > just fine. The only caveat is that logs for current boot will be lost > upon reboot. Such graceful failure should be the norm. > > systemd upstream recently gained a cool feature [1] which allows all > block devices to be toggled read-only as soon as they are detected by > udev. The primary use case is forensics and recovery, but it is also > great for testing read-only boot ;) > > It turns out that systemd itself is not very good in this situation. > For example, any unit with PrivateTmp=yes would fail to start :( > But it turns out that this is fairly easy to solve. I opened two > PRs today [2, 3], and with that, systemd boots to a working > multi-user.target with no services bundled with systemd failing! > > But I was not able to go all the way to a gnome session :( > I have been opening bugs as I went along: dnf [4], python [5], sssd [6], > gssproxy [7], gdm [8], btrfs [9], xfs[10]. The good new is that the > first is almost solved, the second is already solved, the next two > seem fairly easy, and the btrfs one is being handled. The one for gdm > unfortunately looks tougher :( > > I hope we can all cooperate to make read-only boots nicely robust and > functional. Please play with this and report bugs. I'll try to solve any > that relate to systemd. The systemd version with udev.blockdev-read-only > is not released yet, but is available from koji ci builds [11]. > > [1] https://github.com/systemd/systemd/commit/95ac523030 > [2] https://github.com/systemd/systemd/pull/16340 > [3] https://github.com/systemd/systemd/pull/16344 > [4] https://bugzilla.redhat.com/show_bug.cgi?id=1852365 > [5] https://bugzilla.redhat.com/show_bug.cgi?id=1852941 > [6] https://bugzilla.redhat.com/show_bug.cgi?id=1853261 > [7] https://bugzilla.redhat.com/show_bug.cgi?id=1853293 > [8] https://bugzilla.redhat.com/show_bug.cgi?id=1853409 > [9] https://bugzilla.redhat.com/show_bug.cgi?id=1851608 > [10] https://bugzilla.redhat.com/show_bug.cgi?id=1829792 > [11] https://src.fedoraproject.org/rpms/systemd/pull-request/29 > > Zbyszek > +about a billion or so! Particularly fun if you forget to set the root password after install and can't even get to the prompt. langdon > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
booting successfully with read-only file system
Hi, this is partially an outgrowth of the discussion about btrfs as default, but makes sense independently too... It would be great if we could fairly reliably boot with a read-only root file system, all the way to the graphical environment. Obviously, such a machine will not be fully functional, but for users, debugging a disk problem when they have the normal environment with windows, tabbed terminals, graphical editors, and internet is vastly easier. It also creates an image of robustness. Imagine that instead of being rudely dropped to a terminal prompt, the user is instead able to log in as usual and see a popup like > Your home directory is read-only. Do this and that. See https://... Is the goal to have *everything* working? No. Some services will and should fail. If I have a database or anything else which only makes sense with permanent storage, failing early and loudly is appropriate. But services which need writable storage only tangentially or not at all should be robust and not fail. Journald behaves in a fashion where it stores logs to /run during early boot and them flushes them to /var/log when that becomes available. If /var/log never become available, we have a functional logs, with journalctl showing previous and current boot just fine. The only caveat is that logs for current boot will be lost upon reboot. Such graceful failure should be the norm. systemd upstream recently gained a cool feature [1] which allows all block devices to be toggled read-only as soon as they are detected by udev. The primary use case is forensics and recovery, but it is also great for testing read-only boot ;) It turns out that systemd itself is not very good in this situation. For example, any unit with PrivateTmp=yes would fail to start :( But it turns out that this is fairly easy to solve. I opened two PRs today [2, 3], and with that, systemd boots to a working multi-user.target with no services bundled with systemd failing! But I was not able to go all the way to a gnome session :( I have been opening bugs as I went along: dnf [4], python [5], sssd [6], gssproxy [7], gdm [8], btrfs [9], xfs[10]. The good new is that the first is almost solved, the second is already solved, the next two seem fairly easy, and the btrfs one is being handled. The one for gdm unfortunately looks tougher :( I hope we can all cooperate to make read-only boots nicely robust and functional. Please play with this and report bugs. I'll try to solve any that relate to systemd. The systemd version with udev.blockdev-read-only is not released yet, but is available from koji ci builds [11]. [1] https://github.com/systemd/systemd/commit/95ac523030 [2] https://github.com/systemd/systemd/pull/16340 [3] https://github.com/systemd/systemd/pull/16344 [4] https://bugzilla.redhat.com/show_bug.cgi?id=1852365 [5] https://bugzilla.redhat.com/show_bug.cgi?id=1852941 [6] https://bugzilla.redhat.com/show_bug.cgi?id=1853261 [7] https://bugzilla.redhat.com/show_bug.cgi?id=1853293 [8] https://bugzilla.redhat.com/show_bug.cgi?id=1853409 [9] https://bugzilla.redhat.com/show_bug.cgi?id=1851608 [10] https://bugzilla.redhat.com/show_bug.cgi?id=1829792 [11] https://src.fedoraproject.org/rpms/systemd/pull-request/29 Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
