Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-11 Thread Chris Murphy
On Tue, Oct 11, 2016 at 1:53 AM, Gerd Hoffmann  wrote:

>> 4. An issue with using syslinux's format, as well as the original
>> bootloader spec format, and a major source of disagreement, is the
>> assumption that the kernel and initrd are on the same file system as
>> the bootloader and its configuration file.
>
> Dropping that assumption makes the boot loaders alot more complex
> though.  Suddenly you can't rely on the firmware any more for file
> access, need to understand partition schemes and filesystems, ...

No. The format must provide superset functionality for all
bootloaders. The bootloader can choose how to fail gracefully if a
particular boot entry snippet references files in a manner it doesn't
support.


>> This is necessary to avoid putting the kernel and initrd on the EFI
>> System partition, and causing a lot of installation grief with how to
>> deal with dual boot support.
>
> Hmm, not sure why you want avoid that, as I understand it the idea of
> the efi system partition is that everything needed to boot is there (for
> all operating systems in case of dual/multi-boot).  So why don't use it
> that way?

No. The EFI system partition is intended to be for the firmware:
bootloaders and configuration files. That's why by default no system
installer, not Fedora's, Ubuntu's, openSUSE, Apple or Microsoft, make
them bigger than 260MiB. Many are a good deal smaller. One of the
first things the Microsoft bootloader teaches the firmware is how to
read NTFS, so it can find and boot the Windows kernel. It's no
different than what Fedora and most other distros do.

And as I said earlier in the thread, for dual and multiboot systems,
the existing ESP is only barely big enough to share when restricted to
bootloaders and their files. There simply isn't enough room to put
kernels and initramfs's on them at all. And there's no practical way
to resize or replace them.



>
> It isn't that simple on !EFI systems though.

Very simple technically, not at all simple politically/cooperatively.


>
>> Anyway, regardless of what format you want to base things on, it
>> probably should be a superset of the menu entry functions, including a
>> way to specify volumes by device designation, LVM, or volume UUID, or
>> now your format isn't actually compatible with GRUB on UEFI as well as
>> a bunch of less common scenarios.
>
> Well, the bootloaderspec menu *entries* should not need that, just place
> them next to kernel + initrd and have a pointer to the directory (plus
> optional volume in case you place them outside the efi system partition,
> or on !EFI systems) to scan into the bootloader config file.

It's not workable for all the reasons I've previously mentioned, more
reasoning is in this proposed variant of the spec [1]. Plus it's a
step backwards for anyone wanting to snapshot /boot along with /usr,
so that kernels aren't orphaned from their modules.


[1]
https://www.freedesktop.org/wiki/MatthewGarrett/BootLoaderSpec/


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-11 Thread Gerd Hoffmann
  Hi,

> > ... and have a fixed grub2.cfg which basically has the command to parse
> > the syslinux config file?
> >
> 
> OK so there are a few things:
> 
> 1. The information needed to present boot options, the "menu entries",
> are a separate thing from the bootloader specific configuration file
> and file format. And yet all the bootloaders out there conflate these
> two things, and make the problem of cross compatibility more
> difficult.

Yep.  You see some fallout from that on arm for example, with uboot
printing "don't understand $this, skipping line" style messages when
parsing extlinux.conf.

> 3. The result is bootloader specific configuration files becomes
> static, non-changing and don't need to contain boot entries (although
> it's not disallowed). And individual boot menu drop in scripts are how
> new boot entries are added. The bootloader then merges them together.

And that would make the grub2.cfg mess alot less painful.

> 4. An issue with using syslinux's format, as well as the original
> bootloader spec format, and a major source of disagreement, is the
> assumption that the kernel and initrd are on the same file system as
> the bootloader and its configuration file.

Dropping that assumption makes the boot loaders alot more complex
though.  Suddenly you can't rely on the firmware any more for file
access, need to understand partition schemes and filesystems, ...

> This is necessary to avoid putting the kernel and initrd on the EFI
> System partition, and causing a lot of installation grief with how to
> deal with dual boot support.

Hmm, not sure why you want avoid that, as I understand it the idea of
the efi system partition is that everything needed to boot is there (for
all operating systems in case of dual/multi-boot).  So why don't use it
that way?

It isn't that simple on !EFI systems though.

> Anyway, regardless of what format you want to base things on, it
> probably should be a superset of the menu entry functions, including a
> way to specify volumes by device designation, LVM, or volume UUID, or
> now your format isn't actually compatible with GRUB on UEFI as well as
> a bunch of less common scenarios.

Well, the bootloaderspec menu *entries* should not need that, just place
them next to kernel + initrd and have a pointer to the directory (plus
optional volume in case you place them outside the efi system partition,
or on !EFI systems) to scan into the bootloader config file.

cheers,
  Gerd
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-10 Thread Chris Murphy
On Mon, Oct 10, 2016 at 2:17 AM, Gerd Hoffmann  wrote:
>   Hi,
>
>> > Adam, the only other distro that has serious alternate architecture 
>> > support,
>> > AFAIK, is Debian. How do they handle grub2 + non-x86? Likewise, the
>> > alternate architectures that we support, how do we handle their 
>> > bootloaders?
>> > Are they grub-based? Ext/Syslinux based? Grub-legacy?
>>
>> Nothing in Fedora uses GRUB legacy.
>>
>> ARM is using Das U-Boot. I'm not sure if grubby is involved here or not.
>>
>> Fedora uses isolinux, extlinux, grub2, yaboot (power7 and older), and
>> zipl (s390). If the last two are gone or going away then it's syslinux
>> (and variants), grub2, and uboot.
>
> Hmm, uboot can use extlinux-style config files, and I recently noticed
> grub2 has a command to parse syslinux config files too.  Have not tried
> to use that though.

GRUB's syslinuxcfg command is called from an existing grub.cfg, so
using extlinux.conf files do not obviate the need for grub.cfg.
syslinuxcfg pretty much just parses for boot menu entries in the
extlinux.conf file and merges them with its own boot menu entries.

At the moment, I expect this to be broken because, again, Fedora's
GRUB depends on commands linuxefi/linux16 and initrdefi/initrd16,
which no one else uses as far as I know. Not upstream or other
distros. I do not know why this detail goes in the configuration file
rather than being substituted in code based on what firmware the
system has - it seems the other distros do it that way. But this seems
surmountable.

However...





> So possibly we can settle on syslinux syntax for bootloader config
> long-term ...
>
>> > I agree with Kevin that grub2 is nonintuitive.
>
> ... and have a fixed grub2.cfg which basically has the command to parse
> the syslinux config file?
>

OK so there are a few things:

1. The information needed to present boot options, the "menu entries",
are a separate thing from the bootloader specific configuration file
and file format. And yet all the bootloaders out there conflate these
two things, and make the problem of cross compatibility more
difficult.

2. The most important and useful part of bootloader spec is splitting
them out. A standardized boot menu entry file, and file format, as
drop-in snippets, which is "loosely based" on the GRUB 1 grub.conf
format. It's just menu entries, and quite simple.

3. The result is bootloader specific configuration files becomes
static, non-changing and don't need to contain boot entries (although
it's not disallowed). And individual boot menu drop in scripts are how
new boot entries are added. The bootloader then merges them together.

4. An issue with using syslinux's format, as well as the original
bootloader spec format, and a major source of disagreement, is the
assumption that the kernel and initrd are on the same file system as
the bootloader and its configuration file. No cross volume hopping.
This is necessary to avoid putting the kernel and initrd on the EFI
System partition, and causing a lot of installation grief with how to
deal with dual boot support.

Anyway, regardless of what format you want to base things on, it
probably should be a superset of the menu entry functions, including a
way to specify volumes by device designation, LVM, or volume UUID, or
now your format isn't actually compatible with GRUB on UEFI as well as
a bunch of less common scenarios.



-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-10 Thread Matthew Miller
On Mon, Oct 10, 2016 at 10:17:56AM +0200, Gerd Hoffmann wrote:
> Hmm, uboot can use extlinux-style config files, and I recently noticed
> grub2 has a command to parse syslinux config files too.  Have not tried
> to use that though.

It does?!

> So possibly we can settle on syslinux syntax for bootloader config
> long-term ...

Oh yes please.



-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-10 Thread Gerd Hoffmann
  Hi,

> > Adam, the only other distro that has serious alternate architecture support,
> > AFAIK, is Debian. How do they handle grub2 + non-x86? Likewise, the
> > alternate architectures that we support, how do we handle their bootloaders?
> > Are they grub-based? Ext/Syslinux based? Grub-legacy?
> 
> Nothing in Fedora uses GRUB legacy.
> 
> ARM is using Das U-Boot. I'm not sure if grubby is involved here or not.
> 
> Fedora uses isolinux, extlinux, grub2, yaboot (power7 and older), and
> zipl (s390). If the last two are gone or going away then it's syslinux
> (and variants), grub2, and uboot.

Hmm, uboot can use extlinux-style config files, and I recently noticed
grub2 has a command to parse syslinux config files too.  Have not tried
to use that though.

So possibly we can settle on syslinux syntax for bootloader config
long-term ...

> > I agree with Kevin that grub2 is nonintuitive.

... and have a fixed grub2.cfg which basically has the command to parse
the syslinux config file?

cheers,
  Gerd
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-09 Thread Peter Robinson
>> My recollection is grubby was going to get a rethink, but I don't know
>> the scope. There are test cases built-into grubby that are considered
>> valuable, I'm not sure about the rest. Gene found the code difficult.
>> I think the main issue is, whether grubby or something else, it needs
>> to be easier to follow the trail of breadcrumbs, self describing. If
>> it's too easy, of course, it just means telling different people "no"
>> about their use case. If you're going to support all use cases that's
>> hard to invent and maintain. See GRUB.
>
> IMHO, you should just drop grubby entirely and configure GRUB the way
> upstream intended. As you wrote in the other thread:

Except that grubby doesn't just configure grub, it's also used for
syslinux/extlinux boot options also in use in Fedora. Those use cases
would need to be accounted for as well.

Peter
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-08 Thread Chris Murphy
On Sat, Oct 8, 2016 at 5:24 AM, Josh Boyer  wrote:
> On Fri, Oct 7, 2016 at 8:01 PM, Andrew Lutomirski  wrote:
>> On Fri, Oct 7, 2016 at 4:55 PM, Josh Boyer  wrote:
>>> On Fri, Oct 7, 2016 at 5:14 PM, Chris Murphy  
>>> wrote:
 Changed this subject to match the other one I changed, so if I'm doing
 it wrong at least I'm consistent!

 On Fri, Oct 7, 2016 at 9:22 AM, Kevin Fenzi  wrote:
> So, I think we would need to get buyin to get our changes into the spec
> and get everyone to agree on it before we would want to move to it.

 The original spec is unworkable. And attempts to expand the upstream
 spec for real world use cases needed in Fedora haven't gone anywhere.
 It exchanges reduced complexity in the bootloader, by increasing it in
 the installer and increasing wasted space on the drive. I'd try to
 help make it salvageable but I think it's pointless. Make a Fedora
 specific version and if people like it they can adopt it.
>>>
>>> Which, in effect, could just as well be writing a "specification" for 
>>> grubby.
>>
>> I disagree.
>>
>> The bootloader spec, update-grub, and pretty much everything except
>> grubby are stateless in the sense that the list of installed kernels
>> and other things is generated freshly each time it updates.  (The
>> bootloader spec does this at every boot, and update-grub does it
>> whenever it's run.)  These solutions don't try to let users customize
>> the result after the fact.  Instead, any customization is written to a
>> file intended for the specific purpose of customizing the boot
>> process, and that file is never written by automatic tools.
>> Bootloader spec parsing, update-grub, etc are all idempotent, which is
>> a *really* nice property.
>>
>> Grubby attempts to parse a configuration file and apply a change for a
>> single kernel addition or removal to it without losing other
>> customizations.  This is a much harder problem to solve well.  It's
>> also not idempotent.  Unsurprisingly (to me, anyway), it doesn't seem
>> to work all that well and it's a PITA to maintain.
>
> You're focusing on the technical aspects of the implementations.  I
> was making a subtle point about the fact that any Fedora-specific
> version of a spec will simply remain a Fedora specific version, which
> defeats the purpose of a specification to begin with.  In essence,
> we'll wind up repeating the implied lesson of grubby.
>
> If we're going to work on a specification, it should be done in
> coordination with other distros.

My idea of a spec was just to document what Fedora needs to support,
so it's known what work needs to be done, including documentation.
That is, start with what we're actually willing to do, rather than
some fiction we won't even follow anyway.

Of the two bootloader specs, we don't exactly follow either one of
them, in meaningful ways: Like where the configuration files live,
which is a fundamental point of those specs.

If GRUB upstream code is an implicit spec, we don't follow that
either. Fedora's GRUB puts the bootloader config file in a different
location depending on the firmware being used; upstream and other
distros don't do that. And Fedora's GRUB uses kernel and initrd
commands that upstream does not use, nor any other distro I've looked
at, and therefore our GRUB is unable to use other distro's grub.cfgs.
When it comes to discovery of other OS's our installer team rejects
the idea they're supposed to activate all LVM LV's so os-prober can
discover those OS's. Fedora is divergent. Even if there were
coordination done with other distros on a spec, why would we suddenly
start following that one?

Anyway, coordination has been tried and failed. Whether the distros
just don't care about the problem, or actually enjoy the mutual
incompatibility, I can't say. But in the end it's gone no where.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-08 Thread Kevin Kofler
Chris Murphy wrote:
> My recollection is grubby was going to get a rethink, but I don't know
> the scope. There are test cases built-into grubby that are considered
> valuable, I'm not sure about the rest. Gene found the code difficult.
> I think the main issue is, whether grubby or something else, it needs
> to be easier to follow the trail of breadcrumbs, self describing. If
> it's too easy, of course, it just means telling different people "no"
> about their use case. If you're going to support all use cases that's
> hard to invent and maintain. See GRUB.

IMHO, you should just drop grubby entirely and configure GRUB the way 
upstream intended. As you wrote in the other thread:

> I don't have a complete or recent evaluation but as of a couple years
> ago, before Gene Czarcinski passed away, we found no other
> distribution using grubby. Distros we ran into use grub-mkconfig as
> recommended by upstream to obliterate the existing grub.cfg and create
> an entirely new one.

As a result, all the configuration GUIs for GRUB, such as kcm-grub2, just 
run grub(2)-mkconfig and will those overwrite any changes you or grubby made 
to grub.cfg directly. Alternative installers such as Calamares also do not 
use grubby, but only grub(2)-mkconfig. (Thankfully, grubby has no problems 
editing the resulting configuration later, when it is invoked by new-kernel-
pkg. But all other distros just rerun grub(2)-mkconfig in that situation.)

Fedora doesn't even remove the comment at the beginning of grub.cfg saying:
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
even though nothing in Fedora actually does that.

Dropping the Fedora-only grubby hack would make life easier for everybody.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-08 Thread Josh Boyer
On Fri, Oct 7, 2016 at 8:01 PM, Andrew Lutomirski  wrote:
> On Fri, Oct 7, 2016 at 4:55 PM, Josh Boyer  wrote:
>> On Fri, Oct 7, 2016 at 5:14 PM, Chris Murphy  wrote:
>>> Changed this subject to match the other one I changed, so if I'm doing
>>> it wrong at least I'm consistent!
>>>
>>> On Fri, Oct 7, 2016 at 9:22 AM, Kevin Fenzi  wrote:
 So, I think we would need to get buyin to get our changes into the spec
 and get everyone to agree on it before we would want to move to it.
>>>
>>> The original spec is unworkable. And attempts to expand the upstream
>>> spec for real world use cases needed in Fedora haven't gone anywhere.
>>> It exchanges reduced complexity in the bootloader, by increasing it in
>>> the installer and increasing wasted space on the drive. I'd try to
>>> help make it salvageable but I think it's pointless. Make a Fedora
>>> specific version and if people like it they can adopt it.
>>
>> Which, in effect, could just as well be writing a "specification" for grubby.
>
> I disagree.
>
> The bootloader spec, update-grub, and pretty much everything except
> grubby are stateless in the sense that the list of installed kernels
> and other things is generated freshly each time it updates.  (The
> bootloader spec does this at every boot, and update-grub does it
> whenever it's run.)  These solutions don't try to let users customize
> the result after the fact.  Instead, any customization is written to a
> file intended for the specific purpose of customizing the boot
> process, and that file is never written by automatic tools.
> Bootloader spec parsing, update-grub, etc are all idempotent, which is
> a *really* nice property.
>
> Grubby attempts to parse a configuration file and apply a change for a
> single kernel addition or removal to it without losing other
> customizations.  This is a much harder problem to solve well.  It's
> also not idempotent.  Unsurprisingly (to me, anyway), it doesn't seem
> to work all that well and it's a PITA to maintain.

You're focusing on the technical aspects of the implementations.  I
was making a subtle point about the fact that any Fedora-specific
version of a spec will simply remain a Fedora specific version, which
defeats the purpose of a specification to begin with.  In essence,
we'll wind up repeating the implied lesson of grubby.

If we're going to work on a specification, it should be done in
coordination with other distros.

josh
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-07 Thread Andrew Lutomirski
On Fri, Oct 7, 2016 at 4:55 PM, Josh Boyer  wrote:
> On Fri, Oct 7, 2016 at 5:14 PM, Chris Murphy  wrote:
>> Changed this subject to match the other one I changed, so if I'm doing
>> it wrong at least I'm consistent!
>>
>> On Fri, Oct 7, 2016 at 9:22 AM, Kevin Fenzi  wrote:
>>> So, I think we would need to get buyin to get our changes into the spec
>>> and get everyone to agree on it before we would want to move to it.
>>
>> The original spec is unworkable. And attempts to expand the upstream
>> spec for real world use cases needed in Fedora haven't gone anywhere.
>> It exchanges reduced complexity in the bootloader, by increasing it in
>> the installer and increasing wasted space on the drive. I'd try to
>> help make it salvageable but I think it's pointless. Make a Fedora
>> specific version and if people like it they can adopt it.
>
> Which, in effect, could just as well be writing a "specification" for grubby.

I disagree.

The bootloader spec, update-grub, and pretty much everything except
grubby are stateless in the sense that the list of installed kernels
and other things is generated freshly each time it updates.  (The
bootloader spec does this at every boot, and update-grub does it
whenever it's run.)  These solutions don't try to let users customize
the result after the fact.  Instead, any customization is written to a
file intended for the specific purpose of customizing the boot
process, and that file is never written by automatic tools.
Bootloader spec parsing, update-grub, etc are all idempotent, which is
a *really* nice property.

Grubby attempts to parse a configuration file and apply a change for a
single kernel addition or removal to it without losing other
customizations.  This is a much harder problem to solve well.  It's
also not idempotent.  Unsurprisingly (to me, anyway), it doesn't seem
to work all that well and it's a PITA to maintain.

--Andy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org


Re: grub, grubby, btrfs, was: PSA: Do not run 'dnf update' inside GNOME, KDE ...

2016-10-07 Thread Josh Boyer
On Fri, Oct 7, 2016 at 5:14 PM, Chris Murphy  wrote:
> Changed this subject to match the other one I changed, so if I'm doing
> it wrong at least I'm consistent!
>
> On Fri, Oct 7, 2016 at 9:22 AM, Kevin Fenzi  wrote:
>> So, I think we would need to get buyin to get our changes into the spec
>> and get everyone to agree on it before we would want to move to it.
>
> The original spec is unworkable. And attempts to expand the upstream
> spec for real world use cases needed in Fedora haven't gone anywhere.
> It exchanges reduced complexity in the bootloader, by increasing it in
> the installer and increasing wasted space on the drive. I'd try to
> help make it salvageable but I think it's pointless. Make a Fedora
> specific version and if people like it they can adopt it.

Which, in effect, could just as well be writing a "specification" for grubby.

I don't see the point in writing something nobody will adopt twice.

josh
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org