Bug#1031639: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[James Valleroy]

On Sun, 19 Feb 2023 19:01:22 +0100 Paul Gevers  wrote:


> On 2023-02-15 21:04:36 +0100, Sebastian Ramacher wrote:
> > 
> > On 2023-02-14 01:01:38 +0100, Daniel Leidert wrote:
> > > 
> > > This problem breaks e.g. vmdb2. I can no longer create a Bullseye

> > > system image with vmdb2 on Sid, because the grub-install step in the
> > > Bullseye chroot now fails, because the created filesystem (created with
> > > e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
> > > e2fsprogs to the version in Testing to get the build going again. It
> > > also means that a Bookworm system can never be used to format and
> > > debootstrap a Bullseye or Buster system that requires a grub
> > > installation.
> > > 
> > > I guess that the fix applied to grub2 in Sid would have to be applied

> > > to grub2 in Bullseye as well (and basically to any grub2 package in any
> > > Debian or Ubuntu or Raspbian release supported by debootstrap).
> > > 
> > > This situation is really messy. It breaks basically all my image builds

> > > with vmdb2.
> > 
> > Regardless of the outcome of #1031325, this issue will need to be fixed

> > in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
> > account for the feature and disable it if necessary for older
> > distributions.
> > 
> > Cloning and reassign to vmdb2.
> 
> Based on more feedback from #10313225, I am also cloning and reassigning

> this issue to fai and grml-debootstrap. Dear maintainers, please check
> whether this issue is relevant for your packages.

The same appears to apply for debos and freedom-maker. Dear maintainers, 
please check whether this issue is relevant for your packages.


Control: severity -1 normal

The images built by freedom-maker have btrfs root partition by default, which 
are not affected by e2fsprogs. Although freedom-maker includes code to support 
ext4 root partition, to actually build an image using ext4 requires patching 
the source code of freedom-maker. If one modifies freedom-maker in this way, 
then there will be an error if building a bullseye image on a bookworm host:

2023-02-23 13:08:25,297 - ERROR - Target failed - virtualbox-amd64-ext4
Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "/home/james/salsa/freedombox-team/freedom-maker/freedommaker/__main__.py", 
line 13, in 
Application().run()
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/application.py", 
line 57, in run
builder.build()
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/builders/vm.py", 
line 21, in build
self.make_image()
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/builder.py", line 
106, in make_image
self._get_builder_backend().make_image()
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/internal.py", 
line 43, in make_image
self._install_boot_loader()
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/internal.py", 
line 352, in _install_boot_loader
library.install_grub(self.state)
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/library.py", line 
476, in install_grub
run_in_chroot(state, ['grub-install', device] + args)
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/library.py", line 
49, in run_in_chroot
return run(*args, **kwargs)
   
  File 
"/home/james/salsa/freedombox-team/freedom-maker/freedommaker/library.py", line 
43, in run
return cliapp.runcmd(*args, **kwargs)
   ^^
  File "/usr/lib/python3/dist-packages/cliapp/runcmd.py", line 64, in runcmd
raise cliapp.AppException(msg)
cliapp.app.AppException: Command failed: chroot /tmp/tmpxa690dib grub-install 
/dev/loop0
b''
b'Installing for i386-pc platform.\ngrub-install: error: unknown filesystem.\n'

Building a bookworm image that uses ext4 does not have this error.

In addition, it's not clear to me that there is a requirement for freedom-maker 
in bookworm to build bullseye images.


OpenPGP_signature
Description: OpenPGP digital signature

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Paul Gevers]

Control: clone -1 -2
Control: reassign -2 freedom-make 0.32
Control: clone -1 -3
Control: reassign -3 debos 1.1.1-2


On 2023-02-15 21:04:36 +0100, Sebastian Ramacher wrote:
> Control: clone -1 -2
> Control: reassign -2 vmdb2 0.26-2
> 
> On 2023-02-14 01:01:38 +0100, Daniel Leidert wrote:

> > Hi Steve,
> > 
> > I believe that your fix to grub2 in Sid is not enough to handle

> > #1030939/#1030846.
> > 
> > This problem breaks e.g. vmdb2. I can no longer create a Bullseye

> > system image with vmdb2 on Sid, because the grub-install step in the
> > Bullseye chroot now fails, because the created filesystem (created with
> > e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
> > e2fsprogs to the version in Testing to get the build going again. It
> > also means that a Bookworm system can never be used to format and
> > debootstrap a Bullseye or Buster system that requires a grub
> > installation.
> > 
> > I guess that the fix applied to grub2 in Sid would have to be applied

> > to grub2 in Bullseye as well (and basically to any grub2 package in any
> > Debian or Ubuntu or Raspbian release supported by debootstrap).
> > 
> > This situation is really messy. It breaks basically all my image builds

> > with vmdb2.
> 
> Regardless of the outcome of #1031325, this issue will need to be fixed

> in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
> account for the feature and disable it if necessary for older
> distributions.
> 
> Cloning and reassign to vmdb2.


Based on more feedback from #10313225, I am also cloning and reassigning
this issue to fai and grml-debootstrap. Dear maintainers, please check
whether this issue is relevant for your packages.


The same appears to apply for debos and freedom-maker. Dear maintainers, 
please check whether this issue is relevant for your packages.


Paul


OpenPGP_signature
Description: OpenPGP digital signature

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Sebastian Ramacher]

Control: clone -1 -2
Control: reassign -2 fai 6.0
Control: clone -1 -3
Control: reassign -3 grml-debootstrap 0.102

On 2023-02-15 21:04:36 +0100, Sebastian Ramacher wrote:
> Control: clone -1 -2
> Control: reassign -2 vmdb2 0.26-2
> 
> On 2023-02-14 01:01:38 +0100, Daniel Leidert wrote:
> > Hi Steve,
> > 
> > I believe that your fix to grub2 in Sid is not enough to handle
> > #1030939/#1030846.
> > 
> > This problem breaks e.g. vmdb2. I can no longer create a Bullseye
> > system image with vmdb2 on Sid, because the grub-install step in the
> > Bullseye chroot now fails, because the created filesystem (created with
> > e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
> > e2fsprogs to the version in Testing to get the build going again. It
> > also means that a Bookworm system can never be used to format and
> > debootstrap a Bullseye or Buster system that requires a grub
> > installation.
> > 
> > I guess that the fix applied to grub2 in Sid would have to be applied
> > to grub2 in Bullseye as well (and basically to any grub2 package in any
> > Debian or Ubuntu or Raspbian release supported by debootstrap).
> > 
> > This situation is really messy. It breaks basically all my image builds
> > with vmdb2.
> 
> Regardless of the outcome of #1031325, this issue will need to be fixed
> in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
> account for the feature and disable it if necessary for older
> distributions.
> 
> Cloning and reassign to vmdb2.

Based on more feedback from #10313225, I am also cloning and reassigning
this issue to fai and grml-debootstrap. Dear maintainers, please check
whether this issue is relevant for your packages.

Cheers
-- 
Sebastian Ramacher

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Sebastian Ramacher]

Control: clone -1 -2
Control: reassign -2 vmdb2 0.26-2

On 2023-02-14 01:01:38 +0100, Daniel Leidert wrote:
> Hi Steve,
> 
> I believe that your fix to grub2 in Sid is not enough to handle
> #1030939/#1030846.
> 
> This problem breaks e.g. vmdb2. I can no longer create a Bullseye
> system image with vmdb2 on Sid, because the grub-install step in the
> Bullseye chroot now fails, because the created filesystem (created with
> e2fsprogs on Sid) cannot be recognized by grub. I have to downgrade
> e2fsprogs to the version in Testing to get the build going again. It
> also means that a Bookworm system can never be used to format and
> debootstrap a Bullseye or Buster system that requires a grub
> installation.
> 
> I guess that the fix applied to grub2 in Sid would have to be applied
> to grub2 in Bullseye as well (and basically to any grub2 package in any
> Debian or Ubuntu or Raspbian release supported by debootstrap).
> 
> This situation is really messy. It breaks basically all my image builds
> with vmdb2.

Regardless of the outcome of #1031325, this issue will need to be fixed
in vmdb2 eventually. vmdb2, similar to other bootstraping tools, has to
account for the feature and disable it if necessary for older
distributions.

Cloning and reassign to vmdb2.

Cheers
-- 
Sebastian Ramacher

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 16:53 -0500 schrieb Theodore Ts'o:

[snip]

Your arrogant and ignorant attitude is frustrating, to say the least.
You don't care about the mess you create, for a feature, that probably
only a handful of people will ever need (I did a quick search and
didn't find anything regarding this feature - so why turn it on by
default then?). But yet you have to make it the default and break
running toolchains and methods. Talking to you is pointless. You cost
me hours of useless work yesterday and today because you ignore the
rules we have set out as a project to not frustrate each other.

I have reported this to the release team now.

Daniel

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Theodore Ts'o]

On Tue, Feb 14, 2023 at 07:35:51PM +0100, Daniel Leidert wrote:
> 
> As soon as this version hits testing, you have successfully disabled
> the last working environment to use vmdb2 to create images of Ubuntu
> and Debian. As soon as this version hits Testing, one then can no
> longer build images for either any Ubuntu release nor Debian Bullseye
> or older. I can then only build images for Bookworm and Sid. Please
> think about that.

The number of users who use vmdb2 are quite small; for those users who
do, there is a simple fix.  You can either diable the feature using
tune2fs, or you can just make an edit to your /etc/mke2fs.conf file to
not enable the feature.  A one line change to /etc/mke2fs.conf is
hardly what I'd call "impossible".

> You *seriously* break the debootstrap method of installing Debian or
> Ubuntu.

As you have pointed out, this is not a debootstrap issue, since it
doesn't create the file system.  The uestion is in how the file system
is created, and this is not hard to fix.  You can just run "mke2fs -O
^metadata_csum_seed _file_or_block_device_"; you can run "tune2fs -O
^metadata_csum_seed _file_or_block_device"; you can make a one-line
change to /etc/mke2fs.conf.

> You haven't documented how to disable that
> breaking change when creating filesystems for distributions where grub
> does not support this (there is not even a NEWS entry). You haven't
> even announced that breaking change. And yet you are going to break one
> of our two standard methods of installing Debian or Ubuntu. How is any
> of what you have been saying so far addressing any of these issues?

Sorry, as far as I'm concerned, vmdb2 is not that important.  We don't
document in a NEWS entry or anywhere else, how to build a binary that
links against a newer version of glibc so it will work on an older
system.  And I would consider the compiler toolchain to be far more
common a usecase than vmdb2.  Indeed, your use of "toolchain" for file
system utilities is a new one for me.  I've never heard the term
"toolchain" used in such a way before.

> I do not understand why you are pushing 1.47.0 over 1.46.6, which you
> had uploaded just five days before the former. You haven't even
> presented a reason yet.

It has anumber of new features that will improve ext4's performance on
highly parallel workloads; it makes it possible for cloud VM's to be
able to safely update the root file systems's UUID while it is
mounted, among other changes.

- Ted

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 12:34 -0500 schrieb Theodore Ts'o:
> 
[..]
> In any case, a version of grub that will support the csum_seed feature
> will be landing in Bookworm in just a few days.  So at that point,
> you'll be able to create VM images for Bookworm and Sid that will work
> with the e2fsprogs in sid.  The current plan of record is that it will
> only be at that point that e2fsprogs will be allowed to migrate into
> Bookworm.

As soon as this version hits testing, you have successfully disabled
the last working environment to use vmdb2 to create images of Ubuntu
and Debian. As soon as this version hits Testing, one then can no
longer build images for either any Ubuntu release nor Debian Bullseye
or older. I can then only build images for Bookworm and Sid. Please
think about that.

You *seriously* break the debootstrap method of installing Debian or
Ubuntu. vmdb2 ist just another tool that is broken by now, a tool that
I use very often. I explained the impacts of what you are doing often
enough now. By now the impact should be clear. And you are still not
dealing with the aftermath of your intended action?! You haven't done a
proper transition. You haven't given any affected toolchain the
necessary time to adopt. You haven't documented how to disable that
breaking change when creating filesystems for distributions where grub
does not support this (there is not even a NEWS entry). You haven't
even announced that breaking change. And yet you are going to break one
of our two standard methods of installing Debian or Ubuntu. How is any
of what you have been saying so far addressing any of these issues? 

> [..]
> By the way, in the case of the csum_seed feature, it's pretty
> straightforward to just run "tune2fs -O ^metadata_csum_seed
> /tmp/boot.img".  If the UUID has been changed since the file system
> was created, you'll have to do this while the file system is
> unmounted
> and it will take a few seconds, but that's almost certainly not the
> case with vmdb2.

Well, how do you intend to distribute that information, so people who
have to use the deboostrap method to install a Debian or Ubuntu release
with a grub not supporting csum_seed (basically all existing releases,
except for the upcoming Bookworm) know that?

I do not understand why you are pushing 1.47.0 over 1.46.6, which you
had uploaded just five days before the former. You haven't even
presented a reason yet.

Regards, Daniel

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Theodore Ts'o]

There is another issue with vmdb2 if you are using XFS.  Starting with
xfsprogs 5.15 (which is already in testing), bigtime is enabled by
default, so that newly created XFS file systems won't be subject to
timestamp overflow in 2038.  Grub didn't land support for this feature
until 8b1e5d1936ff ("fs/xfs: Add bigtime incompat feature support") in
May 2021, despite the fact that XFS has had this feature for years and
years and years.

So if you aren't using the latest security fixes, and you are using
XFS as the boot partition --- it won't work on buster and bullseye.
"Fortunately", there were were massive number security vulnerabilities
in grub2 which forced a backport of grub2 2.06 to bullseye and buster,
so if you have the security updates enabled, you'll probably be OK ---
but it was only because of massive number of security problems forced
that backport.


In any case, a version of grub that will support the csum_seed feature
will be landing in Bookworm in just a few days.  So at that point,
you'll be able to create VM images for Bookworm and Sid that will work
with the e2fsprogs in sid.  The current plan of record is that it will
only be at that point that e2fsprogs will be allowed to migrate into
Bookworm.

For slowly moving upstreams like grub2, distributions *have* to take
updates before grub2 finally gets around to doing a release --- to get
security fixes if nothing else!  The support for csum_seed has been in
Fedora and other distributions for a while, since the patches had
landed in grub2 in June 2021.  I probably should have made sure the
feature had landed in Debian's grub2 packaging earlier; that's my bad,
and my apologies for that.

Note that Debian's grub2 has well over 100 patches, nearly all of
which are backports from grub2's git repo.  So the argument that
"there doesn't exist a formally released grub2 release" isn't
particularly compelling, since all the distros are backporting
patches.  The only question is how *many* commits release has an
individual distribution taken.


By the way, in the case of the csum_seed feature, it's pretty
straightforward to just run "tune2fs -O ^metadata_csum_seed
/tmp/boot.img".  If the UUID has been changed since the file system
was created, you'll have to do this while the file system is unmounted
and it will take a few seconds, but that's almost certainly not the
case with vmdb2.

   - Ted

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 14:58 + schrieb Steve McIntyre:
> On Tue, Feb 14, 2023 at 12:50:18PM +0100, Daniel Leidert wrote:
> 
> 
[..]
> Breakages happen like this,

This breakage is *unnecessary* and it breaks at the momnent *all*
debootstrap installations except for doing a bookworm/sid installation
themselves. That is just stupid.

Get down from your high horse and ackknowledge the problems your
behavior causes.

> and this has happened before in similar
> circumstances.

No it has not. You are doing a breakage on purpose during a freeze
period while the transition period is over. Do it with a proper
transition during the next release cycle.


> > 
[..]
> > Whe whole handling is completely wrong here. First, grub should have
> > been fixed upstream. And the change in e2fsprogs should have been done
> > only after "fixed" grub versions had settled. If we do it the other way
> > around, we have to patch grub in affected distributions as well. And
> > for Debian that means at least to patch Bullseye and any other release
> > we want to be able to install from Bookworm. I even a lot of companies
> > using Buster still.
> 
> And I know of folks still working on Stretch and Jessie. How far back
> do you want to tie things??

How about staying on topic and explaining, why this transition is
necessary and has to be done the wrong way around, instead of picking
the fact that older systems still exist? You break almost *all*
installation right now. You also broke an Ubuntu 20.04 or 22.04 LTS
installation. Are they new enough?

[..]
> > 
> > I'm critizicing the way of handling that breaking change and the
> > ignorance shown reagarding the impact, not that fact that there is a
> > breaking change. And it breaks a lot! This doesn't affect just a few
> > minor use cases. It affects the basic way of installing a clean Ubuntu
> > or Debian (or derivative) on a remote server using the debootstrap
> > method.
> 
> People using these tools need to be aware of the potential issue. What
> would happen if you ran debootstrap with a filesystem that the target
> distro doesn't know how to mount at all, for example?

That is different from introducing a breaking change for which no grub
upstream release has a fix yet. So basically the only system able to
handle a filesystem created with e2fsprogs 1.47.0 is Sid right now. Can
you please ignore your ego and see the problems you are causing?

You push a breaking change for no reason at all. What is the gain here
compared to the issues you are causing?

> > And again: We are in the middle of a freeze here. And e2fsprogs
> > pushes
> > a breaking change that is not even handled by any existing grub
> > upstream release, and is also not properly handled within Debian?!
> 
> Grub upstream is already known to be problematic in terms of release
> cycles.

That is not enough and it is not a solution for the problem. There is
*no* grub version out there supporting this, except for the patched
version is Sid. Is this the sign for a working transition?

> We now know about this particular issue (thanks Ted!) and
> we've fixed it in unstable (and soon testing).

Which helps exactly nobody, as it still breaks all other Debian or
Ubuntu installation.

I cannot belive that you intentionally break one of the standard
methods to install Debian or Ubuntu for no reason at all and without a
proper transition. Version 1.47.0 of e2fsprogs contains nothing
necessary for Bookworm. I'll bring this to the attention of the release
team. I'm sick of your ignorance. Do a proper transition and don't
start it during a freeze.


Daniel

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Steve McIntyre]

On Tue, Feb 14, 2023 at 12:50:18PM +0100, Daniel Leidert wrote:
>Am Dienstag, dem 14.02.2023 um 10:45 + schrieb Steve McIntyre:
>> On Tue, Feb 14, 2023 at 11:34:13AM +0100, Daniel Leidert wrote:
>> > Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:
>> 
>> ...
>> 
>> > > But a generalized requirement that we be able to use debootstrap and
>> > > vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
>> > > seem to be reasonable.
>> > 
>> > You are completyl wrong. This breaks a standard way of installing any
>> > system supported by deboostrap with a grub without a fix to deal with
>> > this version of e2fsprogs. This isn't just about vmdb2.
>> > 
>> > What you are saying is ignorant.
>> > 
>> > If this isn't cared about, then this version of e2fsprogs shouldn't
>> > make it into Bookworm. We are in the middle of a freeze and this breaks
>> > toolchains and a standard way (see [1]) of installing Debian.
>> 
>> Sorry Daniel, but I have to (mostly!) agree with Ted here. If you're
>> creating an image of an older release using newer tools then you'll
>> need to be aware that sometimes the newer tools will create things
>> that don't work there. If there's a bug here, I would strongly argue
>> that it's in vmdb2. deboostrap (for example) includes some
>> release-specific knowledge to cope with issues like this.
>
>debootstrap does nothing related to grub. So it is a bad example.

That's why I said *like* this, not *exactly* like this. debootstrap
has had knowledge of things like fs layouts etc. that older releases
need (e.g. merged-/usr).

>Again I refer to [1]: If the host system contains the problematic
>e2fsprogs and the target system doesn't contain a grub with the fix
>[2], then this breaks installations. This breaks older systems *and*
>current systems. For example, I neither see the necessary grub patch
>in both Ubuntu 20.04 and 22.04 either. So they also cannot be
>installed using the deboostrap method and the toolchain in Sid (and
>Bookworm if e2fsprogs makes it there).

Breakages happen like this, and this has happened before in similar
circumstances. If you're installing an older system using brand-new
tools, you need to be aware of the potential for things to not
work. In this particular case, all you need to do is tweak the flags
on the ext4 filesystem when you create it. This isn't that hard...

>[1] https://www.debian.org/releases/stable/amd64/apds03
>[2] Even "our" grub only contains a patch and not an upstream version
>with support. So how can you even expect the target system to contain a
>fix and be able to handle the created filesystem?!
>
>Whe whole handling is completely wrong here. First, grub should have
>been fixed upstream. And the change in e2fsprogs should have been done
>only after "fixed" grub versions had settled. If we do it the other way
>around, we have to patch grub in affected distributions as well. And
>for Debian that means at least to patch Bullseye and any other release
>we want to be able to install from Bookworm. I even a lot of companies
>using Buster still.

And I know of folks still working on Stretch and Jessie. How far back
do you want to tie things??

>> If we don't allow for this kind of change, that wouldn't allow us to
>> *ever* make breaking changes in some packages, and that's just not
>> sustainable.
>
>I'm critizicing the way of handling that breaking change and the
>ignorance shown reagarding the impact, not that fact that there is a
>breaking change. And it breaks a lot! This doesn't affect just a few
>minor use cases. It affects the basic way of installing a clean Ubuntu
>or Debian (or derivative) on a remote server using the debootstrap
>method.

People using these tools need to be aware of the potential issue. What
would happen if you ran debootstrap with a filesystem that the target
distro doesn't know how to mount at all, for example?

>And again: We are in the middle of a freeze here. And e2fsprogs pushes
>a breaking change that is not even handled by any existing grub
>upstream release, and is also not properly handled within Debian?!

Grub upstream is already known to be problematic in terms of release
cycles. We now know about this particular issue (thanks Ted!) and
we've fixed it in unstable (and soon testing).

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Who needs computer imagery when you've got Brian Blessed?

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Daniel Leidert]

Am Dienstag, dem 14.02.2023 um 10:45 + schrieb Steve McIntyre:
> On Tue, Feb 14, 2023 at 11:34:13AM +0100, Daniel Leidert wrote:
> > Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:
> 
> ...
> 
> > > But a generalized requirement that we be able to use debootstrap and
> > > vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
> > > seem to be reasonable.
> > 
> > You are completyl wrong. This breaks a standard way of installing any
> > system supported by deboostrap with a grub without a fix to deal with
> > this version of e2fsprogs. This isn't just about vmdb2.
> > 
> > What you are saying is ignorant.
> > 
> > If this isn't cared about, then this version of e2fsprogs shouldn't
> > make it into Bookworm. We are in the middle of a freeze and this breaks
> > toolchains and a standard way (see [1]) of installing Debian.
> 
> Sorry Daniel, but I have to (mostly!) agree with Ted here. If you're
> creating an image of an older release using newer tools then you'll
> need to be aware that sometimes the newer tools will create things
> that don't work there. If there's a bug here, I would strongly argue
> that it's in vmdb2. deboostrap (for example) includes some
> release-specific knowledge to cope with issues like this.

debootstrap does nothing related to grub. So it is a bad example. Again
I refer to [1]: If the host system contains the problematic e2fsprogs
and the target system doesn't contain a grub with the fix [2], then
this breaks installations. This breaks older systems *and* current
systems. For example, I neither see the necessary grub patch in both
Ubuntu 20.04 and 22.04 either. So they also cannot be installed using
the deboostrap method and the toolchain in Sid (and Bookworm if
e2fsprogs makes it there). 

[1] https://www.debian.org/releases/stable/amd64/apds03
[2] Even "our" grub only contains a patch and not an upstream version
with support. So how can you even expect the target system to contain a
fix and be able to handle the created filesystem?!

Whe whole handling is completely wrong here. First, grub should have
been fixed upstream. And the change in e2fsprogs should have been done
only after "fixed" grub versions had settled. If we do it the other way
around, we have to patch grub in affected distributions as well. And
for Debian that means at least to patch Bullseye and any other release
we want to be able to install from Bookworm. I even a lot of companies
using Buster still.

> If we don't allow for this kind of change, that wouldn't allow us to
> *ever* make breaking changes in some packages, and that's just not
> sustainable.

I'm critizicing the way of handling that breaking change and the
ignorance shown reagarding the impact, not that fact that there is a
breaking change. And it breaks a lot! This doesn't affect just a few
minor use cases. It affects the basic way of installing a clean Ubuntu
or Debian (or derivative) on a remote server using the debootstrap
method.


And again: We are in the middle of a freeze here. And e2fsprogs pushes
a breaking change that is not even handled by any existing grub
upstream release, and is also not properly handled within Debian?!

Daniel

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Steve McIntyre]

On Tue, Feb 14, 2023 at 11:34:13AM +0100, Daniel Leidert wrote:
>Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:

...

>> But a generalized requirement that we be able to use debootstrap and
>> vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
>> seem to be reasonable.
>
>You are completyl wrong. This breaks a standard way of installing any
>system supported by deboostrap with a grub without a fix to deal with
>this version of e2fsprogs. This isn't just about vmdb2.
>
>What you are saying is ignorant.
>
>If this isn't cared about, then this version of e2fsprogs shouldn't
>make it into Bookworm. We are in the middle of a freeze and this breaks
>toolchains and a standard way (see [1]) of installing Debian.

Sorry Daniel, but I have to (mostly!) agree with Ted here. If you're
creating an image of an older release using newer tools then you'll
need to be aware that sometimes the newer tools will create things
that don't work there. If there's a bug here, I would strongly argue
that it's in vmdb2. deboostrap (for example) includes some
release-specific knowledge to cope with issues like this.

If we don't allow for this kind of change, that wouldn't allow us to
*ever* make breaking changes in some packages, and that's just not
sustainable.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Daniel Leidert]

Am Montag, dem 13.02.2023 um 21:35 -0500 schrieb Theodore Ts'o:
> On Tue, Feb 14, 2023 at 01:01:38AM +0100, Daniel Leidert wrote:
> > Hi Steve,
> > 
> > I believe that your fix to grub2 in Sid is not enough to handle
> > #1030939/#1030846.
> > 
> > This problem breaks e.g. vmdb2. I can no longer create a Bullseye
> > system image with vmdb2 on Sid, because the grub-install step in the
> > Bullseye chroot now fails, because the created filesystem (created with
> > e2fsprogs on Sid) cannot be recognized by grub.
> 
> I'm confused, why does using vmdb2 on *sid* involve running
> grub-install on a *bulleye* chroot?

My workstation is running Sid. I want to create an image for Bullseye
(in this case using vmdb2, but I could do it manually as well). First,
one creates the paritioning and the filesystems for the target system
with the tools provided by the host system. This involves running the
unfortunate version of e2fsprogs with the breaking change. Then, one
installs the base system with deboostrap (also from the host system)
into the created partitions/filesystem. Then one (bind-)mounts the
virtual filesystems into the target systems, does a chroot into it, and
then one finishes the installation inside the chroot, including running
grub-install.

This is standard and common behavior. I have never ever seen in my
whole life someone to use grub2 from Sid to make a grub-install for a
Bullseye target. I have not even seen anybody suggest that.

Consider another example: Server providers use GRML or Bookworm with
e2fsprogs 1.47.0 as their rescue system. Now people are no longer able
to create a Bullseye system using the deboostrap method [1]. If the
host system uses e2fsprogs 1.47.0 or above and the target system for
[1] uses a grub without a fix, then this no longer works.

[1] https://www.debian.org/releases/stable/amd64/apds03

> That seems to be extremely ill-advised.

I'm sorry? I think you are completely missing the point.

> If you are trying to install a bullseye system, why
> can't you using vmdb2 on bullseye?

Because I am using Sid and because I use features of vmdb2 which are
not available in the version in Bullseye. And this feature breaks vmdb2
and similar tools. It also breaks a standard way of installing Debian
systems.

> And if you are trying to install a sid or bookworm system using vmdb2,
> why can't you (or vmdb2) use a sid or bookworm chroot for doing the
> grub-install?

What are you talking about? You basically break the toolchains and then
you suggest to do non-standard stuff to handle this or even avoid doing
installations of affected systems?!

> > I guess that the fix applied to grub2 in Sid would have to be applied
> > to grub2 in Bullseye as well (and basically to any grub2 package in any
> > Debian or Ubuntu or Raspbian release supported by debootstrap).
> 
> I can understand why we need to keep things synchronized so that a
> debian installer for Bookworm be able to generate a bootable system
> using the version of grub and e2fsprogs in Bookworm.
> 
> But a generalized requirement that we be able to use debootstrap and
> vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
> seem to be reasonable.

You are completyl wrong. This breaks a standard way of installing any
system supported by deboostrap with a grub without a fix to deal with
this version of e2fsprogs. This isn't just about vmdb2.

What you are saying is ignorant.

If this isn't cared about, then this version of e2fsprogs shouldn't
make it into Bookworm. We are in the middle of a freeze and this breaks
toolchains and a standard way (see [1]) of installing Debian.

Daniel

Bug#1030846: Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Theodore Ts'o]

On Tue, Feb 14, 2023 at 01:01:38AM +0100, Daniel Leidert wrote:
> Hi Steve,
> 
> I believe that your fix to grub2 in Sid is not enough to handle
> #1030939/#1030846.
> 
> This problem breaks e.g. vmdb2. I can no longer create a Bullseye
> system image with vmdb2 on Sid, because the grub-install step in the
> Bullseye chroot now fails, because the created filesystem (created with
> e2fsprogs on Sid) cannot be recognized by grub.

I'm confused, why does using vmdb2 on *sid* involve running
grub-install on a *bulleye* chroot?  That seems to be extremely
ill-advised.  If you are trying to install a bullseye system, why
can't you using vmdb2 on bullseye?

And if you are trying to install a sid or bookworm system using vmdb2,
why can't you (or vmdb2) use a sid or bookworm chroot for doing the
grub-install?

> I have to downgrade e2fsprogs to the version in Testing to get the
> build going again. It also means that a Bookworm system can never be
> used to format and debootstrap a Bullseye or Buster system that
> requires a grub installation.
> 
> I guess that the fix applied to grub2 in Sid would have to be applied
> to grub2 in Bullseye as well (and basically to any grub2 package in any
> Debian or Ubuntu or Raspbian release supported by debootstrap).

I can understand why we need to keep things synchronized so that a
debian installer for Bookworm be able to generate a bootable system
using the version of grub and e2fsprogs in Bookworm.

But a generalized requirement that we be able to use debootstrap and
vmdb2 to be able to bootstrap an arbitrarily old distribution doesn't
seem to be reasonable.  It would mean that we couldn't update to newer
versions of the C library, or of new file system featuers, because we
are somehow obliged to be able to boostrap ancient versions of the
system.  After all, we don't require that a binary built using
Bookworm be able to run on Bullseye.  How is this any different?

I generate test appliances for file system regression testing which
run on Debian Bullseye on my Debian Bookworm system --- and this gets
done using build chroot[1].  I even have super-convenient scripts to
create the build chroot[2].  This is not hard why can't vmdb2 do
the same thing?

[1] 
https://github.com/tytso/xfstests-bld/blob/master/Documentation/building-xfstests.md
[2] https://github.com/tytso/xfstests-bld/blob/master/setup-buildchroot

- Ted

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Cyril Brulebois]

Theodore Ts'o  (2023-02-10):
> But that problem has already been solved by cloning the bug back to
> e2fpsrogs (#1030939) which will prevent e2fsprogs from transitioning
> to testing, no?  So what's the problem.

I never said there was a problem with the current state of things
(indeed, that's one of the two soltions I described as being equally
fine with me).

Instead, I've explained why your suggested “solution” wouldn't help,
with some context since you seemed unconvinced by previous answers from
others.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Theodore Ts'o]

On Fri, Feb 10, 2023 at 08:31:04AM +0100, Cyril Brulebois wrote:
> > Holding back file system development because grub2 uptsream is super
> > slow doesn't seem like a reasonable way forward, so I really don't
> > want to set this precedent.
> 
> The Bookworm freeze has started, we need to be able to install stuff, so
> we need a solution for the grub2/e2fsprogs situation *now*.
> 
> I really don't care about “setting a precedent”.

But that problem has already been solved by cloning the bug back to
e2fpsrogs (#1030939) which will prevent e2fsprogs from transitioning
to testing, no?  So what's the problem.

The fact that grub2 upstream is super-slow in doing releases is
already a pain in *ass that has caused much pain over the years.
Adding a conflict just adds more pain, without adding any additional
benefit.  So what's the point?

 - Ted

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Cyril Brulebois]

Theodore Ts'o  (2023-02-09):
> On Thu, Feb 09, 2023 at 06:55:08PM +0100, Sven Joachim wrote:
> > That is not going to help, because IIUC grub-install is run from the
> > target system that you are installing, and there is no
> > grub2-common-udeb.
> 
> Right, but if the conflict in e2fsprogs-udeb prevents the installer
> from pulling in an overly new version of e2fsprogs-udeb, that woul be
> sufficient, no?

As Sven rightfully pointed out, there are two different environments:
 - installer, with anna and udebs (but that would be the same with apt
   and debs);
 - target.

There are no connections between both environments, so you can't have
package relationships in a cross-environment manner.

The installer uses whatever was available at build-time, or for
components that aren't built-in, whatever is available on the
installation image, or on the mirror. There's no “pulling in an overly
new version” that can be avoided. There's *one* version in the suite,
that's the one that's getting used, there's no alternative.

TL;DR: Some Conflicts, even if that were possible (which it is not)
   wouldn't achieve anything (except probably not offering any ext*
   option at the partioning step — not a huge win).

> The reason why I really don't want to add the conflicts with e2fsprogs
> 1.47 is because for people who are using sid, there is aboslutely
> nothing wrong with installing e2fsprogs 1.47.  It's only if they use
> the installer that sucks in e2fsprogs that there's a problem --- it's
> not an inherent conflict with grub per se.  If it were, then we
> shouldn't allow installation of e2fsprogs 1.46 because grub2 upstream
> is painfully slow in putting out releases --- and that's not
> reasonable.

*sigh* @ the heavy finger pointing in various parts of your mail.

> Now, what I *could* do is change the mke2fs.conf in e2fsprogs-udeb to
> remove the default use of the csum-seed feature  but is it worth
> it?  Hopefully the new version of grub2 will come out soon, at which
> point I would then need to revert the hacked mke2fs.conf --- and your
> dup'ing this bug should prevent the installer from picking up
> e2fsprogs 1.47 until the new version of grub gets released.

Right now what I'm most concerned with is getting grub2 fixed in
unstable, candidate for migration into testing. Until that happens,
e2fsprogs must be kept out of testing, so that the installer cannot get
a too-new e2fsprogs with a too-old grub2.

Whether we introduce a relationship between both packages making britney
wait on grub2's being ready to allow e2fsprogs to migrate alongside it,
or we keep an RC bug on e2fsprogs to keep it out of testing until grub2
gets fixed and migrated into testing… doesn't matter much to me.


I'll word it differently because I'd like to avoid more mails on this
thread: The installer pulls components for the target system from a
single distribution, there's no set of existing packages that can be
kept around (as that would be the case for existing systems using
testing or unstable), so we need packages in the distribution being
installed to be consistent.

Right now: unstable is broken; testing isn't.


[ snip stuff about grub design ]

> Holding back file system development because grub2 uptsream is super
> slow doesn't seem like a reasonable way forward, so I really don't
> want to set this precedent.

The Bookworm freeze has started, we need to be able to install stuff, so
we need a solution for the grub2/e2fsprogs situation *now*.

I really don't care about “setting a precedent”.

[ snip brainstorming ]


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Bastian Blank]

Hi

On Thu, Feb 09, 2023 at 03:16:15PM -0500, Theodore Ts'o wrote:
> Right, but if the conflict in e2fsprogs-udeb prevents the installer
> from pulling in an overly new version of e2fsprogs-udeb, that woul be
> sufficient, no?

No, it does not.  Conflicts have undefined behaviour for udebs.

Bastian

-- 
There are certain things men must do to remain men.
-- Kirk, "The Ultimate Computer", stardate 4929.4

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Theodore Ts'o]

On Thu, Feb 09, 2023 at 06:55:08PM +0100, Sven Joachim wrote:
> >>
> >> Thanks from me as well :-).  To prevent e2fsprogs from migrating to
> >> testing before grub2 and breaking d-i, I am reassigning a copy of this
> >> bug back to e2fsprogs.  It may be closed once grub2 2.06-8 enters
> >> Bookworm.   Perhaps it would also be a good idea to add a versioned
> >> "Breaks: grub2-common (<< 2.06-8~)" to e2fsprogs.
> >
> > Perhaps we could just add a "Breaks: grub2-common (<< 2.06-8~)" to
> > e2fsprogs-udeb?
> 
> That is not going to help, because IIUC grub-install is run from the
> target system that you are installing, and there is no
> grub2-common-udeb.

Right, but if the conflict in e2fsprogs-udeb prevents the installer
from pulling in an overly new version of e2fsprogs-udeb, that woul be
sufficient, no?

After all, you can install e2fsprogs 1.47, and install a newer version
of grub, and there is no problem --- unless you enable the the
csum-seed function on an already installed system.  And you can't do
that, because you it's an incompat feature.

OTOH, with e2fsprogs 1.46 you can *ready* run the command "tune2fs -O
large_dir /dev/root" on a running system, and then when you reboot,
the system won't come back, because grub will see the large_dir
feature and freak out (unecessarily).  But adding an conflict with
1.46 and grub makes no sense, since it's not _that_ likely that user
will deploy that particular foot-gun.  (It happens with Arch and
Gentoo, but those users tend to be much more adventurous, aided and
abetted by Wiki pages that encourage users to help kernel developers
beta test new features.  :-)

The reason why I really don't want to add the conflicts with e2fsprogs
1.47 is because for people who are using sid, there is aboslutely
nothing wrong with installing e2fsprogs 1.47.  It's only if they use
the installer that sucks in e2fsprogs that there's a problem --- it's
not an inherent conflict with grub per se.  If it were, then we
shouldn't allow installation of e2fsprogs 1.46 because grub2 upstream
is painfully slow in putting out releases --- and that's not
reasonable.

Now, what I *could* do is change the mke2fs.conf in e2fsprogs-udeb to
remove the default use of the csum-seed feature  but is it worth
it?  Hopefully the new version of grub2 will come out soon, at which
point I would then need to revert the hacked mke2fs.conf --- and your
dup'ing this bug should prevent the installer from picking up
e2fsprogs 1.47 until the new version of grub gets released.

I'll note that grub is trying to use an independent implementation of
ext4, as opposed to using the upstream libraries such as libext2 for
ext2/3/4 and libxfs for XFS, combined with grub's very slow release
cycle, means that this kind of thing is going to be happening a *lot*.
It's something that I and Darrick Wong (the XFS maintainer) have
kvetched about a number of times on our weekly conference calls.

For example, recent grub commits that impact XFS that aren't in 2.06
include:

commit a4b495520e4dc41a896a8b916a64eda9970c50ea
Author: Erwan Velu 
Date:   Wed Aug 25 15:31:52 2021 +0200

fs/xfs: Fix unreadable filesystem with v4 superblock

The commit 8b1e5d193 (fs/xfs: Add bigtime incompat feature support)
introduced the bigtime support by adding some features in v3 inodes

Holding back file system development because grub2 uptsream is super
slow doesn't seem like a reasonable way forward, so I really don't
want to set this precedent.

Thinking about this some more, I'd much rather have some kind of
explicit scheme, such as:

   mkfs.xfs -T grub2-dumbdown /dev/XXX

Which disables various new file system features that aren't yet
supported by grub, but which users might very well want to use on
non-boot disks.  

This could be done by doing something as simple as adding to mke2fs.conf:

[fs_types]
 grub2-dumbdown = {
features = ^large_dir,^metadata_csum
 }

Then we could teach the Debian installer to use the file system usage
type "grub2-dumbdown".

Of course, moving forward we'd have to update mke2fs.conf as grub
finally learns new features, and since mke2fs.conf is a config file,
people would have to edit it by hand post-install if they have
customized it in any way.  Unless we add the above in
/etc/mke2fs.conf.d/grub2-dumbdown, and then teach mke2fs to understand
the /etc/mke2fs.conf.d directory...

  - Ted

Bug#1030939: e2fsprogs: generates filesystems that grub-install doesn't recognize

[Sven Joachim]

[Switching over to the cloned bug.]

On 2023-02-09 12:31 -0500, Theodore Ts'o wrote:

> On Thu, Feb 09, 2023 at 06:04:23PM +0100, Sven Joachim wrote:
>>
>> Thanks from me as well :-).  To prevent e2fsprogs from migrating to
>> testing before grub2 and breaking d-i, I am reassigning a copy of this
>> bug back to e2fsprogs.  It may be closed once grub2 2.06-8 enters
>> Bookworm.   Perhaps it would also be a good idea to add a versioned
>> "Breaks: grub2-common (<< 2.06-8~)" to e2fsprogs.
>
> Perhaps we could just add a "Breaks: grub2-common (<< 2.06-8~)" to
> e2fsprogs-udeb?

That is not going to help, because IIUC grub-install is run from the
target system that you are installing, and there is no
grub2-common-udeb.

> After all, it's not that e2fsprogs breaks
> grub2-common, per se, unless the *installer* happens to to use << grub
> 2.06-8~ and e2fsprogs 1.47.0-1.

I was also thinking of the cases where the /boot filesystem is
recreated, e.g. when transferring the installation to a different disk.

Cheers,
   Sven