Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Fri, 27 Oct 2023 20:09:57 +0200 Bastian Blank wrote: > On Fri, Oct 27, 2023 at 01:04:00PM +0300, Adrian Bunk wrote: > > > Sadly in Debian there is no way to make that happen. Think for example > > > about bin-nmu. > > Could you give a complete list of problems? > > There are at least those problems: > - Bin-nmu can't change binary package names. > - There is no way to embed a Debian version into a Debian package name > without loss. (Okay, I think we would only need ~, all the othe > characters are allowed) > > Bastian This whole discussion to me even more strongly suggests that the current way of trying to satisfy these requirements on kernels availability shouldn't really be delegated only to packages content. And that's before we get into ESP size issues. So the idea to split these apart - packages install to /usr/, and a separate mechanism manages what is available under /boot/ and for how long, depending also on how much space there is - seems more and more the right way forward. Just my 2c. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Fri, Oct 27, 2023 at 01:04:00PM +0300, Adrian Bunk wrote: > > Sadly in Debian there is no way to make that happen. Think for example > > about bin-nmu. > Could you give a complete list of problems? There are at least those problems: - Bin-nmu can't change binary package names. - There is no way to embed a Debian version into a Debian package name without loss. (Okay, I think we would only need ~, all the othe characters are allowed) Bastian -- Change is the essential process of all existence. -- Spock, "Let That Be Your Last Battlefield", stardate 5730.2
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Fri, Oct 27, 2023 at 10:55:48AM +0200, Bastian Blank wrote: > On Fri, Oct 27, 2023 at 08:43:46AM +0200, Julian Andres Klode wrote: > > > > ## Image packages contains more version info > > > > > > > > Example: linux-image-6.5.3-cloud-arm64 > > > > > > > It will not longer be possible to reliably derive the package name from > > > > kernel release (see above), as both values are not really related > > > > anymore. > > > > This package name seems to be missing the Debian release, which is > > mandatory to ensure that you can install 6.5.3-2 and 6.5.3-1 in > > parallel, which again is mandatory. That's not something we can > > compromise on; it's absolutely vital that > > Sadly in Debian there is no way to make that happen. Think for example > about bin-nmu. >... Could you give a complete list of problems? The only problem that comes immediately into my mind is the problem that uploads with new binary packages are going into NEW for no good reason. If this is the only problem, I wouldn't worry too much since I'd be optimistic a GR to change this will be sucessful. > Bastian cu Adrian
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Fri, Oct 27, 2023 at 08:43:46AM +0200, Julian Andres Klode wrote: > > > ## Image packages contains more version info > > > > > > Example: linux-image-6.5.3-cloud-arm64 > > > > > It will not longer be possible to reliably derive the package name from > > > kernel release (see above), as both values are not really related > > > anymore. > > This package name seems to be missing the Debian release, which is > mandatory to ensure that you can install 6.5.3-2 and 6.5.3-1 in > parallel, which again is mandatory. That's not something we can > compromise on; it's absolutely vital that Sadly in Debian there is no way to make that happen. Think for example about bin-nmu. In the end we can approximate it in testing (usually not multiple releases are migrated, but bin-nmu might show up), stable (where usualy new upstream releases go in) and security (by uploading as 6.6.1, 6.6.1+1, 6.6.1+2, yes this is a hack, but it reduces the complexity of the whole system). Right now I simply don't see a way to not have multiple releases within the same package, which override each other. > - you can revert to the kernel you last booted succesfully, i.e. 6.5.3-1 > if 6.5.3-2 is broken (think toolchain broke or something on buildds) You can revert to 6.5.2, which is a separate package. Or 6.4. > - the currently booted kernel is not impacted. This means it must be > able to load additional modules. Some platforms even require > additional modules to be loaded to reboot, I've seen this on > laptops. Could you provide an example? Then we have to find another way to make sure modules survive unrelated to what dpkg does. Even right now there is no guarante you can load modules from a different version at all. > Needless to say I do not believe that uname -s is necessarily a > single word. Please provide an example. [ Snipped the rest for now, will come back later ] Bastian -- Where there's no emotion, there's no motive for violence. -- Spock, "Dagger of the Mind", stardate 2715.1
Bug#1040901: Upcoming changes to Debian Linux kernel packages
OK, it seems my original email got lost somewhere in tech hickups, it's possible the kernel crashed before sending the email, AMD just crashes once or twice a day. So I'm writing this email a bit in a hurry, so it's not quite as thought out as the last one weeks ago, but yesterday's email was pretty concerning. We should not rush this. we need to be realistic that this will take a couple of months to sort out a new scheme, don't expect this to be done before the end of the year, and that seems to be very optimistic. On Sat, Oct 07, 2023 at 04:53:33PM +0200, Bastian Blank wrote: > Moin > > On Sun, Sep 24, 2023 at 03:01:51PM +0200, Bastian Blank wrote: > > ## Kernel modules will be signed with an ephemeral key > > This is now > https://salsa.debian.org/kernel-team/linux/-/merge_requests/607. > > > ## Image packages contains more version info > > > > Example: linux-image-6.5.3-cloud-arm64 > > > It will not longer be possible to reliably derive the package name from > > kernel release (see above), as both values are not really related > > anymore. This package name seems to be missing the Debian release, which is mandatory to ensure that you can install 6.5.3-2 and 6.5.3-1 in parallel, which again is mandatory. That's not something we can compromise on; it's absolutely vital that - you can revert to the kernel you last booted succesfully, i.e. 6.5.3-1 if 6.5.3-2 is broken (think toolchain broke or something on buildds) - the currently booted kernel is not impacted. This means it must be able to load additional modules. Some platforms even require additional modules to be loaded to reboot, I've seen this on laptops. It is fine to say "you must reboot", but it is not fine to just break the running system until you do. > > I missed that apt does something similar (maintainers cced). It wants > to see if a particular package matches the current kernel to make the > autoremove prevention work. That lookup is quite a hard problem. > > What should work: We define a new control field. It contains both the > kernel name and a version prefix. > > Example: > - Linux 6.6 (would match 6.6-1, 6.6.1-1) > - Linux 6.6.3 (would match 6.6.3-1, but not 6.6.3+2-1) > - Linux 6.6.3+2 > > The algorithm would be something like this: > - Check $(uname -s) against the first word. Otherwise completely > ignore. Needless to say I do not believe that uname -s is necessarily a single word. > - Check if $(uname -r) matches the version prefix in this field. Mark > as keep if it matches. > - Aggregate packages by version prefix. > - Sort as version, keep newest two(?). I don't really care about the ordering, I want to order by package version. I need to identify: * Which kernel image package is currently booted. Hence I need to go from the uname -r to exactly one installed kernel image package that acts as the key package to determine if a kernel version is installed. * What are other installed kernel image packages * Given a uname / kernel image package, which other kernel packages belong to it This allows me to keep the currently installed kernels around. The flip side of the coin is the code in APT that actually autoremoves kernels: It needs to identify, given a list of kernel images to keep which other kernel packages can be removed. Optimally it can still remove headers for 6.6 if you don't even have 6.6 images installed anymore and _somehow_ pulled them in. Essentially I want to order kernel image packages by version, and keep the currently booted and the latest one, and the 2nd latest if currently booted == latest one, and then remove any other versioned kernel package not related to them. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Thu, Oct 26, 2023 at 01:36:50PM +0200, Bastian Blank wrote: > On Fri, Oct 20, 2023 at 05:54:23PM +0200, Bastian Blank wrote: > > Or would it be easier to re-use normal dependency resolving, like: > > Kernel-Provides: linux (>> 6.6.1~), linux (<< 6.6.1.) > > This would allow full flexibility and re-uses existing code to check > > such definitions. > > Okay, noone complained, so it looks like this should work. > No no it doesn't. Did nobody read my email? Did I even send my email? I don't know what the hell is going on here, but I spend an hour writing an email discussing the options outlined before this ridiculousness and the requirements, but I can't find it in my notmuch. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Fri, Oct 20, 2023 at 05:54:23PM +0200, Bastian Blank wrote: > Or would it be easier to re-use normal dependency resolving, like: > Kernel-Provides: linux (>> 6.6.1~), linux (<< 6.6.1.) > This would allow full flexibility and re-uses existing code to check > such definitions. Okay, noone complained, so it looks like this should work. Regards, Bastian -- Too much of anything, even love, isn't necessarily a good thing. -- Kirk, "The Trouble with Tribbles", stardate 4525.6
Bug#1040901: Upcoming changes to Debian Linux kernel packages
[ Removing some lists ] On Sat, Oct 07, 2023 at 04:53:33PM +0200, Bastian Blank wrote: > > ## Image packages contains more version info > > > > Example: linux-image-6.5.3-cloud-arm64 > > > It will not longer be possible to reliably derive the package name from > > kernel release (see above), as both values are not really related > > anymore. > What should work: We define a new control field. It contains both the > kernel name and a version prefix. Or would it be easier to re-use normal dependency resolving, like: Kernel-Provides: linux (>> 6.6.1~), linux (<< 6.6.1.) This would allow full flexibility and re-uses existing code to check such definitions. Regards, Bastian -- Women professionals do tend to over-compensate. -- Dr. Elizabeth Dehaver, "Where No Man Has Gone Before", stardate 1312.9.
Bug#1040901: Upcoming changes to Debian Linux kernel packages
Moin On Sun, Sep 24, 2023 at 03:01:51PM +0200, Bastian Blank wrote: > ## Kernel modules will be signed with an ephemeral key This is now https://salsa.debian.org/kernel-team/linux/-/merge_requests/607. > ## Image packages contains more version info > > Example: linux-image-6.5.3-cloud-arm64 > It will not longer be possible to reliably derive the package name from > kernel release (see above), as both values are not really related > anymore. I missed that apt does something similar (maintainers cced). It wants to see if a particular package matches the current kernel to make the autoremove prevention work. That lookup is quite a hard problem. What should work: We define a new control field. It contains both the kernel name and a version prefix. Example: - Linux 6.6 (would match 6.6-1, 6.6.1-1) - Linux 6.6.3 (would match 6.6.3-1, but not 6.6.3+2-1) - Linux 6.6.3+2 The algorithm would be something like this: - Check $(uname -s) against the first word. Otherwise completely ignore. - Check if $(uname -r) matches the version prefix in this field. Mark as keep if it matches. - Aggregate packages by version prefix. - Sort as version, keep newest two(?). This means: - Images and headers are always kept with the same versions. - Different images (-arm64, -rt-arm64) are always kept together. Counter proposal: Use see the kernel release as debian version and match on the upstream version. But then we need to re-define what we put into the kernel release field. In 6.6.1-1-cloud-arm64, the upstream version is 6.6.1-1-cloud, not 6.6.1 as we would need. We could of course change that to: 6.6.1-1~cloud+arm64. That should always sort correctly in regard of the package version. > ## Header and tool packages will not longer contain version This is obsolete with the counter proposal of a meta package that always pulls in image and headers of the same version. Regards, Bastian -- Without followers, evil cannot spread. -- Spock, "And The Children Shall Lead", stardate 5029.5
Bug#1040901: Upcoming changes to Debian Linux kernel packages
Hi On Sun, Sep 24, 2023 at 06:05:09PM +0200, Ben Hutchings wrote: > > Multiple uploads of the same upstream version will have > > the same package name, but those rarely happens. > Those happen fairly often for urgent security updates. We could encode that in the upstream version. Aka to have co-installable packages for security updates we do: - 6.6.1-1 - 6.6.1+1-1 - 6.6.1+2-1 This allows for easy semantic, aka we only care for package names about the upstream part of the version, which then needs to follow a certain syntax. Regards, Bastian -- A Vulcan can no sooner be disloyal than he can exist without breathing. -- Kirk, "The Menagerie", stardate 3012.4
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Mon, Sep 25, 2023 at 02:03:35AM +0200, Bastian Blank wrote: > The current way does not work. See all the bug reports about > uninstallable packages and what not with dkms. > > To build modules against version x, you'll need to install version x of > the headers, not x-1 or x+1. This currently works most of the time, but > is by far stable. And if you already have to search for the specific > version, it does not matter if you might have the ability to install > multiple at the same time, the archive will in any case only contain one > version at a time. So you want to change something from where dkms works 99.99% of the time to 0% of the time? Having multiple kernel versions and being able to build for them is a very normal thing to do that almost always works. Why break that? Sure there are some bugs for when dkms has issues, but you don't get bug reports for all the cases where it is working. I am having trouble understanding what the problem was that you are attempting to fix, but so far the cure sounds far worse than the disease. -- Len Sorensen
Bug#1040901: Upcoming changes to Debian Linux kernel packages
Hi Ben On Sun, Sep 24, 2023 at 06:05:09PM +0200, Ben Hutchings wrote: > On Sun, 2023-09-24 at 15:01 +0200, Bastian Blank wrote: > > The same upstream version in testing and backports will have the same > > package name. > This is not OK, because they will be incompatible on architectures > supporting SB (and sometimes incompatible on others due to compiler > differences or required config changes). I don't know what you are talking about. Those two packages have different versions, so won't contain anything compatible. It is the same between 1.2.3-1 vs 1.2.3-2 and 1.2.3-1~bpo13+1 vs 1.2.3-1. > If someone upgrades from stable + backports to testing, and has OOT > modules: > - With DKMS, will a rebuild be triggered if the linux-image package > name doesn't change? The same as with a normal package upgrade, it will rebuilt against the new version. It just runs into the same version skew as everything else. > - With module-assistant, the new linux-image package will satisfy > dependencies of the old modules even though they are incompatible. No, the two linux-image packages have different versions, so won't satisfy the dependencies. > > Multiple uploads of the same upstream version will have > > the same package name, but those rarely happens. > Those happen fairly often for urgent security updates. Right. Maybe we need a manual or automatic override for this, we can do a lot of things. > > It will not longer be possible to reliably derive the package name from > > kernel release (see above), as both values are not really related > > anymore. > Given all the drawbacks, I don't see the benefit of decoupling package > names from release strings. > In the same way that shared library packages must be renamed for every > backward-incompatible ABI changes, I believe we should keep doing this > for linux-image packages. Noted, but I don't see a way to do that. We can't map versions cleanly into package names. We have binNMU, which can't change package names, so will already in violation of that. And we already don't do that, see that huge version ignore list. Also the ABI in shared libraries is to have two independent updateable identities. Nothing is true in case of the kernel, it will just break on every update of either side, which would be the equivalent of a = dependency. So no, shared libraries are not a good comparison. > > ## Header and tool packages will not longer contain version > > > > The headers packages will not longer include the version. It won't be > > reliably possible to derive the package name anyway from the running > > kernel. > > > > This means that only headers of one single version can be available on > > the system at one time. This might be a bit inconvinient for dkms, as > > it can't longer build modules for multiple versions. > > > > But we too often have the problem that image and headers go out of sync > > and then you can't find the correct ones anyway. > > > > Example: linux-headers-cloud-arm64 > > This is all downside with no justification given. Please explain what > the benefit is. The current way does not work. See all the bug reports about uninstallable packages and what not with dkms. To build modules against version x, you'll need to install version x of the headers, not x-1 or x+1. This currently works most of the time, but is by far stable. And if you already have to search for the specific version, it does not matter if you might have the ability to install multiple at the same time, the archive will in any case only contain one version at a time. IMHO the only way around would be to install image and headers always in one piece for those who want to build own modules against. But this will require further restructuring, as the headers for this then need to be built from linux-signed-* and arch-any to be without skew. And use proper dependencies so everything is installed with the same version always. Aka something like that: Package: linux-image-cloud-arm64 Depends: linux-image-1.2.3-cloud-arm64 (= 1.2.3-1) Package: linux-modules-thirdparty-cloud-arm64 Depends: linux-image-1.2.3-cloud-arm64 (= 1.2.3-1), linux-modules-1.2.3-cloud-arm64 (= 1.2.3-1), linux-headers-1.2.3-cloud-arm64 (= 1.2.3-1) Package: linux-image-1.2.3-cloud-arm64 Depends: linux-modules-1.2.3-cloud-arm64 (= 1.2.3-1) Package: linux-headers-1.2.3-cloud-arm64 Depends: linux-modules-1.2.3-cloud-arm64 (= 1.2.3-1) Package: linux-modules-1.2.3-cloud-arm64 However doesn't building modules currently need the vmlinux as well? Which would not be fullfiled anyway right now. > > ## Installer packages will not longer contain too much version > > > > The installer can only ever handle one version of kernel. Also it got > > an internal mechanism to detect which packages belong together > > (the Kernel-Version control entry). So we have no need to rename them > > and force a matching change in d-i itself just because a new kernel > > exists. So it
Bug#1040901: Upcoming changes to Debian Linux kernel packages
On Sun, 2023-09-24 at 15:01 +0200, Bastian Blank wrote: [...] > ## Kernel modules will be signed with an ephemeral key > > The modules will not longer be signed using the Secure Boot CA like the > EFI kernel image itself. Instead a key will be created during the build > and thrown away after. > > Yes, this will make the build unreproducible, but no better solution > currently exists. There are some plans, but no-one is working on them. > If a suitable replacement shows up, we can always switch to that > solution. Builds for the architectures involved are already unreproducible due to inconsistent generation of BTF in both the kernel and modules. Additionally, my "plan" would also get rid of signing modules with the Secure Boot CA, so I'm not going to object to this. [...] > ## Image packages contains more version info > > By renaming the kernel packages we try to make several kernels > installable at the same time. In contrast to rpm, where you can have > the same package installed multiple times in different versions, dpkg > only supports a single one at the same time. So the co-installable > versions needs to have different package names. > > The packages will include the full upstream version. There exists the > exception of devel builds and uploads to experimental, wich will contain > even less of the version, to avoid new names in that cases. > > Example: linux-image-6.5.3-cloud-arm64 > > There are some drawbacks. > > The same upstream version in testing and backports will have the same > package name. This is not OK, because they will be incompatible on architectures supporting SB (and sometimes incompatible on others due to compiler differences or required config changes). If someone upgrades from stable + backports to testing, and has OOT modules: - With DKMS, will a rebuild be triggered if the linux-image package name doesn't change? - With module-assistant, the new linux-image package will satisfy dependencies of the old modules even though they are incompatible. > Multiple uploads of the same upstream version will have > the same package name, but those rarely happens. Those happen fairly often for urgent security updates. > Those packages will > not be compatible and a reboot is necessary to be able to load modules > again. > > It will not longer be possible to reliably derive the package name from > kernel release (see above), as both values are not really related > anymore. Given all the drawbacks, I don't see the benefit of decoupling package names from release strings. In the same way that shared library packages must be renamed for every backward-incompatible ABI changes, I believe we should keep doing this for linux-image packages. > ## Header and tool packages will not longer contain version > > The headers packages will not longer include the version. It won't be > reliably possible to derive the package name anyway from the running > kernel. > > This means that only headers of one single version can be available on > the system at one time. This might be a bit inconvinient for dkms, as > it can't longer build modules for multiple versions. > > But we too often have the problem that image and headers go out of sync > and then you can't find the correct ones anyway. > > Example: linux-headers-cloud-arm64 This is all downside with no justification given. Please explain what the benefit is. > ## Installer packages will not longer contain too much version > > The installer can only ever handle one version of kernel. Also it got > an internal mechanism to detect which packages belong together > (the Kernel-Version control entry). So we have no need to rename them > and force a matching change in d-i itself just because a new kernel > exists. So it will not longer contain the full version in the package > names if not needed. [...] In the installer, netboot images break every time the kernel ABI is bumped. I think there's a specific check and error message for this, but I'm not exactly sure. It should be verified that this detection will work the way you expect, so that the error message doesn't change and create a support burden for the installer team. Currently kernel-wedge generates the udeb package names and would need to add an option to leave out the version part of the names. I'm happy to work on that once we have an agreement for what to do. Ben. -- Ben Hutchings It is easier to change the specification to fit the program than vice versa. signature.asc Description: This is a digitally signed message part
Bug#1040901: Upcoming changes to Debian Linux kernel packages
Hi folks Debian currently does Secure Boot signing using a shim chained to the Microsoft key. This use requires that we follow certain rules. And one of the recent changes to those rules state that our method of signing kernel modules also with the same key will not be allowed anymore. Some information are in #1040901. We could just do the minimal change, sign the modules a different way and let users walk into authenticated failures and other scary error messages. Or we could change the existing ABI setting on every upload, creating a new set of binary packages. But maybe we can enhance the user experience a bit, by reducing the chance of scarry errors, but with the chance of simple errors like "you need to reboot". So let's do some more changes and hopefully don't break the user experience too much. The planned changes are discussed in more detail. ## Kernel modules will be signed with an ephemeral key The modules will not longer be signed using the Secure Boot CA like the EFI kernel image itself. Instead a key will be created during the build and thrown away after. Yes, this will make the build unreproducible, but no better solution currently exists. There are some plans, but no-one is working on them. If a suitable replacement shows up, we can always switch to that solution. ## Kernel release value includes complete Debian version The kernel release is what "uname -r" shows, and how modules are organized in /lib/modules. This value will include the complete version of the binary package, so even binNMU will somehow work. This will make sure the value changes with every upload and modules will not be compatible already from that check. Example: 6.5.3-2+b2-cloud-arm64 ## Image packages contains more version info By renaming the kernel packages we try to make several kernels installable at the same time. In contrast to rpm, where you can have the same package installed multiple times in different versions, dpkg only supports a single one at the same time. So the co-installable versions needs to have different package names. The packages will include the full upstream version. There exists the exception of devel builds and uploads to experimental, wich will contain even less of the version, to avoid new names in that cases. Example: linux-image-6.5.3-cloud-arm64 There are some drawbacks. The same upstream version in testing and backports will have the same package name. Multiple uploads of the same upstream version will have the same package name, but those rarely happens. Those packages will not be compatible and a reboot is necessary to be able to load modules again. It will not longer be possible to reliably derive the package name from kernel release (see above), as both values are not really related anymore. ## Header and tool packages will not longer contain version The headers packages will not longer include the version. It won't be reliably possible to derive the package name anyway from the running kernel. This means that only headers of one single version can be available on the system at one time. This might be a bit inconvinient for dkms, as it can't longer build modules for multiple versions. But we too often have the problem that image and headers go out of sync and then you can't find the correct ones anyway. Example: linux-headers-cloud-arm64 ## Installer packages will not longer contain too much version The installer can only ever handle one version of kernel. Also it got an internal mechanism to detect which packages belong together (the Kernel-Version control entry). So we have no need to rename them and force a matching change in d-i itself just because a new kernel exists. So it will not longer contain the full version in the package names if not needed. ## Further work The changes outlined here try to avoid changes to the initramfs protocol, aka /etc/kernel/. There are larger change is cooking somehow, see https://lists.debian.org/msgid-search/y2gbkyerb10ky...@shell.thinkmo.de Regards, Bastian -- You! What PLANET is this! -- McCoy, "The City on the Edge of Forever", stardate 3134.0
