Re: Immediate fallouts from the big linux changes, and actions

[Roland Clobus]

On 29/12/2023 12:14, Holger Levsen wrote:

On Fri, Dec 29, 2023 at 08:04:19AM +0100, Roland Clobus wrote:

Yesterday I published 3 fixes which got merged really quickly.

[...]

Now Jenkins is running fine for the live sid images again :-)
In the next step, I'll check the trixie live builds.


awesome, thank you!


And today the live-build scripts got support for trixie, for which the 
regular weekly builds start tomorrow.


The hack that I made for live-build, it properly implemented in a MR for 
debian-installer: 
https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/43


That change will automatically deactivate when the newer kernel enters 
trixie.


With kind regards,
Roland Clobus


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Immediate fallouts from the big linux changes, and actions

[Holger Levsen]

On Fri, Dec 29, 2023 at 08:04:19AM +0100, Roland Clobus wrote:
> Yesterday I published 3 fixes which got merged really quickly.
[...]
> Now Jenkins is running fine for the live sid images again :-)
> In the next step, I'll check the trixie live builds.

awesome, thank you!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

20230709: Today was the warmest day on earth in 125,000 years. Today was also
the day with the most planes in the air at one time ever in history. By the time
you read this both of these records have probably been broken.


signature.asc
Description: PGP signature

Re: Immediate fallouts from the big linux changes, and actions

[Timo Jyrinki]

Bastian Blank kirjoitti 24.12.2023 klo 13.46:

- Finally, the armel build fails because it can't find its kernel. The
   marvell flavour seems to have been dropped entirely (at least that's
   how I read the linux changelog for 6.6.3-1~exp1:
 
https://tracker.debian.org/news/1482751/accepted-linux-663-1exp1-source-into-experimental/


Yes, the kermel was broken and the checks for this disabled since quite
some time.


Not that I'd object to removal as such, but to keep documentation 
straight is there some new brokenness that wasn't there at Debian 12 
release time? Kernel 6.1.0 works fine on marvell based devices like QNAP.


The changelog only refers to 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950324 which just 
describes a problem with the default flash layout, fixing of which has 
been automated with a script 
(https://github.com/amouiche/qnap_mtd_resize_for_bullseye).


Debian 11 and Debian 12 have been working fine on these devices.

-Timo

Re: Immediate fallouts from the big linux changes, and actions

[Roland Clobus]

On 25/12/2023 12:43, Holger Levsen wrote:

Hi Roland,

FYI, you might want to read the full thread! ;)

...

On Sun, Dec 24, 2023 at 08:38:57AM +0100, Cyril Brulebois wrote:

This is mostly for information: linux went through a lot of big changes,
initially staged in experimental, and uploaded to unstable as of linux
6.6.8-1.
...> this change also lead to failing live-builds, for trixie (the 
unstable live-builds

are of course also affected but fail earlier due to #1058994), this can be
seen in
https://jenkins.debian.net/job/reproducible_debian_live_build_xfce_trixie/35/console
which failed with

cp: cannot stat 'chroot/debian-installer/build/dest/cdrom/vmlinuz': No such 
file or directory


Yesterday I published 3 fixes which got merged really quickly.
1) The live-build scripts automatically selects the newest kernel 
version, but did not account for '6.6.8' instead of '6.6.8-1'
2) The chroot which is used for building the d-i image does no longer 
contain fakeroot, which is still used by the d-i script (#1058994)
3) Also grub-pc (and dependencies) were no longer present for the 
installer, they needed to be provided as .deb images


Now Jenkins is running fine for the live sid images again :-)

In the next step, I'll check the trixie live builds.

With kind regards,
Roland Clobus


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Immediate fallouts from the big linux changes, and actions

[Holger Levsen]

Hi Roland,

FYI, you might want to read the full thread! ;)

On Sun, Dec 24, 2023 at 08:38:57AM +0100, Cyril Brulebois wrote:
> This is mostly for information: linux went through a lot of big changes,
> initially staged in experimental, and uploaded to unstable as of linux
> 6.6.8-1. These impacted the installer in different ways. This is a
> summary of the obvious impacts, there might be some others down the line
> (just by glancing at hw-detect's massaging of module list by stripping
> the .ko suffix).
> 
> - kernel-image-* packages are now shipping /boot/vmlinuz-* (or
>   /boot/vmlinux-* depending on the arch), instead of just /boot/vmlinuz
>   (respectively /boot/vmlinux).
[...]

this change also lead to failing live-builds, for trixie (the unstable 
live-builds
are of course also affected but fail earlier due to #1058994), this can be
seen in 
https://jenkins.debian.net/job/reproducible_debian_live_build_xfce_trixie/35/console
which failed with

cp: cannot stat 'chroot/debian-installer/build/dest/cdrom/vmlinuz': No such 
file or directory


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Just 100 companies are responsible for 71% of global emissions.
https://www.theguardian.com/sustainable-business/2017/jul/10/100-fossil-fuel-companies-investors-responsible-71-global-emissions-cdp-study-climate-change


signature.asc
Description: PGP signature

Re: Immediate fallouts from the big linux changes, and actions

[Cyril Brulebois]

Bastian Blank  (2023-12-24):
> On Sun, Dec 24, 2023 at 08:38:57AM +0100, Cyril Brulebois wrote:
> > - kernel-image-* packages are now shipping /boot/vmlinuz-* (or
> >   /boot/vmlinux-* depending on the arch), instead of just /boot/vmlinuz
> >   (respectively /boot/vmlinux).
> 
> This was even dependent on architecture.  A lot of secondary
> architectures already had various suffixes.

Looking at kernel-image-* 6.5.13-1 for release archs, I'm only seeing
mips64el (-6.5.0-5-5kc-malta, -6.5.0-5-loongson-3, -6.5.0-5-mips64r2el
suffixes), and its build targets didn't hit the codepath I had to patch
for the general case.

Anyway, since you're not reacting with “oops, we didn't mean to”, I'm
going to count this as a deliberate change.

> > - Modules are compressed now, so the drm workaround needed an updated to
> >   cope with the extra .xz suffix:
> > 
> > https://salsa.debian.org/installer-team/debian-installer/-/commit/bd0f1106f90756e6f4514108492d71e1f2e695ea
> 
> This now hardcodes .xz.  And I'm not really sure what this does and why
> this can't be fixed in the kernel packages.

Because that's a workaround for a problem that's not in the kernel and
that remains to be fully investigated.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Re: Immediate fallouts from the big linux changes, and actions

[Bastian Blank]

On Sun, Dec 24, 2023 at 08:38:57AM +0100, Cyril Brulebois wrote:
> - kernel-image-* packages are now shipping /boot/vmlinuz-* (or
>   /boot/vmlinux-* depending on the arch), instead of just /boot/vmlinuz
>   (respectively /boot/vmlinux).

This was even dependent on architecture.  A lot of secondary
architectures already had various suffixes.

> - Modules are compressed now, so the drm workaround needed an updated to
>   cope with the extra .xz suffix:
> 
> https://salsa.debian.org/installer-team/debian-installer/-/commit/bd0f1106f90756e6f4514108492d71e1f2e695ea

This now hardcodes .xz.  And I'm not really sure what this does and why
this can't be fixed in the kernel packages.

> - Finally, the armel build fails because it can't find its kernel. The
>   marvell flavour seems to have been dropped entirely (at least that's
>   how I read the linux changelog for 6.6.3-1~exp1:
> 
> https://tracker.debian.org/news/1482751/accepted-linux-663-1exp1-source-into-experimental/

Yes, the kermel was broken and the checks for this disabled since quite
some time.

Bastian

-- 
Extreme feminine beauty is always disturbing.
-- Spock, "The Cloud Minders", stardate 5818.4

Re: Re: Immediate fallouts from the big linux changes, and actions

[Luca Boccassi]

> Hm. This same argument can be used to include every storage- and
> filesystem-related
> module into the kernel.  Why don't we have ahci and sd_mod built-in?
> This does
> look quite a bit strange to me to include this stuff..

If you have a block device then there are very little advantages in
terms of workflows, as it's still going through a lot of slow paths.

This on the other hand enables the virtiofsd + qemu direct boot fast
path, which is increasingly popular for fast development cycles, and
changing those kconfigs brings us in line with other distros.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Immediate fallouts from the big linux changes, and actions

[Michael Tokarev]

24.12.2023 11:16, Cyril Brulebois :
...

Searching for information about fuse and virtio, I finally noticed this
entry, which probably explains both fuse's “going away” and ditto for
some (but not all) virtio modules:

 * Set CONFIG_VIRTIO_FS and its dependencies to builtin, to allow building
   images that boot directly to rootfs (skipping the initrd)

as it changes:

 -CONFIG_VIRTIO_PCI=m
 +CONFIG_VIRTIO_PCI=y
 -CONFIG_FUSE_FS=m
 +CONFIG_FUSE_FS=y
 -CONFIG_VIRTIO_FS=m
 +CONFIG_VIRTIO_FS=y


Hm. This same argument can be used to include every storage- and 
filesystem-related
module into the kernel.  Why don't we have ahci and sd_mod built-in?  This does
look quite a bit strange to me to include this stuff..

(This commit is not about big linux changes but about small debian changes ;)

/mjt

Re: Immediate fallouts from the big linux changes, and actions

[Cyril Brulebois]

Cyril Brulebois  (2023-12-24):
> Feel free to reply to this thread if you spot other fallouts.

Another FYI: one thing that I spotted early on (before realizing module
support was entirely broken), when I first diffed the list of files
available in the initramfs, building against testing's or unstable's
linux udebs: a bunch of virtio modules disappeared.

Comparing 6.5.0-5-amd64 and 6.6.8-amd64:

 usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_balloon.ko
 usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_dma_buf.ko
 usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_input.ko
-usr/lib/modules/$ABI/kernel/drivers/virtio/virtio.ko
 usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_mmio.ko
-usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_pci.ko
-usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_pci_legacy_dev.ko
-usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_pci_modern_dev.ko
-usr/lib/modules/$ABI/kernel/drivers/virtio/virtio_ring.ko

A cursory look in linux.git didn't yield an obvious answer:

git diff debian/6.5.13-1..debian/6.6.8-1 -- debian/installer/

Indeed, that's mostly about kernel-versions's getting dropped, fuse
going away, some Depends field being updated, and some i2c-hid-* now
getting excluded in one place, and added conditionally in another one…

Searching for information about fuse and virtio, I finally noticed this
entry, which probably explains both fuse's “going away” and ditto for
some (but not all) virtio modules:

* Set CONFIG_VIRTIO_FS and its dependencies to builtin, to allow building
  images that boot directly to rootfs (skipping the initrd)

as it changes:

-CONFIG_VIRTIO_PCI=m
+CONFIG_VIRTIO_PCI=y
-CONFIG_FUSE_FS=m
+CONFIG_FUSE_FS=y
-CONFIG_VIRTIO_FS=m
+CONFIG_VIRTIO_FS=y

I thought I'd mention the answer by mail since I brought up the topic
earlier (even if only on IRC).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Immediate fallouts from the big linux changes, and actions

[Cyril Brulebois]

Hi,

This is mostly for information: linux went through a lot of big changes,
initially staged in experimental, and uploaded to unstable as of linux
6.6.8-1. These impacted the installer in different ways. This is a
summary of the obvious impacts, there might be some others down the line
(just by glancing at hw-detect's massaging of module list by stripping
the .ko suffix).

- kernel-image-* packages are now shipping /boot/vmlinuz-* (or
  /boot/vmlinux-* depending on the arch), instead of just /boot/vmlinuz
  (respectively /boot/vmlinux). I didn't dive into all the changes but
  I'm not sure whether that was on purpose or a side effect of the big
  changes. In any case, on the short term, since the build system was
  relying on a single file there, that should be moved out of the way,
  I decided to unversion those files to restore the old names. If kernel
  maintainers decide to go back to unversioned files, two reverts should
  be enough:

https://salsa.debian.org/installer-team/debian-installer/-/commit/c17360e632b6f133d4b43a47461944b6203ed730

https://salsa.debian.org/installer-team/debian-installer/-/commit/865c66e497d8e809872bad04aa2269d2a9f889c8

- Modules are compressed now, so the drm workaround needed an updated to
  cope with the extra .xz suffix:

https://salsa.debian.org/installer-team/debian-installer/-/commit/bd0f1106f90756e6f4514108492d71e1f2e695ea

- I'm skipping two commits regarding the new way the ABI is written
  down, that's in line with what was announced initially.

- YIL: If you don't see anything move in the graphical installer (no
  keyboard, no mouse), if your video is garbled, and if you can't load a
  NIC module in the text installer, you might have a totally broken
  support for kernel modules in your installer. This required adjusting
  src:xz-utils to get a liblzma udeb and src:kmod to enable xz support
  in its udebs; “thankfully” the former is NMU-maintained, plus the ftp
  team speed-reviewed it from NEW; thankfully Marco ACK'd my NMU-ing
  kmod so that we unbreak things ASAP. Everything should be sorted out
  in unstable. Plus an extra commit to be extra careful between the
  kmod upload and the cronned daily builds (which were going to start
  before the 01:52Z dinstall):

https://tracker.debian.org/news/1489255/accepted-xz-utils-545-02-source-amd64-all-into-unstable/

https://tracker.debian.org/news/1489268/accepted-xz-utils-545-03-source-into-unstable/
https://salsa.debian.org/md/kmod/-/merge_requests/13

https://tracker.debian.org/news/1489283/accepted-kmod-3020230601-21-source-into-unstable/

https://salsa.debian.org/installer-team/debian-installer/-/commit/c65bdb8128bc68cb566b302feb46267e5c177eb4

- Finally, the armel build fails because it can't find its kernel. The
  marvell flavour seems to have been dropped entirely (at least that's
  how I read the linux changelog for 6.6.3-1~exp1:

https://tracker.debian.org/news/1482751/accepted-linux-663-1exp1-source-into-experimental/

I didn't investigate the last point and I'd appreciate it if porters
could double check that we really want to drop marvell at this point,
and maybe submit (or commit) patches to drop it from the installer:
  https://salsa.debian.org/installer-team/debian-installer

Various netboot and netboot-gtk checks on amd64 seem very positive
after those changes, but I haven't built a netinst image and checked
on actual hardware with the combination of Secure Boot + firmware
packages + reloading of kernel modules. The latter should primarily work
by using module *names* and firmware *paths* as seen in kernel logs, so
having different module *paths* shouldn't make a difference, but that's
just my gut feeling based on my recollection (so nobody should trust
me on this), and seeing how \.ko$ stripping appears in some other files,
I cannot be certain until it's been actually tested.

Feel free to reply to this thread if you spot other fallouts. Of course,
as usual, the best way to report installer bugs is to file them against
the relevant component, X-Debbugs-Cc-ing this list; but leaving an
explicit reference in this thread wouldn't hurt.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature