Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Frédéric Bonnard]

Thank you so much Cyril!

> That being said, we don't have all modules shipped in the installer, and
> some other module might be available and used in the installed system
> (e.g. dedicated driver as opposed to fbdev during a graphical install on
> amd64), which can explain differences. The bootloader configuration
> might be different too.
> 
> As you suggested, this is a regression via a7f0ec26cd, first shipped in
> v6.1.12, which didn't account for the disappearing of-display device
> node in the framebuffer driver. I've tested a trivial patch (see MR)
> and a netboot mini.iso d-i build (similar to daily builds, against udebs
> from unstable) gives me the installer's interface. Available for 15 days
> at: https://people.debian.org/~kibi/bug-1033058/
> 
> I'll open a bug report upstream and submit my patch there. In mainline
> master, another module (not found in 6.1.y) is affected as well, so I
> have this queued up too (but untested):

Excellent!
I was worried seeing the issue coming in rc1 of Bookworm
but also in latest snapshots of Ubuntu 23.04 ( I also
pointed Canonical to this bug and your proposed fix :
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2015994 ).

F.


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Cyril Brulebois]

Cyril Brulebois  (2023-04-12):
> I'll open a bug report upstream and submit my patch there.

Upstream bug:
  https://bugzilla.kernel.org/show_bug.cgi?id=217328

> In mainline master, another module (not found in 6.1.y) is affected as well,
> so I have this queued up too (but untested):
> 
>  drivers/gpu/drm/tiny/ofdrm.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

If you have a way to confirm ofdrm is broken without the patch and fixed
with the patch, please speak up!

Upstream patches:
  https://lore.kernel.org/all/20230412095509.2196162-1-cy...@debamax.com/T/#u


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Cyril Brulebois]

Control: forwarded -1 
https://salsa.debian.org/kernel-team/linux/-/merge_requests/695
Control: reassign -1 src:linux 6.1.12-1
Control: severity -1 important
Control: tag -1 patch

Hello Frédéric,

Frédéric Bonnard  (2023-03-23):
> ---
> $ qemu-system-ppc64le -nographic -nodefaults -serial mon:stdio -cdrom 
> /tmp/mini.iso
> ---
> here we just start a ppc64le vm with no graphic card, and boot the iso.
> You should reach the debian installer menu on the serial console in your
> terminal (if you want to go further, provide a disk file for example).
> 
> But if you try a vm with a graphic card like so :
> ---
> qemu-system-ppc64le  -cdrom /tmp/mini.iso
> ---

ACK on both, thanks! That was indeed super easy and super useful.

> I see the installer media gets build from a kernel-image udeb which I
> guess provides a different kernel runtime env from the machine installed
> with linux-image on disk (which works).

It's slightly more complicated than that, as both vmlinuz are actually
identical:

$ dpkg -x kernel-image-6.1.0-7-powerpc64le-di_6.1.20-2_ppc64el.udeb foo
$ dpkg -x linux-image-6.1.0-7-powerpc64le_6.1.20-2_ppc64el.deb bar
$ cmp foo/boot/vmlinux bar/boot/vmlinux-6.1.0-7-powerpc64le; echo $?
0

That being said, we don't have all modules shipped in the installer, and
some other module might be available and used in the installed system
(e.g. dedicated driver as opposed to fbdev during a graphical install on
amd64), which can explain differences. The bootloader configuration
might be different too.


As you suggested, this is a regression via a7f0ec26cd, first shipped in
v6.1.12, which didn't account for the disappearing of-display device
node in the framebuffer driver. I've tested a trivial patch (see MR)
and a netboot mini.iso d-i build (similar to daily builds, against udebs
from unstable) gives me the installer's interface. Available for 15 days
at: https://people.debian.org/~kibi/bug-1033058/

I'll open a bug report upstream and submit my patch there. In mainline
master, another module (not found in 6.1.y) is affected as well, so I
have this queued up too (but untested):

 drivers/gpu/drm/tiny/ofdrm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

in addition to:

 drivers/video/fbdev/offb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Frédéric Bonnard]

Hi Cyril!

> Could you please guide me into reproducing this issue in QEMU? #987368
> had hints, but at least the openpower.xyz part no longer works (it's no
> longer resolving).

much simpler actually because in the current case, we won't emulate a
full baremetal PowerNV machine.
For example, using this mini.iso from 
http://d-i.debian.org/daily-images/ppc64el/daily/netboot/mini.iso
you can just try those 2 cases  (from an x86 machine or other) :

---
$ qemu-system-ppc64le -nographic -nodefaults -serial mon:stdio -cdrom 
/tmp/mini.iso
---
here we just start a ppc64le vm with no graphic card, and boot the iso.
You should reach the debian installer menu on the serial console in your
terminal (if you want to go further, provide a disk file for example).

But if you try a vm with a graphic card like so :
---
qemu-system-ppc64le  -cdrom /tmp/mini.iso
---

You'll see you won't have the d-i menu displayed on the default
graphical output.
Not sure the pointed patch is at fault, but at least it emphasis some
issue in the installer environment in the context of a framebuffer
display in qemu.
I see the installer media gets build from a kernel-image udeb which I
guess provides a different kernel runtime env from the machine installed
with linux-image on disk (which works).


F.


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Cyril Brulebois]

Hi Frédéric,

Frédéric Bonnard  (2023-03-22):
> Another important detail ... I only get that behavior in qemu in
> graphical mode. On LPARs there is no issue. I didn't try on baremetal
> so far.

Could you please guide me into reproducing this issue in QEMU? #987368
had hints, but at least the openpower.xyz part no longer works (it's no
longer resolving).


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Frédéric Bonnard]

Thanks Cyril.
Another important detail ... I only get that behavior in qemu in
graphical mode.
On LPARs there is no issue. I didn't try on baremetal so far.

F.

On Tue, 21 Mar 2023 17:44:49 +0100 Cyril Brulebois  wrote:
> Frédéric Bonnard  (2023-03-17):
> > > It would be helpful to confirm which exact kernel version is getting
> > > used in both cases (last 6.1.0-4 working, and first 6.1.0-5 not
> > > working), then look at the changes between both versions, in src:linux
> > > (and resulting udebs).
> > 
> > From linux changelog (
> > https://tracker.debian.org/media/packages/l/linux/changelog-6.1.15-1 )
> > and http://snapshot.debian.org/package/linux/,
> > I see :
> > 6.1.11-1 -> Bump ABI to 4 
> > http://snapshot.debian.org/archive/debian/20230211T151657Z/
> > 6.1.12-1 -> Bump ABI to 5 
> > http://snapshot.debian.org/archive/debian/20230217T033139Z/
> > 
> > and tested from those (rebuilding the same d-i source with the 2 kernels
> > from snapshot.d.o . 6.1.12-1 clearly doesn't work.
> > I had already have a look at the kernel changelog but missed that one :
> > ---
> > linux (6.1.12-1) unstable; urgency=medium
> > ...
> > - of: Make OF framebuffer device names unique
> > ...
> > ---
> > 
> > I recompiled the kernel deb source without that one (
> > https://github.com/torvalds/linux/commit/241d2fb56a18473af5f2ff0d512992a996eb64dd.patch
> > ) and the mini.iso actually made it to the menu... and given the
> > behaviour, a framebuffer actually makes sense.
> > 
> > Now, that patch does not harm when the kernel is installed on disk.
> > But it does in the installer...
> 
> Adding the kernel team to the loop: d-i regression on ppc64el.
> 
> 
> Cheers,
> -- 
> Cyril Brulebois (k...@debian.org)
> D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Cyril Brulebois]

Frédéric Bonnard  (2023-03-17):
> > It would be helpful to confirm which exact kernel version is getting
> > used in both cases (last 6.1.0-4 working, and first 6.1.0-5 not
> > working), then look at the changes between both versions, in src:linux
> > (and resulting udebs).
> 
> From linux changelog (
> https://tracker.debian.org/media/packages/l/linux/changelog-6.1.15-1 )
> and http://snapshot.debian.org/package/linux/,
> I see :
> 6.1.11-1 -> Bump ABI to 4 
> http://snapshot.debian.org/archive/debian/20230211T151657Z/
> 6.1.12-1 -> Bump ABI to 5 
> http://snapshot.debian.org/archive/debian/20230217T033139Z/
> 
> and tested from those (rebuilding the same d-i source with the 2 kernels
> from snapshot.d.o . 6.1.12-1 clearly doesn't work.
> I had already have a look at the kernel changelog but missed that one :
> ---
> linux (6.1.12-1) unstable; urgency=medium
> ...
> - of: Make OF framebuffer device names unique
> ...
> ---
> 
> I recompiled the kernel deb source without that one (
> https://github.com/torvalds/linux/commit/241d2fb56a18473af5f2ff0d512992a996eb64dd.patch
> ) and the mini.iso actually made it to the menu... and given the
> behaviour, a framebuffer actually makes sense.
> 
> Now, that patch does not harm when the kernel is installed on disk.
> But it does in the installer...

Adding the kernel team to the loop: d-i regression on ppc64el.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Frédéric Bonnard]

Ok found it...

> (nothing attached)

oops! .. not that much to display though (I also tried to remove the
"quiet" boot option, but it didn't change anything, it does not display
anything before getting in debian's init)

> > same happens with 
> > https://cdimage.debian.org/cdimage/weekly-builds/ppc64el/iso-cd/debian-testing-ppc64el-netinst.iso
> > I tried rebuilding d-i with kernel 6.1.0-4 and it seems to work (6.1.0-6
> > fails too).
> > Now why am I opening this bug against d-i ? If I run the d-i with
> > 6.1.0-4, I complete the installation and then runtime kernel of the
> > machine that gets installed is 6.1.0-6 and when the machine is
> > rebooted, kernel 6.1.0-6 boots fine ...
> > So I'm not sure on which side is the issue :)
> > Any clue what could be happening ?
> 
> It would be helpful to confirm which exact kernel version is getting
> used in both cases (last 6.1.0-4 working, and first 6.1.0-5 not
> working), then look at the changes between both versions, in src:linux
> (and resulting udebs).

From linux changelog (
https://tracker.debian.org/media/packages/l/linux/changelog-6.1.15-1 )
and http://snapshot.debian.org/package/linux/,
I see :
6.1.11-1 -> Bump ABI to 4 
http://snapshot.debian.org/archive/debian/20230211T151657Z/
6.1.12-1 -> Bump ABI to 5 
http://snapshot.debian.org/archive/debian/20230217T033139Z/

and tested from those (rebuilding the same d-i source with the 2 kernels
from snapshot.d.o . 6.1.12-1 clearly doesn't work.
I had already have a look at the kernel changelog but missed that one :
---
linux (6.1.12-1) unstable; urgency=medium
...
- of: Make OF framebuffer device names unique
...
---

I recompiled the kernel deb source without that one (
https://github.com/torvalds/linux/commit/241d2fb56a18473af5f2ff0d512992a996eb64dd.patch
) and the mini.iso actually made it to the menu... and given the
behaviour, a framebuffer actually makes sense.

Now, that patch does not harm when the kernel is installed on disk.
But it does in the installer...

F.


> 
> ISTR trying to boot ppc64el under QEMU requires many manual steps, but
> I'll try and figure that out, once I'm done with other topics.
> 
> 
> Cheers,
> -- 
> Cyril Brulebois (k...@debian.org)
> D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Cyril Brulebois]

Hello Frédéric,

Frédéric Bonnard  (2023-03-16):
> since debian-installer (20230217) uses kernel 6.1.0-5 in mini.iso, the
> kernel hangs before I have the installer menu (see attached .png) ;

(nothing attached)

> same happens with 
> https://cdimage.debian.org/cdimage/weekly-builds/ppc64el/iso-cd/debian-testing-ppc64el-netinst.iso
> I tried rebuilding d-i with kernel 6.1.0-4 and it seems to work (6.1.0-6
> fails too).
> Now why am I opening this bug against d-i ? If I run the d-i with
> 6.1.0-4, I complete the installation and then runtime kernel of the
> machine that gets installed is 6.1.0-6 and when the machine is
> rebooted, kernel 6.1.0-6 boots fine ...
> So I'm not sure on which side is the issue :)
> Any clue what could be happening ?

It would be helpful to confirm which exact kernel version is getting
used in both cases (last 6.1.0-4 working, and first 6.1.0-5 not
working), then look at the changes between both versions, in src:linux
(and resulting udebs).

ISTR trying to boot ppc64el under QEMU requires many manual steps, but
I'll try and figure that out, once I'm done with other topics.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1033058: Booting mini.iso : kernel hangs on ppc64el

[Frédéric Bonnard]

Package: installation-reports

Boot method: CD
Image version: http://d-i.debian.org/daily-images/ppc64el/daily/netboot/mini.iso

Hi!
since debian-installer (20230217) uses kernel 6.1.0-5 in mini.iso, the kernel 
hangs
before I have the installer menu (see attached .png) ; same happens with 
https://cdimage.debian.org/cdimage/weekly-builds/ppc64el/iso-cd/debian-testing-ppc64el-netinst.iso
I tried rebuilding d-i with kernel 6.1.0-4 and it seems to work (6.1.0-6
fails too).
Now why am I opening this bug against d-i ? If I run the d-i with
6.1.0-4, I complete the installation and then runtime kernel of the machine that
gets installed is 6.1.0-6 and when the machine is rebooted, kernel
6.1.0-6 boots fine ...
So I'm not sure on which side is the issue :)
Any clue what could be happening ?
Thanks,

F.


signature.asc
Description: PGP signature