Bug#1058638: grub-installer: boot fails: $bootdev empty [BIOS/MBR, Vista, 2 disks]: Installing grub on ''

[Tj]

Source: grub-installer
Version: 1.194
Followup-For: Bug #1058638

I fetched the proposed patch and applied it to grub-installer inside the
installer and can confirm it solved this issue.

syslog now shows "grub-installer: info: Installing grub on '/dev/vdb'
and examining sector 0 of the disk image file shows the expected
boot-strap code and a reboot correctly starts Debian:

hexdump -n 512 -C disk2-debian-12.bin
  eb 63 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.c..|
0010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.!..|
0020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |8.uu|
0030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.|...t..|
0040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.|.|
0050  00 00 00 00 00 00 00 00  00 00 00 80 01 00 00 00  ||
0060  00 00 00 00 ff fa 90 90  f6 c2 80 74 05 f6 c2 70  |...t...p|
0070  74 02 b2 80 ea 79 7c 00  00 31 c0 8e d8 8e d0 bc  |ty|..1..|
0080  00 20 fb a0 64 7c 3c ff  74 02 88 c2 52 be 80 7d  |. ..d|<.t...R..}|
0090  e8 17 01 be 05 7c b4 41  bb aa 55 cd 13 5a 52 72  |.|.A..U..ZRr|
00a0  3d 81 fb 55 aa 75 37 83  e1 01 74 32 31 c0 89 44  |=..U.u7...t21..D|
00b0  04 40 88 44 ff 89 44 02  c7 04 10 00 66 8b 1e 5c  |.@.D..D.f..\|
00c0  7c 66 89 5c 08 66 8b 1e  60 7c 66 89 5c 0c c7 44  ||f.\.f..`|f.\..D|
00d0  06 00 70 b4 42 cd 13 72  05 bb 00 70 eb 76 b4 08  |..p.B..r...p.v..|
00e0  cd 13 73 0d 5a 84 d2 0f  83 d8 00 be 8b 7d e9 82  |..s.Z}..|
00f0  00 66 0f b6 c6 88 64 ff  40 66 89 44 04 0f b6 d1  |.fd.@f.D|
0100  c1 e2 02 88 e8 88 f4 40  89 44 08 0f b6 c2 c0 e8  |...@.D..|
0110  02 66 89 04 66 a1 60 7c  66 09 c0 75 4e 66 a1 5c  |.f..f.`|f..uNf.\|
0120  7c 66 31 d2 66 f7 34 88  d1 31 d2 66 f7 74 04 3b  ||f1.f.4..1.f.t.;|
0130  44 08 7d 37 fe c1 88 c5  30 c0 c1 e8 02 08 c1 88  |D.}70...|
0140  d0 5a 88 c6 bb 00 70 8e  c3 31 db b8 01 02 cd 13  |.Zp..1..|
0150  72 1e 8c c3 60 1e b9 00  01 8e db 31 f6 bf 00 80  |r...`..1|
0160  8e c6 fc f3 a5 1f 61 ff  26 5a 7c be 86 7d eb 03  |..a.|..}..|
0170  be 95 7d e8 34 00 be 9a  7d e8 2e 00 cd 18 eb fe  |..}.4...}...|
0180  47 52 55 42 20 00 47 65  6f 6d 00 48 61 72 64 20  |GRUB .Geom.Hard |
0190  44 69 73 6b 00 52 65 61  64 00 20 45 72 72 6f 72  |Disk.Read. Error|
01a0  0d 0a 00 bb 01 00 b4 0e  cd 10 ac 3c 00 75 f4 c3  |...<.u..|
01b0  00 00 00 00 00 00 00 00  13 43 3c 32 00 00 80 04  |.C<2|
01c0  01 04 83 fe c2 ff 00 08  00 00 00 90 3b 00 00 00  |;...|
01d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
01f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..U.|
0200

Bug#1058638: grub-installer: boot fails: $bootdev empty [BIOS/MBR, Vista, 2 disks]: Installing grub on ''

[Tj]

Source: grub-installer
Version: 1.194
Followup-For: Bug #1058638

I think this may be due to the same cause as #1035096 and possibly also
#1035085

Bug#1006708: hw-detect: Fails to reload iwlwifi module (modprobe: FATAL: Module iwlwifi is in use)

[Cyril Brulebois]

Pascal Hambourg  (2023-12-13):
> Wouldn't it be more generic to check /sys/module/${module}/holders
> like is done for mhi ?

I was suggesting a quick and dirty way to get away with this, to see if
it helps, answering the question regarding where in the code one might
want to try something.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1058638: grub-installer: boot fails: $bootdev empty [BIOS/MBR, Vista, 2 disks]: Installing grub on ''

[Pascal-liste]

Hello,

On 13/12/2023 at 20:45, Tj wrote:


After a lot of code-chasing it appears the problem is that
grub-installer script fails to set its $bootdev variable:

grub-installer: info: Installing grub on ''

(note the empty quotes)

The target system has two 'disks'. The first has Microsoft Windows Vista
with 2 NTFS partitions using msdos label/MBR. The second is empty and intended
for the Debian OS and boot-loader.

The system boots in BIOS/Legacy/CSM mode.


I suspect this is yet another avatar of bug #1035096, for which I 
submitted a patch, see 
.

Bug#1006708: hw-detect: Fails to reload iwlwifi module (modprobe: FATAL: Module iwlwifi is in use)

[Pascal Hambourg]

Hello,

Le 13/12/2023 à 20:16, Cyril Brulebois a écrit :

Carsten Schoenert  (2023-12-13):

The "trick" is to unload the iwlmvm module instead and load afterwards
the module iwlwifi again.


See the very bottom of check-missing-firmware.sh (hw-detect repository),
the loop over $modules. You could try intercepting $module = iwlwifi,
and adding a modprobe -r theotherone beforehand.


Wouldn't it be more generic to check /sys/module/${module}/holders like 
is done for mhi ?


Note: iwldvm is also loaded by iwlwifi (for older hardware I guess) and 
depends on it, but AFAICS it is not loaded when the requested firmware 
is missing, so there is no need to unload it. Is it expected that the 
behaviour is different with hardware using iwlmvm ?

Bug#1058638: grub-installer: boot fails: $bootdev empty [BIOS/MBR, Vista, 2 disks]: Installing grub on ''

[Tj]

Source: grub-installer
Version: 1.194
Severity: important
Tags: d-i

Whilst helping user "mavaviij" in matrix Debian room I was able to
reproduce in a virtual machine a bug that causes the Debian install to
fail to boot with a blinking cursor after install.

After a lot of code-chasing it appears the problem is that
grub-installer script fails to set its $bootdev variable:

grub-installer: info: Installing grub on ''

(note the empty quotes)

The target system has two 'disks'. The first has Microsoft Windows Vista
with 2 NTFS partitions using msdos label/MBR. The second is empty and intended
for the Debian OS and boot-loader.

The system boots in BIOS/Legacy/CSM mode.

I've not been able to trace back through the code to figure out why the
failure occurs. I have however identifed what may be a clue: the Debian
OS disk sector 0 appears to contain GRUB's g2hdr.img which makes me
wonder if grub-ntldr-img was used - although not sure how that is done
and see no sign of it in the (saved) installer logs.

Reproducer:

Using debian-12.4.0-amd64-netinst.iso

fallocate -l 10 disk1-windows-vista.bin
fallocate -l 20 disk2-debian-12.bin
fdisk disk1-windows-vista.bin
#new: 1, default,512M
#type: 1, 07
#new: 2, default, default
#type: 2, 07
#write

sudo losetup -P --show -f disk1-windows-vista.bin
# /dev/loop0
sudo mount /dev/loop0p1 /mnt/tmp/
# fool os-prober into detecting Vista
sudo mkdir /mnt/tmp/boot
sudo touch /mnt/tmp/bootmgr
echo "W i n d o w s   V i s t a " | sudo dd of=/mnt/tmp/boot/bcd

find /mnt/tmp/ -ls; hexdump -C /mnt/tmp/boot/bcd
5  4 drwxrwxrwx   1 root root 4096 Dec 13 14:41 
/mnt/tmp/
   64  0 drwxrwxrwx   1 root root0 Dec 13 14:42 
/mnt/tmp/boot
   65  1 -rwxrwxrwx   1 root root   27 Dec 13 14:32 
/mnt/tmp/boot/bcd
   67  0 -rwxrwxrwx   1 root root0 Dec 13 14:41 
/mnt/tmp/bootmgr
  57 20 69 20 6e 20 64 20  6f 20 77 20 73 20 20 20  |W i n d o w s   |
0010  56 20 69 20 73 20 74 20  61 20 0a |V i s t a .|
001b

sudo umount /mnt/tmp
sudo os-prober
/dev/loop0p1:Windows Vista:Windows:chain

Now create a (libvirt/QEMU) VM guest with the 2 disk images attached and
go through the install procedure choosing the unpartitioned second disk
as the target.
At the question about installing the boot loader to the "primary disk"
answer No, then when the option to select which boot device to use choose
the 2nd disk where Debian is installed.

Save debug logs, restart VM, and even if using the SeaBIOS boot menu via
pressing Esc key, choosing the VirtIO boot device results in blinking
cursor.

Examine disk images and note that disk2 appears to have GRUB's g2hdr.img
in sector 0.

hexdump -C -n 512 disk2-debian-12.bin
  fa b8 00 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  ||
0010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.!..|
0020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |8.uu|
0030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.|...t..|
0040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.|.|
0050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
01b0  00 00 00 00 00 00 00 00  13 43 3c 32 00 00 00 04  |.C<2|
01c0  01 04 83 fe c2 ff 00 08  00 00 00 90 3b 00 00 00  |;...|
01d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
01f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..U.|
0200


NOTE: on an UEFI system doing the reproducer one needs to edit 

/usr/lib/os-probes/mounted/20microsoft

and comment out the test for UEFI on lines 11-14 if wanting to manually
test os-prober outside the running installer.

Bug#1006708: hw-detect: Fails to reload iwlwifi module (modprobe: FATAL: Module iwlwifi is in use)

[Cyril Brulebois]

Carsten Schoenert  (2023-12-13):
> The "trick" is to unload the iwlmvm module instead and load afterwards
> the module iwlwifi again.

See the very bottom of check-missing-firmware.sh (hw-detect repository),
the loop over $modules. You could try intercepting $module = iwlwifi,
and adding a modprobe -r theotherone beforehand.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1006708: hw-detect: Fails to reload iwlwifi module (modprobe: FATAL: Module iwlwifi is in use)

[Carsten Schoenert]

Hi,

Am Thu, Mar 03, 2022 at 10:04:55AM +0700 schrieb Arnaud Rebillout:
> A Kali Linux user reported a fail install on a Dell XPS 9510. For the
> background, the Kali Linux installer is a super lightweight fork of the
> Debian installer, kept in sync with Debian.

I experience the same issue with a shiny new Lenovo Thinkpad X1 Carbon
X11 using the net-installer from 2023-12-07.
This model is using a Intel Wi-Fi® 6E AX211, 802.11ax 2x2 Wi-Fi +
Bluetooth 5.3 hardware.

The full specs is availabe here:

https://psref.lenovo.com/syspool/Sys/PDF/ThinkPad/ThinkPad_X1_Carbon_Gen_11/ThinkPad_X1_Carbon_Gen_11_Spec.pdf

...
> The reason I'm opening this bug against Debian is because, while looking
> at the logs, a few lines caught my eyes. This is in the screenshot #3 in
> the imgur.com link above:
> 
>   check-missing-firmware: removing-and-loading kernel module iwlwifi
>   check-missing-firmware: modprobe: FATAL: Module iwlwifi is in use.
> 
> I wonder if those lines are harmless, or if it really indicates that the
> iwlwifi module can't be reloaded, therefore the wifi is not functional.
> Since someone documented on the Debian wiki that the firmware-iwlwifi is
> needed, and then the WiFi works, it seems that either those lines are
> harmless, or the iwlwifi is not always successfully reloaded.

No, these lines of course not harmless as they indicate that the iwlwifi
module couldn't be removed as the module is still in use. This happens
as this modul has dependencies as it's used by iwlmvm.

# lsmod | grep iwl
iwlmvm  589824  0
mac80211   1392640  1 iwlmvm
iwlwifi 544768  1 iwlmvm
cfg80211   1343488  3 iwlmvm,iwlwifi,mac80211
rfkill   40960  2 iwlmvm,cfg80211

The "trick" is to unload the iwlmvm module instead and load afterwards
the module iwlwifi again.

Doing so I was able to bring up a WLAN based interface on my X1 G11
after I provided the required firmware files for this chipset within the
installer was running.

I basically understand whats going on in check-missing-firmware.sh, I'm
not able to come up with a patch, if someone has some lines of code
where I can start with to dig deeper into I'm willing to work on this.

My guess is that once the needed firmware for the AX211 chipset is
included in the net-installer images the issue I currently see will not
be happen anymore. But in fact more users in the future will see this
issue as the unloading and new loading of the modules needs to be
handled differently to the current implementation.

Note: The needed firmware files for the X1 G11 are available by the
release of upstream linux-firmware 2023 

Regards
Carsten

Re: Shipping the mini.iso files with the installer-images package?

[Wouter Verhelst]

So.

On Wed, Dec 13, 2023 at 10:45:36AM +0200, Wouter Verhelst wrote:
> That sounds like a much more reasonable way forward, although I'm not
> keen on extending di-netboot-assistant (it's massive already, and does
> very different things). At any rate, I'll have a look at implementing
> this. Thanks for the suggestion!

This turned out to be fairly simple in the end. First, create a apt.conf
file like so:

Acquire::IndexTargets::deb::SHA256SUMS {
MetaKey 
"$(COMPONENT)/installer-$(ARCHITECTURE)/current/images/SHA256SUMS";
ShortDescription "SHA256SUMS";
Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) d-i SHA256SUMS 
(deb)";
};

Next, create a config file like this:

images:
  mini.iso:
mirror_type: deb
limit:
  Suite: stable
  Architecture: amd64
basedir: main/installer-amd64/current/images
relative_name: ./netboot/gtk/mini.iso
target_filename: /var/lib/libvirt/images/bookworm-mini.iso
  netboot.iso:
mirror_type: cd
basedir: current/amd64/iso-cd
filename_regex: debian-[0-9.]+-amd64-netinst.iso
target_filename: /var/lib/libvirt/images/bookworm-netinst.iso

And then once you have that, the following perl script should do the
job.

(config file stored either as /etc/debian-isosync/config.yaml or
passed as the first argument to the script)

---
#!/usr/bin/perl -w

use v5.36;

use Crypt::Digest::SHA256 qw/sha256_file_hex/;
use YAML::XS qw/LoadFile Dump/;
use Dpkg::Control::HashCore;
use LWP::UserAgent;
use File::Temp qw/tempdir/;

my $ua = LWP::UserAgent->new(show_progress => 1);
$ua->env_proxy;

my $cfilename = shift;
if(!defined($cfilename)) {
$cfilename = "/etc/debian-isosync/config.yaml";
}

my $cfile = LoadFile($cfilename) or die $!;

die "need list of images in config file" unless exists($cfile->{images});
die "images setting in config file is not a hash" unless ref($cfile->{images}) 
eq "HASH";

sub get_apt_sha256sum {
my $rv = [];
my $found;
open my $indextargets, "-|", "apt-get", "indextargets";
do {
my $indexes = Dpkg::Control::HashCore->new;
$found = $indexes->parse($indextargets, "index targets");
push @$rv, $indexes if $found;
} while $found;

close $indextargets;

return $rv;
}

sub ensure_file($url, $sha256_expected, $filename) {
if(-f $filename) {
my $sha256sum_found = sha256_file_hex($filename);
if ($sha256sum_found eq $sha256_expected) {
say "$filename is up-to-date, no update required";
return;
}
} else {
say "$filename not found, downloading";
}
$ua->get($url, ":content_file" => $filename);
}

NAME:
foreach my $name(keys %{$cfile->{images}}) {
say "checking $name...";
my $image = $cfile->{images}{$name};
die "invalid entry $name in config file: missing mirror type" unless 
exists($image->{mirror_type});
die "invalid entry $name in config file: missing directory with 
SHA256SUMS file" unless exists($image->{basedir});
die "invalid entry $name in config file: missing target filename" 
unless exists($image->{target_filename});
next if $image->{disabled};

if($image->{mirror_type} eq "deb") {
die "invalid entry $name in config file: missing relative 
filename" unless exists($image->{relative_name});
my $indexes = get_apt_sha256sum;
INDEX:
foreach my $index(@$indexes) {
next INDEX unless $index->{ShortDesc} eq "SHA256SUMS";
if(exists($image->{limit})) {
foreach my $limit(keys %{$image->{limit}}) {
next INDEX unless $index->{$limit} eq 
$image->{limit}{$limit};
}
}
open my $shafile, "<", $index->{Filename};
my $sha256sum_expected;
SHASUM:
while(my $line = <$shafile>) {
chomp $line;
my $filename;
($sha256sum_expected, $filename) = split /\s+/, 
$line;
last SHASUM if $filename eq 
$image->{relative_name};
$sha256sum_expected = undef;
}
next INDEX unless defined($sha256sum_expected);
ensure_file(join('/', $index->{"Base-URI"}, 
$image->{basedir}, $image->{relative_name}), $sha256sum_expected, 
$image->{target_filename});
last INDEX;
}
} elsif($image->{mirror_type} eq "cd") {
die "invalid entry $name in config file: missing filename 
regex" unless exists($image->{filename_regex});
next 

Re: Shipping the mini.iso files with the installer-images package?

[Wouter Verhelst]

On Tue, Dec 12, 2023 at 11:16:51AM +0100, Philip Hands wrote:
> Wouter Verhelst  writes:
> 
> > Hi,
> >
> > I just created
> > https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
> > which removes the "grep -v mini.iso" from the "get-images.sh" script.
> > The idea being that it can be useful to have a mini.iso available
> > locally in case you want to install a VM with libvirt. While it's
> > possible to PXE boot a VM and install that way, this involves some
> > infrastructure that needs to be set up, and pointing to a .iso file that
> > exists locally seems easier.
> >
> > I usually have a netboot mini.iso file in my home directory for that
> > purpose, but it needs to be kept up to date with every point release,
> > and that's a bit of an annoyance. I think it would be easier to just
> > have it in the debian-installer-netboot-images package.
> 
> I presume you're currently downloading a published mini.iso, rather than
> building them locally, is that right?

Correct.

> Rather than packaging up the mini.iso's, how about having a package that
> acts as an installer, and downloads the published image that matches the
> relevant kerrnel version.

... that could very much work too! I hadn't thought of that.

> The reason I think that might be worth the effort is that I suspect that
> quite a lot of the mini.iso's would have almost no users (depending on
> architecture etc), so copying them across the whole mirror network seems
> like a significant waste of resources.
> 
> Personally, I tend to use the netinst for the purpose you describe, and
> think that it might be quite useful to have a package that would
> maintain a copy of the current image on my systems, so such a package
> could be more widely useful than just for mini.iso downloads.

I had a quick look, and it turns out that both the d-i directory in the
mirror network and the CD mirror network have a SHA256SUMS file with
image names and relative paths. Both have other checksum files too, but
the two are not compatible there (d-i has MD5SUMS (yuck), cdimage has
SHA512SUMS).

I could imagine writing a tool that, when given a URI for a SHA256SUMS
file and a local path, and updates the file if the checksum does not
match (or the file is absent) 

They are both signed, but unfortunately not in the same way -- the d-i
directory is signed through (In)Release(.gpg), the cd network through a
SHA256SUMS.sign file right next to the SHA256SUMS one. But that can be
worked around.

That sounds like a much more reasonable way forward, although I'm not
keen on extending di-netboot-assistant (it's massive already, and does
very different things). At any rate, I'll have a look at implementing
this. Thanks for the suggestion!

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.