bug#36942: Reconfigure broke GRUB

[ison]

On Mon, Aug 26, 2019 at 12:20:04PM +0200, Ludovic Courtès wrote:
> 
> Closing, because this is apparently the same as
> , which fortunately was fixed a
> while back!
> 
> Thanks,
> Ludo’.

For the record I attempted another install just now and it was
successful. I'm not sure exactly what caused the initial breakage or
what fixed it, but I'm glad it's working now.

Thanks again for the help.

bug#36942: Reconfigure broke GRUB

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic  skribis:

> On Tue, 06 Aug 2019 15:48:43 -0400
> zerodaysford...@sdf.lonestar.org (Jakob L. Kreuze) wrote:
>
>> Danny Milosavljevic  writes:
>> 
>> > Hi,
>> >
>> > I've examined /var/guix/gcroots without #36947 and I get:
>> >
>> > lrwxrwxrwx 1 root root   18 24. Jul 11:32 profiles -> /var/guix/profiles
>> > lrwxrwxrwx 1 root root   19 24. Jul 11:32 current-system -> 
>> > /run/current-system
>> > lrwxrwxrwx 1 root root   18 24. Jul 11:32 booted-system -> 
>> > /run/booted-system
>> > lrwxrwxrwx 1 root root   26 29. Jul 22:26 bootcfg -> 
>> > //var/guix/gcroots/bootcfg
>> >
>> > So I guess I'm one "guix gc" away from also destroying my
>> > installation.  
>> 
>> Was this fixed after applying #36947?
>
> Yes.

Closing, because this is apparently the same as
, which fortunately was fixed a
while back!

Thanks,
Ludo’.

bug#36942: Reconfigure broke GRUB

[Ludovic Courtès]

Hello ison,

ison  skribis:

> At least it's giving an error this time:
> error: '/gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install 
> --boot-directory /mnt/boot --bootloader-id=Guix --efi-directory /boot/efi' 
> exited with status 1: output follows:
>   /gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install: error: 
> /gnu/store/drz35fc...-grub-efi-2.02/lib/grub/i386-pc/modinfo.sh doesn't 
> exist. Please specify --target or --directory.
>
> My bootloader is section is still:
>   (bootloader (bootloader-configuration
>(bootloader grub-efi-bootloader)
>(target "/boot/efi")))

The message above is telling that it’s trying to do a “BIOS” (as opposed
to EFI) install of GRUB, but you specified ‘grub-efi’, hence the
failure.

Is your system really EFI?

Anyway, this issue is different from the original one, so perhaps we
should discuss it on help-guix?

Thanks,
Ludo’.

bug#36942: Reconfigure broke GRUB

[ison]

On Tue, Aug 20, 2019, Jakob L. Kreuze wrote:
> This is perfect (I mean, not that it still doesn't work, but that we
> have some error output). I think I know what's going on. The new 'guix
> system reconfigure' uses G-Expressions, and it looks like Grub is being
> ungexp'd for the wrong architecture. To be sure, what's the architecture
> of this machine?
> 
> I'll investigate further and let you know what I find. Thanks again for
> doing this.
> 
> Regards,
> Jakob

That's great, thanks for investigating this.
Both the machine architecture and the liveUSB are x86_64.

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

ison  writes:

> So I attempted a re-install and surprisingly it still fails.
> I attempted 3 times. Using the graphical installer without efi,
> manual install without efi, manual install with efi. All downloaded
> and built everything successfully only to fail at the bootloader.
> If the problem was my config then I'd think my first attempt would
> have succeeded because the graphical installation automatically
> built the config file.
>
> At least it's giving an error this time:
> error: '/gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install 
> --boot-directory /mnt/boot --bootloader-id=Guix --efi-directory /boot/efi' 
> exited with status 1: output follows:
>   /gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install: error: 
> /gnu/store/drz35fc...-grub-efi-2.02/lib/grub/i386-pc/modinfo.sh doesn't 
> exist. Please specify --target or --directory.
>
> My bootloader is section is still:
>   (bootloader (bootloader-configuration
>(bootloader grub-efi-bootloader)
>(target "/boot/efi")))

This is perfect (I mean, not that it still doesn't work, but that we
have some error output). I think I know what's going on. The new 'guix
system reconfigure' uses G-Expressions, and it looks like Grub is being
ungexp'd for the wrong architecture. To be sure, what's the architecture
of this machine?

I'll investigate further and let you know what I find. Thanks again for
doing this.

Regards,
Jakob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[ison]

On Sat, Aug 17, 2019, Jakob L. Kreuze wrote:
> ison  writes:
> 
> > Sounds good, I'll reinstall it over the weekend
> 
> Thank you :)

So I attempted a re-install and surprisingly it still fails.
I attempted 3 times. Using the graphical installer without efi,
manual install without efi, manual install with efi. All downloaded
and built everything successfully only to fail at the bootloader.
If the problem was my config then I'd think my first attempt would
have succeeded because the graphical installation automatically
built the config file.

At least it's giving an error this time:
error: '/gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install --boot-directory 
/mnt/boot --bootloader-id=Guix --efi-directory /boot/efi' exited with status 1: 
output follows:
  /gnu/store/drz35fc...-grub-efi-2.02/sbin/grub-install: error: 
/gnu/store/drz35fc...-grub-efi-2.02/lib/grub/i386-pc/modinfo.sh doesn't exist. 
Please specify --target or --directory.

My bootloader is section is still:
  (bootloader (bootloader-configuration
   (bootloader grub-efi-bootloader)
   (target "/boot/efi")))

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

ison  writes:

> I do have a /boot/grub actually. Is that strange?
> It looks like a normal /boot/grub to me, and contains a grub.cfg

I think that's normal -- my virtual machine had one, too.

> Sounds good, I'll reinstall it over the weekend

Thank you :)

Regards,
JAkob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[ison]

On Fri, Aug 16, 2019, Jakob L. Kreuze wrote:
> ison  writes:
> 
> > /boot/efi looks ok as far as I can tell at least. It's tree is:
> > /boot/efi/
> > └── EFI
> > ├── grub
> > │   └── grubx64.efi
> > ├── Guix
> > │   └── grubx64.efi
> > └── GuixSD
> > └── grubx64.efi
> > All 3 grubx64.efi files differ from each other and are around 120kb.
> 
> Interesting... I only have '/boot/efi/EFI/Guix'. Do you have a
> '/boot/grub'?

I do have a /boot/grub actually. Is that strange?
It looks like a normal /boot/grub to me, and contains a grub.cfg

> I think it may be indicative of a more serious problem, since you
> mentioned that this was fine up until the commit refactoring 'guix
> system reconfigure'. If you're willing to reinstall and see if you can
> reproduce it, that maywould be helpful -- you could send me the relevant
> 'install-bootloader.scm' store item if it breaks again.
> 
> Thank you very much, ison. I appreciate your cooperation here greatly.

Sounds good, I'll reinstall it over the weekend

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

ison  writes:

> Sorry it took me so long to reply, I didn't have access to the broken
> machine until now.

No worries :)

> As you requested here is the partitions as listed by parted:
>
> Number  Start   End SizeFile system NameFlags
>  1  1049kB  3046kB  2097kB  grubbios_grub
>  2  3146kB  540MB   537MB   fat32   efi boot, esp
>  3  540MB   2588MB  2048MB  linux-swap(v1)  swap
>  4  2588MB  750GB   748GB   ext4guixsd

Hm. Nearly the same as what I had, though my partitions had different
sizes and I didn't have a swap partition. I may try again with this
exact setup when I get back to the United States just to see what
happens.

> /boot/efi looks ok as far as I can tell at least. It's tree is:
> /boot/efi/
> └── EFI
> ├── grub
> │   └── grubx64.efi
> ├── Guix
> │   └── grubx64.efi
> └── GuixSD
> └── grubx64.efi
> All 3 grubx64.efi files differ from each other and are around 120kb.

Interesting... I only have '/boot/efi/EFI/Guix'. Do you have a
'/boot/grub'?

> But considering you can't reproduce the issue would it be a good
> idea for me to reinstall and see if I can even reproduce it?
> Although I think this time I would leave off the bios_grub partition
> I'm willing to keep testing the current bug if it's important or
> indicative of a more serious problem. But please don't feel
> obligated to keep working on this just for me, I'm perfectly fine
> just reinstalling. The issue doesn't seem to affect my desktop
> anyway, I was able to upgrade it smoothly.
>
> It's always possible the issue was caused by some strange
> combination involving the patched bug, from earlier in the
> discussion, and my weird partition layout. Might not even be worth
> investigating unless it happens to someone else
> (or me again after a fresh install).

I think it may be indicative of a more serious problem, since you
mentioned that this was fine up until the commit refactoring 'guix
system reconfigure'. If you're willing to reinstall and see if you can
reproduce it, that maywould be helpful -- you could send me the relevant
'install-bootloader.scm' store item if it breaks again.

Thank you very much, ison. I appreciate your cooperation here greatly.

Regards,
Jakob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[ison]

On Wed, Aug 14, 2019 at 03:50:00PM -0400, Jakob L. Kreuze wrote:
> However, I still cannot seem to reproduce this issue with the most
> recent master. Everything boots fine. I've extracted the bootloader
> installation "script", and everything appears to be normal to me.

Sorry it took me so long to reply, I didn't have access to the
broken machine until now. As you requested here is the partitions as
listed by parted:

Number  Start   End SizeFile system NameFlags
 1  1049kB  3046kB  2097kB  grubbios_grub
 2  3146kB  540MB   537MB   fat32   efi boot, esp
 3  540MB   2588MB  2048MB  linux-swap(v1)  swap
 4  2588MB  750GB   748GB   ext4guixsd

/boot/efi looks ok as far as I can tell at least. It's tree is:
/boot/efi/
└── EFI
├── grub
│   └── grubx64.efi
├── Guix
│   └── grubx64.efi
└── GuixSD
└── grubx64.efi
All 3 grubx64.efi files differ from each other and are around 120kb.


But considering you can't reproduce the issue would it be a good
idea for me to reinstall and see if I can even reproduce it?
Although I think this time I would leave off the bios_grub partition
I'm willing to keep testing the current bug if it's important or
indicative of a more serious problem. But please don't feel
obligated to keep working on this just for me, I'm perfectly fine
just reinstalling. The issue doesn't seem to affect my desktop
anyway, I was able to upgrade it smoothly.

It's always possible the issue was caused by some strange
combination involving the patched bug, from earlier in the
discussion, and my weird partition layout. Might not even be worth
investigating unless it happens to someone else
(or me again after a fresh install).

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

Hi ison,

ison  writes:

> On Wed, Aug 07, 2019, Jakob L. Kreuze wrote:
>> I'll continue to investigate; perhaps I'll be able to reproduce if I
>> copy your exact partition scheme in the virtual machine. I'm sorry that
>> you had to go through that whole 'guix init' spiel again.
>
> No problem, it's a backup machine anyway, I've held off on updating
> my main (but it doesn't use efi so maybe I can). If this is a
> problem that only I'm having and nobody else then I wonder if a
> fresh install would fix it. Although I'm surprised nobody else seems to be 
> experiencing it.
>
> The only unusual thing I did before it broke is that I tried to consolidate 
> my configs from 2 machines into a single one by adding case statements.
> For example at the top I had (define local-profile 'laptop)
> and then my bootloader was something like:
> (bootloader (bootloader-configuration
>  (case local-profile
>   ((desktop)
>(bootloader grub-bootloader)
>...)
>   ((laptop)
>(bootloader grub-efi-bootloader)
>...
>
> But I've since reverted back to my old config for all the subsequent
> "guix init" and reconfigures I've done, so it seems unlikely to be the
> cause but I thought I'd mention it anyway.
> Maybe having a BIOS boot partition on /dev/sda1 even though I'm using
> efi is somehow messing it up?

After some further research, I was able to create a virtual machine with
the partition scheme and bootloader configuration that you described.

;; This is an operating system configuration template
;; for a "bare bones" setup, with no X11 display server.

(use-modules (gnu))
(use-service-modules networking ssh)
(use-package-modules screen)

(operating-system
  (host-name "komputilo")
  (timezone "Europe/Berlin")
  (locale "en_US.utf8")

  ;; Boot in "legacy" BIOS mode, assuming /dev/sdX is the
  ;; target hard disk, and "my-root" is the label of the target
  ;; root file system.
  (bootloader (bootloader-configuration
(bootloader grub-efi-bootloader)
(target "/boot/efi")))
  (file-systems (cons* (file-system
 (device "/dev/sda2")
 (mount-point "/boot/efi")
 (type "vfat"))
   (file-system
 (device (file-system-label "my-root"))
 (mount-point "/")
 (type "ext4"))
   %base-file-systems))

  ;; This is where user accounts are specified.  The "root"
  ;; account is implicit, and is initially created with the
  ;; empty password.
  (users (cons (user-account
(name "alice")
(comment "Bob's sister")
(group "users")

;; Adding the account to the "wheel" group
;; makes it a sudoer.  Adding it to "audio"
;; and "video" allows the user to play sound
;; and access the webcam.
(supplementary-groups '("wheel"
"audio" "video")))
   %base-user-accounts))

  ;; Globally-installed packages.
  (packages (cons screen %base-packages))

  ;; Add services to the baseline: a DHCP client and
  ;; an SSH server.
  (services (append (list (service dhcp-client-service-type)
  (service openssh-service-type
   (openssh-configuration
(port-number 
%base-services)))

However, I still cannot seem to reproduce this issue with the most
recent master. Everything boots fine. I've extracted the bootloader
installation "script", and everything appears to be normal to me.

(begin
  (use-modules (gnu build bootloader)
   (gnu build install)
   (guix build utils)
   (guix store)
   (guix utils)
   (ice-9 binary-ports)
   (srfi srfi-34)
   (srfi srfi-35))
  (let* ((gc-root (string-append "/" %gc-roots-directory "/bootcfg"))
 (temp-gc-root (string-append gc-root ".new")))
(switch-symlinks temp-gc-root 
"/gnu/store/xlb81742i5sb4cdmidfhabprc17ijwck-grub.cfg")
(install-boot-config "/gnu/store/xlb81742i5sb4cdmidfhabprc17ijwck-grub.cfg" 
"/boot/grub/grub.cfg" "/")
(when #t
  (catch #t
(lambda ()
  ((lambda (bootloader efi-dir mount-point)
 (let ((grub-install (string-append bootloader 
"/sbin/grub-install"))
   (install-dir (string-append mount-point "/boot"))
   (target-esp (if (file-exists? (string-append mount-point 
efi-dir))
   (string-append mount-point efi-dir)
   efi-dir)))
   (setenv "GRUB_ENABLE_CRYPTODISK" "y")
   (invoke/quiet grub-install "--boot-directory" install-dir 
"--bootloader-id=Guix" 

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

Hi ison,

ison  writes:

> I'm sorry to report that it actually didn't fix the problem.

I'll continue to investigate; perhaps I'll be able to reproduce if I
copy your exact partition scheme in the virtual machine. I'm sorry that
you had to go through that whole 'guix init' spiel again.

> Also I don't know why it didn't occur to me to mention this detail
> before, but the reason I previously thought the message mentioned in
> bug#36878 was important is because it's actually the very last thing I
> see on the command line after reconfiguring. The line I'm talking
> about is of course: shepherd: Evaluating user expression (let*
> ((services (map primitive-load (?))) # ?) ?)
>
> I'm pretty sure even if that message is normal, it's not normal for it
> to be the last line you see. Perhaps the reconfigure is dying at that
> point, or some point shortly after, without outputting any other
> errors.

Do you see a "bootloader successfully installed on ..." message earlier
in the output? 5c8c8c455 changed the order that things are done in so
that services are upgraded after the bootloader is installed.

Regards,
Jakob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[ison]

On Tue, Aug 06, 2019 at 02:18:15PM -0400, Jakob L. Kreuze wrote:
> Hi ison,
> 
> I wasn't able to reproduce with an OVMF virtual machine, but the
> symptoms you've described lead me to suspect that the issue concerns
> improper installation of a GC root for grub. Would you be willing to see
> if the patch provided by #36947 fixes your issue?

I'm sorry to report that it actually didn't fix the problem.

Also I don't know why it didn't occur to me to mention this detail
before, but the reason I previously thought the message mentioned
in bug#36878 was important is because it's actually the very last
thing I see on the command line after reconfiguring. The line I'm
talking about is of course:
shepherd: Evaluating user expression (let* ((services (map primitive-load (?))) 
# ?) ?)

I'm pretty sure even if that message is normal, it's not normal
for it to be the last line you see.
Perhaps the reconfigure is dying at that point, or some point
shortly after, without outputting any other errors.

bug#36942: Reconfigure broke GRUB

[Danny Milosavljevic]

Hi Jakob,

On Tue, 06 Aug 2019 15:48:43 -0400
zerodaysford...@sdf.lonestar.org (Jakob L. Kreuze) wrote:

> Danny Milosavljevic  writes:
> 
> > Hi,
> >
> > I've examined /var/guix/gcroots without #36947 and I get:
> >
> > lrwxrwxrwx 1 root root   18 24. Jul 11:32 profiles -> /var/guix/profiles
> > lrwxrwxrwx 1 root root   19 24. Jul 11:32 current-system -> 
> > /run/current-system
> > lrwxrwxrwx 1 root root   18 24. Jul 11:32 booted-system -> 
> > /run/booted-system
> > lrwxrwxrwx 1 root root   26 29. Jul 22:26 bootcfg -> 
> > //var/guix/gcroots/bootcfg
> >
> > So I guess I'm one "guix gc" away from also destroying my
> > installation.  
> 
> Was this fixed after applying #36947?

Yes.

> > The double slash looks very suspicious...  
> 
> Any reason in particular? The second slash is there because
> '%gc-roots-directory' has a leading slash, and that gets appended to
> 'target', which in this case is '/'.

Ah okay.


pgpKgBY478aW8.pgp
Description: Digitale Signatur von OpenPGP

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

Hi Danny,

Danny Milosavljevic  writes:

> Hi,
>
> I've examined /var/guix/gcroots without #36947 and I get:
>
> lrwxrwxrwx 1 root root   18 24. Jul 11:32 profiles -> /var/guix/profiles
> lrwxrwxrwx 1 root root   19 24. Jul 11:32 current-system -> 
> /run/current-system
> lrwxrwxrwx 1 root root   18 24. Jul 11:32 booted-system -> /run/booted-system
> lrwxrwxrwx 1 root root   26 29. Jul 22:26 bootcfg -> 
> //var/guix/gcroots/bootcfg
>
> So I guess I'm one "guix gc" away from also destroying my
> installation.

Was this fixed after applying #36947?

> The double slash looks very suspicious...

Any reason in particular? The second slash is there because
'%gc-roots-directory' has a leading slash, and that gets appended to
'target', which in this case is '/'.

Regards,
Jakob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[Danny Milosavljevic]

Hi,

I've examined /var/guix/gcroots without #36947 and I get:

lrwxrwxrwx 1 root root   18 24. Jul 11:32 profiles -> /var/guix/profiles
lrwxrwxrwx 1 root root   19 24. Jul 11:32 current-system -> /run/current-system
lrwxrwxrwx 1 root root   18 24. Jul 11:32 booted-system -> /run/booted-system
lrwxrwxrwx 1 root root   26 29. Jul 22:26 bootcfg -> //var/guix/gcroots/bootcfg

So I guess I'm one "guix gc" away from also destroying my installation.

The double slash looks very suspicious...


pgpVkjj_lufoH.pgp
Description: Digitale Signatur von OpenPGP

bug#36942: Reconfigure broke GRUB

[Jakob L. Kreuze]

Hi ison,

Thanks for opening a new ticket for tracking this.

ison  writes:

> Continuing this issue from https://issues.guix.gnu.org/issue/36878
>
> guix reconfigure recently broke GRUB for me. When rebooting I get
> dropped to a "grub rescue>" shell with an error about
> "grub_file_filters" being an unknown symbol.
> If I try doing the usual commands to tell GRUB how to boot I just
> see the above error repeated, or "unknown command" when I run
> things such as insmod or "configfile".
>
> So to fix the problem I can boot to a Guix install disk and do
> "guix init" which rebuilds the system using older package
> definitions from the disk.
> That allows me to get a working system, but if I do another
> "guix pull" and reconfigure using the same config file it breaks
> GRUB again when I reboot. "guix describe" shows that my latest
> attempt was with commit 35600cd.
>
> Here is the bootloader and filesystem sections of my config:
>   (bootloader (bootloader-configuration
> (bootloader grub-efi-bootloader)
> (target "/boot/efi")))
>   (file-systems (cons* (file-system
>  (device "/dev/sda2")
>  (mount-point "/boot/efi")
>  (type "vfat"))
>(file-system
>  (device (file-system-label "guixsd-root"))
>  (mount-point "/")
>  (type "ext4"))
>%base-file-systems))

This narrows it down quite a bit -- I suspect that we're broken for
'grub-efi-bootloader', since I've been able to successfully reconfigure
with 'grub-bootloader' on both of my machines.

> I should make a note that I usually don't use efi, and I'm not
> completely confident it's all set up properly. I do have a
> "BIOS boot" partition on /dev/sda1 too, is that even needed with
> efi?

I don't believe so. I think the ESP partition on /dev/sda2 is all you
need.

> Although, I have been using this setup, and the above
> definitions, for about 6 months now without any bootloader or
> filesystem issues. And the same config is being used to fix the
> system when GRUB breaks as well as to reconfigure afterward
> (causing the breakage). So my guess is some new update is the
> culprit.

It certainly sounds like it, and I'm the last person to have touched the
bootloader installation code :)

I'll look into this ASAP. Thanks again for the bug report!

Regards,
Jakob


signature.asc
Description: PGP signature

bug#36942: Reconfigure broke GRUB

[ison]

Continuing this issue from https://issues.guix.gnu.org/issue/36878

guix reconfigure recently broke GRUB for me. When rebooting I get
dropped to a "grub rescue>" shell with an error about
"grub_file_filters" being an unknown symbol.
If I try doing the usual commands to tell GRUB how to boot I just
see the above error repeated, or "unknown command" when I run
things such as insmod or "configfile".

So to fix the problem I can boot to a Guix install disk and do
"guix init" which rebuilds the system using older package
definitions from the disk.
That allows me to get a working system, but if I do another
"guix pull" and reconfigure using the same config file it breaks
GRUB again when I reboot. "guix describe" shows that my latest
attempt was with commit 35600cd.

Here is the bootloader and filesystem sections of my config:
  (bootloader (bootloader-configuration
(bootloader grub-efi-bootloader)
(target "/boot/efi")))
  (file-systems (cons* (file-system
 (device "/dev/sda2")
 (mount-point "/boot/efi")
 (type "vfat"))
   (file-system
 (device (file-system-label "guixsd-root"))
 (mount-point "/")
 (type "ext4"))
   %base-file-systems))

I should make a note that I usually don't use efi, and I'm not
completely confident it's all set up properly. I do have a
"BIOS boot" partition on /dev/sda1 too, is that even needed with
efi? Although, I have been using this setup, and the above
definitions, for about 6 months now without any bootloader or
filesystem issues. And the same config is being used to fix the
system when GRUB breaks as well as to reconfigure afterward
(causing the breakage). So my guess is some new update is the
culprit.