Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[James A. Peltier]

- Original Message -
| On Mon, Aug 22, 2016 at 1:24 PM, James A. Peltier  wrote:
| >
| >
| > When running grub2-install from within recovery mode I can assure you it is
| > not a user error because simply installing the grub2-efi-modules package
| > allows for grub2-install to work.
| 
| No, this logic is flawed. Running grub2-install is obsolete on UEFI,
| it only applies for users who know exactly what they're getting
| themselves into and have a use case for modules in grub2-efi-modules
| that are not already in the grubx64.efi binary that's included in the
| grub2-efi package. If you run grub2-install, it blows away that
| grubx64.efi from the grub2-efi package in favor of a custom built one,
| which has completely different and for the most part undocumented
| behavior.

Perhaps this should be documented then.  I had assumed (yes, I know) to this 
working the same on an EFI based system system since apparently this it isn't 
documented.

| For example the grubx64.efi bootloader in grub2-efi expects to find
| grub.cfg on the ESP in the same directory as the grubx64.efi binary.
| If you run grub2-install, the resulting grubx64.efi expects to find
| grub.cfg in /boot/grub2/ which is on your boot volume, not the EFI
| System Partition. If this is UEFI system with Secure Boot enabled, the
| grub2-install created grubx64.efi is not signed, so it'll fail Secure
| Boot unless you go down the rabbit hole of signing it yourself.
| Whereas the CentOS supplied grubx64.efi in the grub2-efi package is
| already signed. And so on.
| 
| How are you booting the CentOS installation media? How was that media
| created? This matters because it's possible to end up with a CSM-BIOS
| boot inadvertently, and the installer will install a grub for BIOS
| firmware, and not the entirely separate bootloader for UEFI. So it
| might be worth booting from that install media, and get to a shell and
| check if in fact this is an UEFI mode boot by running efibootmgr. If
| you get an error message, it's not a UEFI mode boot, it's using
| CSM-BIOS mode, and that would explain why the wrong bootloader is
| being installed by the installer.


We're doing a pretty standard iPXE/Kickstart installation that has worked on 
other UEFI based systems R710/720/730, etc based systems.  This is the first 
system that I've run into this with.  We will try booting the installation 
media, because we're also including the updates repository as part of the 
installation and perhaps this is causing the issue.

-- 
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 604-365-6432
Fax : 778-782-3045
E-Mail  : jpelt...@sfu.ca
Website : http://www.sfu.ca/itservices
Twitter : @sfu_rcg
Powering Engagement Through Technology
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[Chris Murphy]

On Mon, Aug 22, 2016 at 1:24 PM, James A. Peltier  wrote:
>
>
> When running grub2-install from within recovery mode I can assure you it is 
> not a user error because simply installing the grub2-efi-modules package 
> allows for grub2-install to work.

No, this logic is flawed. Running grub2-install is obsolete on UEFI,
it only applies for users who know exactly what they're getting
themselves into and have a use case for modules in grub2-efi-modules
that are not already in the grubx64.efi binary that's included in the
grub2-efi package. If you run grub2-install, it blows away that
grubx64.efi from the grub2-efi package in favor of a custom built one,
which has completely different and for the most part undocumented
behavior.

For example the grubx64.efi bootloader in grub2-efi expects to find
grub.cfg on the ESP in the same directory as the grubx64.efi binary.
If you run grub2-install, the resulting grubx64.efi expects to find
grub.cfg in /boot/grub2/ which is on your boot volume, not the EFI
System Partition. If this is UEFI system with Secure Boot enabled, the
grub2-install created grubx64.efi is not signed, so it'll fail Secure
Boot unless you go down the rabbit hole of signing it yourself.
Whereas the CentOS supplied grubx64.efi in the grub2-efi package is
already signed. And so on.

How are you booting the CentOS installation media? How was that media
created? This matters because it's possible to end up with a CSM-BIOS
boot inadvertently, and the installer will install a grub for BIOS
firmware, and not the entirely separate bootloader for UEFI. So it
might be worth booting from that install media, and get to a shell and
check if in fact this is an UEFI mode boot by running efibootmgr. If
you get an error message, it's not a UEFI mode boot, it's using
CSM-BIOS mode, and that would explain why the wrong bootloader is
being installed by the installer.


-- 
Chris Murphy
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[James A. Peltier]

- Original Message -
| On Fri, Aug 19, 2016 at 4:59 PM, James A. Peltier  wrote:
| >
| >
| > - Original Message -
| > | On Thu, Aug 18, 2016 at 11:57 AM, James A. Peltier 
| > | wrote:
| > | > Hi All,
| > | >
| > | > I have a Dell R710 that has 6x1TB in a RAID-5 configuration.
| > |
| > |
| > | This is hardware RAID 5? Because it's pretty screwy how this ends up
| > | working when using software RAID and might take additional
| > | troubleshooting.
| >
| > Yes, it's a Dell R710XD
| >
| > | >  When installing CentOS 7 using the full disk capacity and booting in
| > | >  UEFI
| > | >  mode the machine dumps me into a GRUB rescue mode prompt.
| > | >   error: disk `,gpt2' not found
| > | >   Entering rescue mode...
| > | >   grub rescue>
| > |
| > |
| > | This is confusing to me because there should be no such thing as grub
| > | rescue on UEFI. On BIOS systems, there is boot.img (formerly stage 1)
| > | and core.img in the MBR gap or on BIOS Boot if GPT disk (formerly
| > | stage 1.5 and stage 2). The core.img is where grub rescue comes from
| > | when it can't find grub modules, in particular normal.mod.
| > |
| > | But on UEFI, core.img, normal.mod, and a pile of other modules are all
| > | baked into the grubx64.efi file founds on the EFI system partition.
| > |
| > | I suspect two things that can cause normal.mod to not be found:
| > | a. The system is not in fact booting in UEFI mode and there's been
| > | some mistake in the installation of grub.
| > | b. The system is in UEFI mode, but either the installer, or
| > | post-install, grub2-install was run which obliterates the grub2-efi
| > | package installed grubx64.efi, i.e. it's not really proper to run
| > | grub2-install on UEFI systems.
| >
| > I suspect this is the case.  when attempting to run grub-install the system
| > claims that the grub2-efi-modules packages aren't installed, so this may
| > be an installer bug.
| 
| What is attempting to run grub-install? Or even grub2-install? If the
| installer is doing this, it's an installer bug. If the user is doing
| it, it's user error.
| 
| Also, you will need to check the NVRAM for stale values because
| grub2-install also populates NVRAM with what will become the wrong
| entry. You'll need to use 'efibootmgr -v' to get a listing to find the
| bogus entry, which will be pointing to a path that includes
| grubx64.efi, note the boot number and then do 'efibootmgr -b 
| -B'   Where bootnum is the four digit value for the bogus entry.
| 
| What should happen if there are no valid entries is shim.efi will work
| with fallback.efi to create a proper NVRAM entry. The proper entry can
| be found with the earlier grep efibootmgr command, and you can just
| use that, while adding an additional \ for each \, so that it's \\.
| NVRAM should point to shim.efi and it's shim.efi that loads the
| prebaked grubx64.efi.

When running grub2-install from within recovery mode I can assure you it is not 
a user error because simply installing the grub2-efi-modules package allows for 
grub2-install to work.  We will try the efibootmgr -v out too.


-- 
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 604-365-6432
Fax : 778-782-3045
E-Mail  : jpelt...@sfu.ca
Website : http://www.sfu.ca/itservices
Twitter : @sfu_rcg
Powering Engagement Through Technology
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[Chris Murphy]

On Fri, Aug 19, 2016 at 4:59 PM, James A. Peltier  wrote:
>
>
> - Original Message -
> | On Thu, Aug 18, 2016 at 11:57 AM, James A. Peltier  wrote:
> | > Hi All,
> | >
> | > I have a Dell R710 that has 6x1TB in a RAID-5 configuration.
> |
> |
> | This is hardware RAID 5? Because it's pretty screwy how this ends up
> | working when using software RAID and might take additional
> | troubleshooting.
>
> Yes, it's a Dell R710XD
>
> | >  When installing CentOS 7 using the full disk capacity and booting in UEFI
> | >  mode the machine dumps me into a GRUB rescue mode prompt.
> | >   error: disk `,gpt2' not found
> | >   Entering rescue mode...
> | >   grub rescue>
> |
> |
> | This is confusing to me because there should be no such thing as grub
> | rescue on UEFI. On BIOS systems, there is boot.img (formerly stage 1)
> | and core.img in the MBR gap or on BIOS Boot if GPT disk (formerly
> | stage 1.5 and stage 2). The core.img is where grub rescue comes from
> | when it can't find grub modules, in particular normal.mod.
> |
> | But on UEFI, core.img, normal.mod, and a pile of other modules are all
> | baked into the grubx64.efi file founds on the EFI system partition.
> |
> | I suspect two things that can cause normal.mod to not be found:
> | a. The system is not in fact booting in UEFI mode and there's been
> | some mistake in the installation of grub.
> | b. The system is in UEFI mode, but either the installer, or
> | post-install, grub2-install was run which obliterates the grub2-efi
> | package installed grubx64.efi, i.e. it's not really proper to run
> | grub2-install on UEFI systems.
>
> I suspect this is the case.  when attempting to run grub-install the system 
> claims that the grub2-efi-modules packages aren't installed, so this may be 
> an installer bug.

What is attempting to run grub-install? Or even grub2-install? If the
installer is doing this, it's an installer bug. If the user is doing
it, it's user error.

Also, you will need to check the NVRAM for stale values because
grub2-install also populates NVRAM with what will become the wrong
entry. You'll need to use 'efibootmgr -v' to get a listing to find the
bogus entry, which will be pointing to a path that includes
grubx64.efi, note the boot number and then do 'efibootmgr -b 
-B'   Where bootnum is the four digit value for the bogus entry.

What should happen if there are no valid entries is shim.efi will work
with fallback.efi to create a proper NVRAM entry. The proper entry can
be found with the earlier grep efibootmgr command, and you can just
use that, while adding an additional \ for each \, so that it's \\.
NVRAM should point to shim.efi and it's shim.efi that loads the
prebaked grubx64.efi.


-- 
Chris Murphy
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[James A. Peltier]

- Original Message -
| On Thu, Aug 18, 2016 at 11:57 AM, James A. Peltier  wrote:
| > Hi All,
| >
| > I have a Dell R710 that has 6x1TB in a RAID-5 configuration.
| 
| 
| This is hardware RAID 5? Because it's pretty screwy how this ends up
| working when using software RAID and might take additional
| troubleshooting.

Yes, it's a Dell R710XD

| >  When installing CentOS 7 using the full disk capacity and booting in UEFI
| >  mode the machine dumps me into a GRUB rescue mode prompt.
| >   error: disk `,gpt2' not found
| >   Entering rescue mode...
| >   grub rescue>
| 
| 
| This is confusing to me because there should be no such thing as grub
| rescue on UEFI. On BIOS systems, there is boot.img (formerly stage 1)
| and core.img in the MBR gap or on BIOS Boot if GPT disk (formerly
| stage 1.5 and stage 2). The core.img is where grub rescue comes from
| when it can't find grub modules, in particular normal.mod.
| 
| But on UEFI, core.img, normal.mod, and a pile of other modules are all
| baked into the grubx64.efi file founds on the EFI system partition.
| 
| I suspect two things that can cause normal.mod to not be found:
| a. The system is not in fact booting in UEFI mode and there's been
| some mistake in the installation of grub.
| b. The system is in UEFI mode, but either the installer, or
| post-install, grub2-install was run which obliterates the grub2-efi
| package installed grubx64.efi, i.e. it's not really proper to run
| grub2-install on UEFI systems.

I suspect this is the case.  when attempting to run grub-install the system 
claims that the grub2-efi-modules packages aren't installed, so this may be an 
installer bug.

| Boot off install media with boot parameter inst.rescue and choose all
| the default options; this ought to assemble the file system per fstab,
| and you can
| 
| chroot /mnt/sysimage
| yum reinstall grub2-efi
| efibootmgr -v
| grep efibootmgr /var/log/anaconda/program.log   ## I think that's
| right it might be anaconda.program.log though
| 
| 
| It's really just reinstalling grub2-efi that should fix the problem,
| the following two options are just information gathering in case the
| reboot still doesn't work.

We'll try this and get back to you soon.

-- 
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 604-365-6432
Fax : 778-782-3045
E-Mail  : jpelt...@sfu.ca
Website : http://www.sfu.ca/itservices
Twitter : @sfu_rcg
Powering Engagement Through Technology
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos

Re: [CentOS] GRUB 2 dumps to grub prompt when installed on >4TB disk

[Chris Murphy]

On Thu, Aug 18, 2016 at 11:57 AM, James A. Peltier  wrote:
> Hi All,
>
> I have a Dell R710 that has 6x1TB in a RAID-5 configuration.


This is hardware RAID 5? Because it's pretty screwy how this ends up
working when using software RAID and might take additional
troubleshooting.



>  When installing CentOS 7 using the full disk capacity and booting in UEFI 
> mode the machine dumps me into a GRUB rescue mode prompt.
>   error: disk `,gpt2' not found
>   Entering rescue mode...
>   grub rescue>


This is confusing to me because there should be no such thing as grub
rescue on UEFI. On BIOS systems, there is boot.img (formerly stage 1)
and core.img in the MBR gap or on BIOS Boot if GPT disk (formerly
stage 1.5 and stage 2). The core.img is where grub rescue comes from
when it can't find grub modules, in particular normal.mod.

But on UEFI, core.img, normal.mod, and a pile of other modules are all
baked into the grubx64.efi file founds on the EFI system partition.

I suspect two things that can cause normal.mod to not be found:
a. The system is not in fact booting in UEFI mode and there's been
some mistake in the installation of grub.
b. The system is in UEFI mode, but either the installer, or
post-install, grub2-install was run which obliterates the grub2-efi
package installed grubx64.efi, i.e. it's not really proper to run
grub2-install on UEFI systems.

Boot off install media with boot parameter inst.rescue and choose all
the default options; this ought to assemble the file system per fstab,
and you can

chroot /mnt/sysimage
yum reinstall grub2-efi
efibootmgr -v
grep efibootmgr /var/log/anaconda/program.log   ## I think that's
right it might be anaconda.program.log though


It's really just reinstalling grub2-efi that should fix the problem,
the following two options are just information gathering in case the
reboot still doesn't work.

-- 
Chris Murphy
___
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos