Re: SOLVED - Re: Deb10 installer can't install grub
On Sun, Mar 7, 2021 at 7:03 AM Dave Sherohman wrote: > On Wed, Mar 03, 2021 at 10:28:34AM -0500, Marc Auslander wrote: > > One potential gotcha. When you boot from an mdadm file system containing > > /boot/grub, grub will not write to the file system. In particular, it > will > > not update grub/grubenv even if you have a save_env line in grub.cfg. > So if > > you use grub-reboot to specify an alternate line in grub.cfg, you need to > > reset grubenv afterwards. I do this in a root @reboot cron job. > > > > If you don't know what I'm talking about, you probably don't need to > worry > > about this. > > I don't *know* know what you're talking about, but I think I get the > gist of it. I'm not doing anything fancy enough to have to deal with > the details of grubenv and grub-reboot, but your initial comment about > grub refusing to write to a mdadm filesystem is consistent with what I > was seeing, and it's nice to have confirmation that it actually does > work that way. > Dave the issues you describe dealing with mdadm setup, well that's why I eventually just switched to LVM-based mirroring and striping. Early on, LVM wasn't supported as a boot volume, but the original code base had supported that on HP-UX and AIX. So years ago now, that restriction was lifted. Another mdadm issue is just the use of it in a kind of "oral tradition": it's not that setup is hard, it just still seems like magic spells after all this time. -- > Dave Sherohman > >
Re: SOLVED - Re: Deb10 installer can't install grub
On Wed, Mar 03, 2021 at 10:28:34AM -0500, Marc Auslander wrote: > One potential gotcha. When you boot from an mdadm file system containing > /boot/grub, grub will not write to the file system. In particular, it will > not update grub/grubenv even if you have a save_env line in grub.cfg. So if > you use grub-reboot to specify an alternate line in grub.cfg, you need to > reset grubenv afterwards. I do this in a root @reboot cron job. > > If you don't know what I'm talking about, you probably don't need to worry > about this. I don't *know* know what you're talking about, but I think I get the gist of it. I'm not doing anything fancy enough to have to deal with the details of grubenv and grub-reboot, but your initial comment about grub refusing to write to a mdadm filesystem is consistent with what I was seeing, and it's nice to have confirmation that it actually does work that way. -- Dave Sherohman
Re: Deb10 installer can't install grub
On Mi, 03 mar 21, 17:16:14, Felix Miata wrote: > Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200): > > > Felix Miata wrote: > > >> To start with, RAID1 is marginally slower than ordinary filesystems on > >> partitions. > > > This is true for some workloads, for others it can be significantly > > faster. > > > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/ > > I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in > any of > the graphs there, and the subject of RAID1 barely touched, basically > describing > its purely mirror topology and little else. Quote from the article: In our performance charts, we show a line from two disks through eight disks for RAID10. The first datapoint, for two disks, is obviously a simple RAID1. The datapoints for four, six, and eight disks are RAID10. We draw a line through all points for ease of interpretation—but there is no three, five, or seven disk RAID10 in the actual data, for hopefully obvious reasons. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: Deb10 installer can't install grub
Hello, On Fri, Mar 05, 2021 at 10:04:34AM +, Darac Marjal wrote: > So, if your file is small, then yes, you won't see any performance > benefit. But if your file is larger than a block, or if you want > to access more than one file at once, then RAID can read the > second block from a different drive. However, Linux MD RAID-1 will only ever achieve the read performance of a single device per thread, so in order to get the read performance benefit of multiple copies you either need to use multiple reading threads / processes or else use RAID-10 instead. https://www.spinics.net/lists/raid/msg47240.html ZFS does not suffer from this. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Deb10 installer can't install grub
On 05/03/2021 09:25, Anssi Saari wrote: > Greg Wooledge writes: > >> to...@tuxteam.de (to...@tuxteam.de) wrote: 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s >>> Thanks. Real data :) >>> >>> The doubling in read throughput is somewhat surprising to me. Some >>> learning to do, it seems. >> There are two copies of everything. So, if you need to read a big >> file, you can read half of it from disk 1 and the other half from disk 2, >> roughly simultaneously. > I have a vague memory of reading once that mdraid doesn't stripe reads > to a single file. I.e. with raid1 you get the double speed if you read > different files (or different parts of disk) but reading a single file, > no. RAID won't know what's a file or not. It works below the file system layer, so only concerns itself with blocks. For any particular block, a RAID1 array has N copies of that block to read from. It makes no sense to read all of those copies - imagine if you were really paranoid and had a 10-disk RAID array. Instead, the driver will choose one of the constituent drives and read the block from that drive. So, if your file is small, then yes, you won't see any performance benefit. But if your file is larger than a block, or if you want to access more than one file at once, then RAID can read the second block from a different drive. Remember that the I in RAID stands for Inexpensive. RAID assumes that the disk is slow, the bus is faster and the applications are impatient :) > > So is this different in zfs? I guess I could test this myself. The way > Bonnie++ was used for those results above is definitely parallel. ZFS will work in much the same manner, except when performing a "scrub". During a scrub, all copies of all blocks are read and compared. If a block differs across the mirrors, then ZFS has a decision to make: if there are more identical copies than different copies, ZFS assumes that the identical copies are good (hence why having a three-way mirror is recommended). Alternatively, ZFS uses checksums on stored data. If the block doesn't match the checksum, then ZFS can also recover that data. More details here https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integrity > OpenPGP_signature Description: OpenPGP digital signature
Re: Deb10 installer can't install grub
Greg Wooledge writes: > to...@tuxteam.de (to...@tuxteam.de) wrote: >> > 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s >> > 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s > >> Thanks. Real data :) >> >> The doubling in read throughput is somewhat surprising to me. Some >> learning to do, it seems. > > There are two copies of everything. So, if you need to read a big > file, you can read half of it from disk 1 and the other half from disk 2, > roughly simultaneously. I have a vague memory of reading once that mdraid doesn't stripe reads to a single file. I.e. with raid1 you get the double speed if you read different files (or different parts of disk) but reading a single file, no. So is this different in zfs? I guess I could test this myself. The way Bonnie++ was used for those results above is definitely parallel.
Re: Deb10 installer can't install grub
On Thu, Mar 04, 2021 at 08:09:46AM -0500, Greg Wooledge wrote: > to...@tuxteam.de (to...@tuxteam.de) wrote: > > > 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , > > > r=204MB/s > > > 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s > > > Thanks. Real data :) > > > > The doubling in read throughput is somewhat surprising to me. Some > > learning to do, it seems. > > There are two copies of everything. So, if you need to read a big > file, you can read half of it from disk 1 and the other half from disk 2, > roughly simultaneously. Makes sense, thanks. Cheers - t signature.asc Description: Digital signature
Re: Deb10 installer can't install grub
to...@tuxteam.de (to...@tuxteam.de) wrote: > > 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s > > 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s > Thanks. Real data :) > > The doubling in read throughput is somewhat surprising to me. Some > learning to do, it seems. There are two copies of everything. So, if you need to read a big file, you can read half of it from disk 1 and the other half from disk 2, roughly simultaneously.
Re: Deb10 installer can't install grub
Dave Sherohman writes: > Any tips on making use of the grub shell to make further progress, such > as getting the system to boot in non-rescue mode (i.e., not chrooted > from the installer)? The help information available in the grub shell > itself isn't terribly useful because it scrolls off the screen with no > (obvious) pager or scrollback buffer. There is a pager in Grub but it really isn't obvious. For the future it's enabled by set pager=1. Why it's not on by default I don't know...
Re: Deb10 installer can't install grub
On Wed, Mar 03, 2021 at 05:40:39PM -0500, Dan Ritter wrote: > Felix Miata wrote: > > Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200): > > > > > Felix Miata wrote: > > [...] Do > > you know of, or can you provide a reference to, any way RAID1 performance > > can be > > better than single disk? > > Sure. > > https://calomel.org/zfs_raid_speed_capacity.html > > First two lines of the first table of results: > > 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s > 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s > > Which is as theory predicts: slight reduction in write > performance (write being dominated by disk cache -> disk, not > system -> disk cache), and approximate doubling in read > performance. Thanks. Real data :) The doubling in read throughput is somewhat surprising to me. Some learning to do, it seems. Cheers - t signature.asc Description: Digital signature
Re: Deb10 installer can't install grub
On Wed, Mar 03, 2021 at 05:16:14PM -0500, Felix Miata wrote: > Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200): > > > Felix Miata wrote: > > >> To start with, RAID1 is marginally slower than ordinary filesystems on > >> partitions. > > > This is true for some workloads, for others it can be significantly > > faster. > > > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/ > > [...] Do > you know of, or can you provide a reference to, any way RAID1 performance can > be > better than single disk? I can just provide a hint: when accessing mirrored disks, reading is done whenever the first disk finishes; writing whenever both disks are done. So reading is potentially faster, writing potentially slower. That's about latency. The throughput shouldn't change too much (unless you're combining very different disks, but that'd be asking for it). Compose this with several layers of buffering and I doubt you'll be able to make out any significant difference though. No experimental data to back this up, alas. > Evolution as taught in public schools, like religion, > is based on faith, not on science. OK. Now all bets wrt RAID are off, anyway. Cheers - t signature.asc Description: Digital signature
Re: Deb10 installer can't install grub
Felix Miata wrote: > Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200): > > > Felix Miata wrote: > > >> To start with, RAID1 is marginally slower than ordinary filesystems on > >> partitions. > > > This is true for some workloads, for others it can be significantly > > faster. > > > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/ > > I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in > any of > the graphs there, and the subject of RAID1 barely touched, basically > describing > its purely mirror topology and little else. I can't recall any graph there > that > described performance of anything other than single disk, RAID6, and RAID10. > Do > you know of, or can you provide a reference to, any way RAID1 performance can > be > better than single disk? Sure. https://calomel.org/zfs_raid_speed_capacity.html First two lines of the first table of results: 1x 4TB, single drive, 3.7 TB, w=108MB/s , rw=50MB/s , r=204MB/s 2x 4TB, mirror (raid1),3.7 TB, w=106MB/s , rw=50MB/s , r=488MB/s Which is as theory predicts: slight reduction in write performance (write being dominated by disk cache -> disk, not system -> disk cache), and approximate doubling in read performance. -dsr-
Re: Deb10 installer can't install grub
Andrei POPESCU composed on 2021-03-03 17:50 (UTC+0200): > Felix Miata wrote: >> To start with, RAID1 is marginally slower than ordinary filesystems on >> partitions. > This is true for some workloads, for others it can be significantly > faster. > https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/ I wrote not RAID, but RAID1, very purposely. I found no mention of RAID1 in any of the graphs there, and the subject of RAID1 barely touched, basically describing its purely mirror topology and little else. I can't recall any graph there that described performance of anything other than single disk, RAID6, and RAID10. Do you know of, or can you provide a reference to, any way RAID1 performance can be better than single disk? -- Evolution as taught in public schools, like religion, is based on faith, not on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/
Re: Deb10 installer can't install grub
On Ma, 02 mar 21, 14:01:52, Felix Miata wrote: > > To start with, RAID1 is marginally slower than ordinary filesystems on > partitions. This is true for some workloads, for others it can be significantly faster. https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/ Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: SOLVED - Re: Deb10 installer can't install grub
On 3/3/2021 6:30 AM, Dave Sherohman wrote: Based on this, I'm guessing that the original problem was that the installer forgot to include mdadm support in its grub options, even though it was configured with an mdadm boot device. And then I missed a couple steps after adding mdadm support, so it didn't all get installed to the EFI partitions correctly. One potential gotcha. When you boot from an mdadm file system containing /boot/grub, grub will not write to the file system. In particular, it will not update grub/grubenv even if you have a save_env line in grub.cfg. So if you use grub-reboot to specify an alternate line in grub.cfg, you need to reset grubenv afterwards. I do this in a root @reboot cron job. If you don't know what I'm talking about, you probably don't need to worry about this.
SOLVED - Re: Deb10 installer can't install grub
So, after my last message, I ran across a general mdadm setup guide at https://raid.wiki.kernel.org/index.php/Setting_up_a_(new)_system and, in the "booting" section of that document, it said to add GRUB_CMDLINE_LINUX="domdadm" to /etc/default/grub. Didn't have that, so I added it, did the checks mentioned in the document, and reinstalled grub to both EFI partitions. Still didn't work. Then I found https://linoxide.com/linux-how-to/grub-rescue-commands/ and was able to start the installed kernel with the command sequence: set root=(md/1) linux (md/1)/vmlinuz root=/dev/md1 initrd (md/1)/boot/initrd.img-4.19.0-14-amd64 insmod normal boot This still ultimately died with an error saying the EFI partition had an "Invalid FSINFO signature". Searches on that phrase mostly seemed to talk about secure boot problems, so I double-checked the bios options and secure boot was off, so that wasn't it. So I went back to the linoxide.com document and went through all the steps that I hadn't already done manually: Running os-prober and update-grub. Then grub-install to the EFI partitions again, reboot, and... success! Based on this, I'm guessing that the original problem was that the installer forgot to include mdadm support in its grub options, even though it was configured with an mdadm boot device. And then I missed a couple steps after adding mdadm support, so it didn't all get installed to the EFI partitions correctly. -- Dave Sherohman
Re: Deb10 installer can't install grub
General update on my situation: Just before knocking off for the day, I tried telling the installer to finish up without installing a boot loader, then used the netinst USB stick as a rescue disk to boot my installed system. From there, I was able to use the regular grub-install program (instead of the install environment's grub-installer script). grub-install ran with no errors and I now have grub in the (separate, non-RAIDed) EFI partitions of both nvme drives... but it's still not directly bootable. Trying to boot directly from nvme drops me into a grub command line. Thinking that this may be an initrd or similar issue, I tried reinstalling the kernel image and grub-efi-amd64 packages, so that their postinst scripts would run and rebuild initrd and reinstall grub to the drives, but that had no effect. Still just getting the grub shell when I boot from nvme. Any tips on making use of the grub shell to make further progress, such as getting the system to boot in non-rescue mode (i.e., not chrooted from the installer)? The help information available in the grub shell itself isn't terribly useful because it scrolls off the screen with no (obvious) pager or scrollback buffer. Alternately, suggestions for things I can try in the chroot environment would also be good, since that's considerably less restrictive than the installer environment I was trying to work within yesterday. I guess the most obvious explanation for the current grub issues is that grub isn't smart enough to boot an mdadm filesystem directly and I need to repartition with a non-RAID /boot, but I don't consider that a desirable solution, since it would then leave me with no /boot if the device holding that partition dies. (No new text below this point, just my original post for context.) On Tue, Mar 02, 2021 at 05:57:37AM -0600, Dave Sherohman wrote: > I've got a new server and am currently fighting with the Debian 10 > installer (build 20190702) in my attempts to get it up and running. > After much wailing and gnashing of teeth, I managed to get it to stop > complaining about being unble to mount /boot/efi and complete the > "Partition disks" step successfully, but now I'm completely stuck on the > "Install GRUB" step. > > The GUI installer shows the error: > > Unable to install GRUB in dummy > Executing 'grub-install dummy' failed. > > Checking the syslog output on virtual console 4 shows a bunch of > os-prober activity (as expected), then finally: > > Installing for x64_64-efi platform. > grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`. > error: Running 'grub-install --force "dummy"' failed. > > I assume that the "canonical path" it's looking for is a /dev/sda-type > device name, but I have no idea how to assign one of those to an nvme > drive. (And I thought that kind of name was supposed to have been > banished in favor of "predictable" names by now anyhow.) > > Several of the old-style names are already in use; the installer is > showing sda for the USB stick that the installation was booted from and > sdb-sdi for eight large disks, but the operating system is to boot and > run from the drives /dev/nvme0n1 and /dev/nvme1n1. > > What do I need to do to get this working? > > Also, would that solution also work for md devices as well as for nvme > devices? I had previously tried putting UEFI onto a RAID1 mirror > between the two nvme drives, but got a similar error from grub about not > being able to find the canonical path of /dev/md1. > > -- > Dave Sherohman > -- Dave Sherohman
Re: Deb10 installer can't install grub
On Tue, Mar 02, 2021 at 05:57:37AM -0600, Dave Sherohman wrote: > I've got a new server and am currently fighting with the Debian 10 > installer (build 20190702) in my attempts to get it up and running. > After much wailing and gnashing of teeth, I managed to get it to stop > complaining about being unble to mount /boot/efi and complete the > "Partition disks" step successfully, but now I'm completely stuck on the > "Install GRUB" step. > > The GUI installer shows the error: > > Unable to install GRUB in dummy > Executing 'grub-install dummy' failed. > > Checking the syslog output on virtual console 4 shows a bunch of > os-prober activity (as expected), then finally: > > Installing for x64_64-efi platform. > grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`. > error: Running 'grub-install --force "dummy"' failed. Check the settings for the disk in BIOS equivalent - I had lots of problems with an HP because I'd inadvertently told the HP software to take over partitioning of the disks and put them into an array. Andy C.
Re: Deb10 installer can't install grub
Dan Ritter composed on 2021-03-02 07:54 (UTC-0500): > Dave Sherohman wrote: >> I've got a new server... > You'll want to create the following partitions on each, > identically: > 1 efi - type efi Not RAID I presume? > 2 boot (or boot/root) - type MDADM volume I never have /boot on RAID, but that's just me I suppose. > 3 root, if using separate boot - type MDADM volume Agree. > 4 swap - type MDADM volume Think about this: To start with, RAID1 is marginally slower than ordinary filesystems on partitions. Does a server use swap to hibernate? Does a server /want/ to use swap for anything? Does anyone want swap on any type of SSD? If I was to put swap on NVME, I would not use a RAID, just like on my old RAID systems on HDDs. If I had 32G or more RAM, I'd just let the kernel swap to file instead of devoting discrete partition space to swap. -- Evolution as taught in public schools, like religion, is based on faith, not on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/
Re: Deb10 installer can't install grub
On Tue, 2 Mar 2021 09:50:22 -0500 Dan Ritter wrote: > > I'm not positive that that's the correct approach, however, > > especially given that I suspect that /boot will need to be visible > > to GRUB and may thus need to be outside the LVM. > > grub can find a root in LVM, but needs /boot to be outside. > /boot in an mdadm RAID1 is a good idea there. I'm not sure you need a separate /boot partition. I am using Bullseye weekly builds, so I may have a more recent d-i and grub than you do. Also this is on i386 on a small (60GB) drive, and a DOS type MBR. I have one LVM partition, with swap and root in that. The LVM partition starts at sector 2048, so I expect grub resides on the first 2048 sectors. I suspect one can get an idea of things grub can do by listing /boot/grub/i386-pc/. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: Deb10 installer can't install grub
On 2021-03-02 at 09:50, Dan Ritter wrote: > The Wanderer wrote: > >> On 2021-03-02 at 07:54, Dan Ritter wrote: >> >>> You'll want to create the following partitions on each, >>> identically: >>> >>> 1 efi - type efi >>> 2 boot (or boot/root) - type MDADM volume >>> 3 root, if using separate boot - type MDADM volume >>> 4 swap - type MDADM volume >>> >>> Then you go to the mdadm setup and create MDADM RAID1 devices >>> out of each pair of boot, root and swap. >> >> Out of interest, how would you modify this for someone who intends to >> run LVM on top of the RAID-1 being defined here, and wants to define as >> many of the partitions as possible (but at least /) at the LVM level >> rather than the mdadm level? >> >> I'm guessing that the answer would be to define the EFI partition here >> directly, then define one further partition as an appropriate type, set >> up LVM inside that, and define the further partitions from there. >> >> I'm not positive that that's the correct approach, however, especially >> given that I suspect that /boot will need to be visible to GRUB and may >> thus need to be outside the LVM. > > grub can find a root in LVM, but needs /boot to be outside. > /boot in an mdadm RAID1 is a good idea there. So, three partitions, then? 1. efi 2. /boot 3. LVM And define everything else inside the LVM. (In practice, "everything else" here will be /, /var, /tmp, and possibly swap. /home will be on a separate RAID array.) It'll still probably be interesting figuring out what to specify as the destination for the GRUB install, but if I'm lucky it might Just Work. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: Deb10 installer can't install grub
The Wanderer wrote: > On 2021-03-02 at 07:54, Dan Ritter wrote: > > > You'll want to create the following partitions on each, > > identically: > > > > 1 efi - type efi > > 2 boot (or boot/root) - type MDADM volume > > 3 root, if using separate boot - type MDADM volume > > 4 swap - type MDADM volume > > > > Then you go to the mdadm setup and create MDADM RAID1 devices > > out of each pair of boot, root and swap. > > Out of interest, how would you modify this for someone who intends to > run LVM on top of the RAID-1 being defined here, and wants to define as > many of the partitions as possible (but at least /) at the LVM level > rather than the mdadm level? > > I'm guessing that the answer would be to define the EFI partition here > directly, then define one further partition as an appropriate type, set > up LVM inside that, and define the further partitions from there. > > I'm not positive that that's the correct approach, however, especially > given that I suspect that /boot will need to be visible to GRUB and may > thus need to be outside the LVM. grub can find a root in LVM, but needs /boot to be outside. /boot in an mdadm RAID1 is a good idea there. -dsr-
Re: Deb10 installer can't install grub
On Tue, Mar 02, 2021 at 09:26:21AM -0500, The Wanderer wrote: > I didn't parse what he wrote that way. He said: > > You'll want to create the following partitions on each, > identically: > > 1 efi - type efi > 2 boot (or boot/root) - type MDADM volume > 3 root, if using separate boot - type MDADM volume > 4 swap - type MDADM volume > > Then you go to the mdadm setup and create MDADM RAID1 devices > out of each pair of boot, root and swap. > > Note that he specified to RAID the three listed MDADM partitions, but > did not mention the same for the EFI partition. Ah, right. When he mentioned RAIDing the boot, I took that to mean the EFI partitions. Forgot that he as using a separate /boot. So I took a look at the grub-installer script, and guess what I found as a comment just before the `ls` that kills it? --- # XXX cjwatson 2019-03-25: This is all far too complicated and fragile, and # should be replaced with in-target or similar. # Ensure proc is mounted in all the $chroot calls; # needed for RAID+LVM for example initial_proc_contents="$(ls $ROOT/proc)" --- "Far too complicated and fragile", indeed! -- Dave Sherohman
Re: Deb10 installer can't install grub
On 2021-03-02 at 09:20, Dave Sherohman wrote: > On Tue, Mar 02, 2021 at 09:09:52AM -0500, The Wanderer wrote: > >> On 2021-03-02 at 09:01, Dave Sherohman wrote: >>> RAID Device #1 is 1.9 TB, ext4fs, and set to mount on / >>> RAID Device #2 is 2.0 GB, ESP, and bootable >> >> So the EFI partitions themselves are also RAIDed? I wouldn't have >> expected that to work. > > That was my expectation as well, but Dan said it should work, so I went > with it. I didn't parse what he wrote that way. He said: You'll want to create the following partitions on each, identically: 1 efi - type efi 2 boot (or boot/root) - type MDADM volume 3 root, if using separate boot - type MDADM volume 4 swap - type MDADM volume Then you go to the mdadm setup and create MDADM RAID1 devices out of each pair of boot, root and swap. Note that he specified to RAID the three listed MDADM partitions, but did not mention the same for the EFI partition. > Of course, I've also tried it with then not RAIDed and the results > are identical, aside from the "canonoical path" error message in > syslog changing from /dev/md2 to /dev/nvme0n1p1. Not terribly surprising, but it was worth a check. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: Deb10 installer can't install grub
On Tue, Mar 02, 2021 at 09:09:52AM -0500, The Wanderer wrote: > On 2021-03-02 at 09:01, Dave Sherohman wrote: > > > On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote: > > >> Then you go to the mdadm setup and create MDADM RAID1 devices > >> out of each pair of boot, root and swap. > > > > RAID Device #1 is 1.9 TB, ext4fs, and set to mount on / > > RAID Device #2 is 2.0 GB, ESP, and bootable > > So the EFI partitions themselves are also RAIDed? I wouldn't have > expected that to work. That was my expectation as well, but Dan said it should work, so I went with it. Of course, I've also tried it with then not RAIDed and the results are identical, aside from the "canonoical path" error message in syslog changing from /dev/md2 to /dev/nvme0n1p1. > Is grub-installer specific to the Debian installer environment? I don't > find it at a glance in my installed system, or with 'apt-file search'; I > do find grub-install, which is a binary (where the error you quote seems > to indicate a script), and is what I seem to recall having used for the > manual installation of GRUB that finally got my current machine working > with a RAIDed boot-drive pair all those years ago. (Albeit not with EFI > boot partitioning, so that may not be directly comparable.) Yes, it seems to be unique to the installer environment. I made that same trip in reverse, first trying to run grub-install, and then having to go looking when that didn't exist. > If grub-installer is a script and is the correct thing to use, the first > thing I'd be inclined to do in your place is to examine that script and > see what it's trying to do at the point of the error, so that I can > figure out what it expects as input and how I can potentially try to > provide that. Good thought. It's a shell script, so I'll see if I can make sense of what it's doing in [filesystem]/proc. -- Dave Sherohman
Re: Deb10 installer can't install grub
On 2021-03-02 at 09:01, Dave Sherohman wrote: > On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote: >> Then you go to the mdadm setup and create MDADM RAID1 devices >> out of each pair of boot, root and swap. > > RAID Device #1 is 1.9 TB, ext4fs, and set to mount on / > RAID Device #2 is 2.0 GB, ESP, and bootable So the EFI partitions themselves are also RAIDed? I wouldn't have expected that to work. >> After install to those RAID1 devices, grub will want to install. >> Do that to the boot sector of /dev/nvme0n1, and then again to >> the boot sector of /dev/nvme1n1. > > ...and this is where it dies. I'm now in the text-mode "expert > install", and there is no option to manually specify a partition to > install GRUB on, only a yes/no "Force GRUB installation to the EFI > removable media path?" prompt. Regardless of which answer I choose, I > get "Unable to install GRUB in dummy" and syslog records "failed to get > canonical path of /dev/md2". > > I also tried dropping to a shell and manually running `grub-installer > /dev/nvme0n1` and got the error "ls: /dev/nvme0n1/proc: Not a directory" > Manually running grub-installer for /dev/md2 and /dev/nvme1n1 > (unsurprisingly) produced similar results. Is grub-installer specific to the Debian installer environment? I don't find it at a glance in my installed system, or with 'apt-file search'; I do find grub-install, which is a binary (where the error you quote seems to indicate a script), and is what I seem to recall having used for the manual installation of GRUB that finally got my current machine working with a RAIDed boot-drive pair all those years ago. (Albeit not with EFI boot partitioning, so that may not be directly comparable.) If grub-installer is a script and is the correct thing to use, the first thing I'd be inclined to do in your place is to examine that script and see what it's trying to do at the point of the error, so that I can figure out what it expects as input and how I can potentially try to provide that. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: Deb10 installer can't install grub
On Tue, Mar 02, 2021 at 07:54:01AM -0500, Dan Ritter wrote: > In the installer, you want expert mode. Does the disk > partitioner recognize both nvme0n1 and nvme1n1 ? Yes. The list of disks shown in the partitioner is currently: RAID device #1 RAID device #2 /dev/nvme0n1 /dev/nvme1n1 SCSI2 (0,0,0) (sda) - the USB stick I booted from SCSI1 (0,0,0) (sdb) - the first storage drive ... the other seven storage drives ... > You'll want to create the following partitions on each, > identically: > > 1 efi - type efi > 2 boot (or boot/root) - type MDADM volume > 3 root, if using separate boot - type MDADM volume > 4 swap - type MDADM volume Partitioning is identical on both nvme drives, consisting of: 1.0 MB free space 2.0 GB ESP (bootable) 8.0 GB swap 1.9 TB RAID physical device 331.8 kB free space > Then you go to the mdadm setup and create MDADM RAID1 devices > out of each pair of boot, root and swap. RAID Device #1 is 1.9 TB, ext4fs, and set to mount on / RAID Device #2 is 2.0 GB, ESP, and bootable > After install to those RAID1 devices, grub will want to install. > Do that to the boot sector of /dev/nvme0n1, and then again to > the boot sector of /dev/nvme1n1. ...and this is where it dies. I'm now in the text-mode "expert install", and there is no option to manually specify a partition to install GRUB on, only a yes/no "Force GRUB installation to the EFI removable media path?" prompt. Regardless of which answer I choose, I get "Unable to install GRUB in dummy" and syslog records "failed to get canonical path of /dev/md2". I also tried dropping to a shell and manually running `grub-installer /dev/nvme0n1` and got the error "ls: /dev/nvme0n1/proc: Not a directory" Manually running grub-installer for /dev/md2 and /dev/nvme1n1 (unsurprisingly) produced similar results. -- Dave Sherohman
Re: Deb10 installer can't install grub
On 2021-03-02 at 07:54, Dan Ritter wrote: > You'll want to create the following partitions on each, > identically: > > 1 efi - type efi > 2 boot (or boot/root) - type MDADM volume > 3 root, if using separate boot - type MDADM volume > 4 swap - type MDADM volume > > Then you go to the mdadm setup and create MDADM RAID1 devices > out of each pair of boot, root and swap. Out of interest, how would you modify this for someone who intends to run LVM on top of the RAID-1 being defined here, and wants to define as many of the partitions as possible (but at least /) at the LVM level rather than the mdadm level? I'm guessing that the answer would be to define the EFI partition here directly, then define one further partition as an appropriate type, set up LVM inside that, and define the further partitions from there. I'm not positive that that's the correct approach, however, especially given that I suspect that /boot will need to be visible to GRUB and may thus need to be outside the LVM. This isn't purely academic interest; I'm not moving forward right now because I can't currently afford all the components, but I'm planning a new computer build as soon as that changes, and one part of the design I've decided on is a pair of M.2 SSDs in RAID-1. I remember having had enough trouble getting GRUB installed and booting with a pair of SATA SSDs in RAID-1 when I built my current machine, which doesn't use GPT or EFI boot, and I'd prefer to avoid having to juggle that on the fly again - especially not given that the need to cater to the partitioning requirements of EFI boot is likely to make things more complicated. -- The Wanderer The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. -- George Bernard Shaw signature.asc Description: OpenPGP digital signature
Re: Deb10 installer can't install grub
Dave Sherohman wrote: > I've got a new server and am currently fighting with the Debian 10 > installer (build 20190702) in my attempts to get it up and running. > After much wailing and gnashing of teeth, I managed to get it to stop > complaining about being unble to mount /boot/efi and complete the > "Partition disks" step successfully, but now I'm completely stuck on the > "Install GRUB" step. > > The GUI installer shows the error: > > Unable to install GRUB in dummy > Executing 'grub-install dummy' failed. > > Checking the syslog output on virtual console 4 shows a bunch of > os-prober activity (as expected), then finally: > > Installing for x64_64-efi platform. > grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`. > error: Running 'grub-install --force "dummy"' failed. > > I assume that the "canonical path" it's looking for is a /dev/sda-type > device name, but I have no idea how to assign one of those to an nvme > drive. (And I thought that kind of name was supposed to have been > banished in favor of "predictable" names by now anyhow.) > > Several of the old-style names are already in use; the installer is > showing sda for the USB stick that the installation was booted from and > sdb-sdi for eight large disks, but the operating system is to boot and > run from the drives /dev/nvme0n1 and /dev/nvme1n1. > > What do I need to do to get this working? > > Also, would that solution also work for md devices as well as for nvme > devices? I had previously tried putting UEFI onto a RAID1 mirror > between the two nvme drives, but got a similar error from grub about not > being able to find the canonical path of /dev/md1. /dev/nvme0n1 is your first NVMe drive; /dev/nvme1n1 is your second. Add a p1 to each to reference the first partition. If you want to install to an mdadm RAID, that's do-able with grub as well. In the installer, you want expert mode. Does the disk partitioner recognize both nvme0n1 and nvme1n1 ? You'll want to create the following partitions on each, identically: 1 efi - type efi 2 boot (or boot/root) - type MDADM volume 3 root, if using separate boot - type MDADM volume 4 swap - type MDADM volume Then you go to the mdadm setup and create MDADM RAID1 devices out of each pair of boot, root and swap. After install to those RAID1 devices, grub will want to install. Do that to the boot sector of /dev/nvme0n1, and then again to the boot sector of /dev/nvme1n1. After a reboot into the new OS, you should set up your storage disks. If this doesn't work, tell us where in the process it stops, and what the error is there. -dsr-
Deb10 installer can't install grub
I've got a new server and am currently fighting with the Debian 10 installer (build 20190702) in my attempts to get it up and running. After much wailing and gnashing of teeth, I managed to get it to stop complaining about being unble to mount /boot/efi and complete the "Partition disks" step successfully, but now I'm completely stuck on the "Install GRUB" step. The GUI installer shows the error: Unable to install GRUB in dummy Executing 'grub-install dummy' failed. Checking the syslog output on virtual console 4 shows a bunch of os-prober activity (as expected), then finally: Installing for x64_64-efi platform. grub-install: error: failed to get canonical path of `/dev/nvme0n1p1`. error: Running 'grub-install --force "dummy"' failed. I assume that the "canonical path" it's looking for is a /dev/sda-type device name, but I have no idea how to assign one of those to an nvme drive. (And I thought that kind of name was supposed to have been banished in favor of "predictable" names by now anyhow.) Several of the old-style names are already in use; the installer is showing sda for the USB stick that the installation was booted from and sdb-sdi for eight large disks, but the operating system is to boot and run from the drives /dev/nvme0n1 and /dev/nvme1n1. What do I need to do to get this working? Also, would that solution also work for md devices as well as for nvme devices? I had previously tried putting UEFI onto a RAID1 mirror between the two nvme drives, but got a similar error from grub about not being able to find the canonical path of /dev/md1. -- Dave Sherohman
