Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
On Tuesday 22 Oct 2013 08:10:18 Nicolas Sebrecht wrote: The 21/10/13, Mick wrote: I'm fast gravitating towards this option ... Although with metadata 0.90 I was able to progress with the installation (after I deselected the swap partitions) the grub-install script wanted to install in /dev/md127p1 but it failed. I had to override the Ubuntu installer since I could only install grub in the /dev/md127 block device. Which is the one we expect. /dev/md127p1 is the first partition of /dev/md127. Right, although Ubuntu's installer would point only to /dev/md127p1. I had to ask it to not install GRUB in the MBR (why would it choose /dev/md127p1 as the device where the MBR resides is another matter) which then allowed me to edit the entry and point it to /dev/md127. Pointing GRUB installer to /dev/sda or sdb failed (no filesystem found). Either way, it won't boot again. Now it stays on a blank screen, no error at all shown. I don't understand why this blank screen. Or do you mean a black screen? Yes, sorry, poor choice of words the colour of the screen is black, and the content blank (well, there is a horizontal, non-flashing cursor at the top left of the screen). I'll have another go with sysrescueCD to see if I install grub on /dev/sda and /dev/sdb and if this does not work either, It should work. Linux software RAID is assembled once the kernel is up and running. Before, the system boot as usual on a single disk. Though, I'm not sure how mdadm will handle the disk change behind his back. Yes, it will! :-) OK, having tried a couple of options this is what I have concluded. Superblock with metadata 0.90 is written at the front of the disk. No need to partition each disk separately, because any partitions created on /dev/md0 also show up on each disk, i.e. the partition table created on /dev/md0 seems to be readily recognised on each /dev/sda sdb disks, as can be verified with fdisk -l. Installing grub on /dev/sda and /dev/sdb is a straight forward exercise either with the Ubuntu installer, or afterwards once booted into the new installation with sysrescueCD. With any other metadata format things are more complicated. This may be because the location of the metadata changes (not always at the start of the disk) or because the format is different and the kernel doesn't deal with it natively. Not sure and it doesn't matter. This is what I did to be able to use mdadm -e 1.2, which is the default metadata format these days: Boot with sysrescueCD to create the array and partitions on it == mdadm --create --auto=mdp --verbose /dev/md0 --level=mirror --raid-devices=2 \ /dev/sda /dev/sdb Then fdisk /dev/md0 and create 4 partitions. Leave them all with the default linux type of 83 (unless you want to use some fs exotica here). Then 'fdisk /dev/sda' which will create an MSDOS partition table automatically, in case the disk doesn't yet have a partition table on it. Then I created one primary partition accepting the default start and end values offered by fdisk. Repeat for /dev/sdb. Create fs on the RAID array: === 'mkfs.ext4 -c -L label_name /dev/md0p1' and repeat for each partition on the RAID except for the swap partition (if you have created one for this purpose). Boot with Ubuntu server CD to install the OS: Go through the tedious installation process; yeah, I know - it is faster than Gentoo, but it *feels* more tedious to me! :p You'll find that the fs and labels are not recognised by Ubuntu's partition manager. Select each RAID partition (except swap) and set a fs plus a mount point. These are used by the installer to populate the fstab with. Leave the partitions unformatted and ignore any warnings that come up later when you write these changes on disk. Continue with the installation until it is time to install GRUB. The installer will choose /dev/md127p1. Select to *not* install GRUB in the MBR, which will allow you to edit the drive entry. Choose /dev/md127 and complete the installation. Format a swap partition on the RAID: === Reboot with sysrescueCD, use fdisk /dev/md0 (that's how it will be recognised by sysrescueCD) and change the fs type of the swap partition to 82. Then, 'mkswap -c -l swap /dev/md0p2' or whichever partition you have your swap on. Edit /etc/fstab to include your new swap partition. I don't know why Ubuntu's installer would not accept the swap partition on a RAID1 device, but this was the work around I used. Install GRUB on each disk: = Reboot sysrescueCD, but now use the alternative option to boot into a Linux OS on the disk. It will probe the disks, assemble the RAID1 array and boot into Ubuntu OS. Install grub on /dev/sda and /dev/sdb - no need to stop the array. Then 'update-initramfs
[gentoo-user] Re: [O/T] RAID help - now won't boot
The 21/10/13, Mick wrote: I'm fast gravitating towards this option ... Although with metadata 0.90 I was able to progress with the installation (after I deselected the swap partitions) the grub-install script wanted to install in /dev/md127p1 but it failed. I had to override the Ubuntu installer since I could only install grub in the /dev/md127 block device. Which is the one we expect. /dev/md127p1 is the first partition of /dev/md127. BTW, I'm still at a loss as to why for Ubuntu the RAID 1 is seen as /dev/md127 and not /dev/md0 which I created originally with sysrescuecd. Names of RAID devices are built at boot time. It depends on /etc/mdadm.conf which should be part of the initramfs. Otherwise, consider the name random. Either way, it won't boot again. Now it stays on a blank screen, no error at all shown. I don't understand why this blank screen. Or do you mean a black screen? I'll have another go with sysrescueCD to see if I install grub on /dev/sda and /dev/sdb and if this does not work either, It should work. Linux software RAID is assembled once the kernel is up and running. Before, the system boot as usual on a single disk. Though, I'm not sure how mdadm will handle the disk change behind his back. Once the installed system is bootable, I suggest you to try to reinstall grub. This will be required at some point in time in the future to update it either way. I'll stop wasting time and follow your suggestion of installing on a single disk first, before I mirror it thereafter. -- Nicolas Sebrecht
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
Mick michaelkintz...@gmail.com wrote: On Sunday 20 Oct 2013 15:31:12 jo...@antarean.org wrote: I would suggest trying it by usong the older metadata format. Check the man pages, but I thinl it would be --metadata=0.90 (or similar) during creation. That might put the metadata at the end, rather then at the front. (Or it's the other way round and new metadata does it at the end.) -- Joost Ps. I have never tried it this way (full disk raid for boot device) using linux software raid. Ha! Yes, this made a difference, thanks! With metadata 0.90 I can see the same partitions I set up on /dev/md0, also on /dev/sda and /dev/sdb. The only problem now is that the Ubuntu server CD wants to format /dev/sda2 as swap and fails at that stage. :-/ Not sure how to by-pass this. I may also try metadata=1.0 to see if this makes a difference, which also positions the RAID data superblock at the end of the device: Sub-Version Superblock Position on Device --- - 0.9 At the end of the device 1.0 At the end of the device 1.1 At the beginning of the device 1.2 4K from the beginning of the device -- Regards, Mick To bypass the swap format you could try either deselecting the format option (if it exists) or setting the partition type to something else. The partition type can be set back to swap later from a livecd without having to reinstall. Other option: 1 install to single disk 2 using sysresccd create a degraded raid1 using the 2nd drive 3 copy the partitions and date from drive 1 to the degraded raid device 4 add disk 1 to the raid 5 wait for the raid device is synchronized 6 change fstab and grub config to reflect the new disklayout -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[gentoo-user] Re: [O/T] RAID help - now won't boot
The 21/10/13, J. Roeleveld wrote: Ha! Yes, this made a difference, thanks! With metadata 0.90 I can see the same partitions I set up on /dev/md0, also on /dev/sda and /dev/sdb. Sorry to come back late in this thread. As other contributors pointed out correctly, the problem was RAID metadata at the beginning. The only problem now is that the Ubuntu server CD wants to format /dev/sda2 as swap and fails at that stage. :-/ Not sure how to by-pass this. Yes. Most of the installers suck at that game. What I would do (already done this way) is: - install the disks in another machine with virtualization capacity; - create the RAID 1 (metadata=0.90); - create a virtual machine with the built RAID as single disk; - boot on the CD to install any distro; - move the disk out to the target bare metal machine; - update fstab and grub if needed. This has the advantage to not require to bypass the installer at some stage at the price of a temporary installation of the disks somewhere else. I may also try metadata=1.0 to see if this makes a difference, which also positions the RAID data superblock at the end of the device: Sub-Version Superblock Position on Device --- - 0.9 At the end of the device 1.0 At the end of the device 1.1 At the beginning of the device 1.2 4K from the beginning of the device To bypass the swap format you could try either deselecting the format option (if it exists) or setting the partition type to something else. The partition type can be set back to swap later from a livecd without having to reinstall. Other option: 1 install to single disk 2 using sysresccd create a degraded raid1 using the 2nd drive 3 copy the partitions and date from drive 1 to the degraded raid device What is copy the date? 4 add disk 1 to the raid I might miss something but I guess you're going to erase the installed system (on disk 1) from the unused disk (disk 2), here. I believe it would only be possible by installing the system on the degraded RAID, which will likely mean coming back to the original swap problem. 5 wait for the raid device is synchronized 6 change fstab and grub config to reflect the new disklayout -- Nicolas Sebrecht
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
Nicolas Sebrecht nsebre...@piing.fr wrote: The 21/10/13, J. Roeleveld wrote: Other option: 1 install to single disk 2 using sysresccd create a degraded raid1 using the 2nd drive 3 copy the partitions and date from drive 1 to the degraded raid device What is copy the date? A typo. I meant to say copy the data. 4 add disk 1 to the raid I might miss something but I guess you're going to erase the installed system (on disk 1) from the unused disk (disk 2), here. No. At this point, the raid (with disk2) has a copy of disk1. I believe it would only be possible by installing the system on the degraded RAID, which will likely mean coming back to the original swap problem. That is why I suggested installing on a normal single disk. The copying over onto the degraded raid using disk2. 5 wait for the raid device is synchronized 6 change fstab and grub config to reflect the new disklayout -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
On Monday 21 Oct 2013 09:55:42 J. Roeleveld wrote: Nicolas Sebrecht nsebre...@piing.fr wrote: I believe it would only be possible by installing the system on the degraded RAID, which will likely mean coming back to the original swap problem. That is why I suggested installing on a normal single disk. The copying over onto the degraded raid using disk2. I'm fast gravitating towards this option ... Although with metadata 0.90 I was able to progress with the installation (after I deselected the swap partitions) the grub-install script wanted to install in /dev/md127p1 but it failed. I had to override the Ubuntu installer since I could only install grub in the /dev/md127 block device. BTW, I'm still at a loss as to why for Ubuntu the RAID 1 is seen as /dev/md127 and not /dev/md0 which I created originally with sysrescuecd. Either way, it won't boot again. Now it stays on a blank screen, no error at all shown. I'll have another go with sysrescueCD to see if I install grub on /dev/sda and /dev/sdb and if this does not work either, I'll stop wasting time and follow your suggestion of installing on a single disk first, before I mirror it thereafter. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
[gentoo-user] Re: [O/T] RAID help - now won't boot
On Wednesday 16 Oct 2013 21:14:38 Nicolas Sebrecht wrote: On Wed, Oct 16, 2013 at 08:10:40PM +0200, Nicolas Sebrecht wrote: On Tue, Oct 15, 2013 at 10:42:18PM +0100, Mick wrote: mdadm --create --auto=mdp --verbose /dev/md_d0 --level=mirror --raid-devices=2 /dev/sda /dev/sdb which is thereafter partitioned with fdisk. This is the one I have used in the past. Which one is preferable, or what are the pros cons of each? For a basic RAID1, the best is to keep it as simple as possible. So mirroring while disk looks better. It will also keep MBR/GPT synced. s/while/the whole/ I tend to make manual partitions that I mirror but this is because I usually require to do more complex setups (e.g. mixing mirror types), or because I need to have the setup more flexible. OK, I spent some time to experiment in a VM. Two small un-partitioned virtual disks which I used to create /dev/md0 as RAID 1 using sysrescuecd. Then I used fdisk to create a MSDOS partition table on /dev/md0, followed by 4 partitions on /dev/md0: == ~$ fdisk -l Disk /dev/sda: 10.5 GB, 10522460160 bytes 255 heads, 63 sectors/track, 1279 cylinders, total 20551680 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x Disk /dev/sda doesn't contain a valid partition table Disk /dev/sdb: 10.5 GB, 10522460160 bytes 255 heads, 63 sectors/track, 1279 cylinders, total 20551680 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x Disk /dev/sdb doesn't contain a valid partition table Disk /dev/md0: 10.5 GB, 10521337856 bytes 2 heads, 4 sectors/track, 2568686 cylinders, total 20549488 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000c3148 Device Boot Start End Blocks Id System /dev/md0p1 *2048 718847 358400 83 Linux /dev/md0p2 718848 3790847 1536000 82 Linux swap / Solaris /dev/md0p3 379084818470911 7340032 83 Linux /dev/md0p41847091220549487 1039288 83 Linux == So, no partition tables on /dev/sda or /dev/sdb drives and of course no partitions at all. The partitions were created on the /dev/md0 block device. I then rebooted with a Ubuntu server CD and installed the OS in the above filesystem. It seemed to have recognised the RAID1 array as /dev/md127, instead of /dev/md0. Trying to install GRUB on /dev/sda, or /dev/sdb, or /dev/md127p1 failed. The only way to install GRUB and complete the Ubuntu server OS installation was to install it on /dev/md127, which it accepted. However, on rebooting it failed with: FATAL: No boot medium found! System halted. Rebooting with sysrescueCD and selecting to scan and boot any linux OS it could find, it picks up the RAID1 installation and it boots into it without any problem. This is what I can see now: == ~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 9.8G 0 disk └─md0 9:00 9.8G 0 raid1 ├─md0p1 259:00 350M 0 md/boot ├─md0p2 259:10 1.5G 0 md[SWAP] ├─md0p3 259:20 7G 0 md/ └─md0p4 259:30 1015M 0 md/home sdb 8:16 0 9.8G 0 disk └─md0 9:00 9.8G 0 raid1 ├─md0p1 259:00 350M 0 md/boot ├─md0p2 259:10 1.5G 0 md[SWAP] ├─md0p3 259:20 7G 0 md/ └─md0p4 259:30 1015M 0 md/home == == ~$ df -h -T Filesystem Type Size Used Avail Use% Mounted on /dev/md0p3 ext4 6.9G 1.2G 5.4G 18% / udev tmpfs 10M 8.0K 10M 1% /dev none tmpfs 146M 352K 146M 1% /run none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 730M 0 730M 0% /run/shm /dev/md0p1 ext2 329M 27M 285M 9% /boot /dev/md0p4 ext4 999M 18M 931M 2% /home == == ~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid1 sda[0] sdb[1] 10274744 blocks super 1.2 [2/2] [UU] unused devices: none == == ~$ sudo blkid /dev/sr0: LABEL=sysrcd-3.8.0 TYPE=iso9660 /dev/sda: UUID=59195572-751a-3bd9-7771-6e5411b032c8 UUID_SUB=3acd1b2c-1c95-7c07-a8b2-8aa1b2a0a169 LABEL=sysresccd:0 TYPE=linux_raid_member /dev/sdb: UUID=59195572-751a-3bd9-7771-6e5411b032c8 UUID_SUB=c63e97ba-42cb- c4f8-550d-f1effae33d3f LABEL=sysresccd:0 TYPE=linux_raid_member /dev/md0p1: UUID=d9dbe2bc-0453-46e4-a5b0-779e55246004 TYPE=ext2 /dev/md0p2:
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
Am 20.10.2013 11:54, schrieb Mick: Any ideas why the Ubuntu installation won't boot? My guess would be, you cannot boot, because if you install grub in /dev/md0. Upon boot the bios cannot find stage1 of the bootloader, which normally lies in the MBR (which also houses the partition table). Is a setup as you wish - sda and sdb as raid1, and partitions only on md0 - bootable in general? signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
On Sunday 20 Oct 2013 13:57:34 Michael Hampicke wrote: Am 20.10.2013 11:54, schrieb Mick: Any ideas why the Ubuntu installation won't boot? My guess would be, you cannot boot, because if you install grub in /dev/md0. Upon boot the bios cannot find stage1 of the bootloader, which normally lies in the MBR (which also houses the partition table). I see ... so installing the MBR code in the /dev/md0 block device is further down the disk than where BIOS is looking for it and that's why it errors out? Meanwhile, there is no MBR or partition table on /dev/sda or /dev/sdb for BIOS to jump to. Hmm ... It seems to me then that I *have* to create normal partitions on /dev/sda /dev/sdb, or I would need a different boot drive. Is there another way to overcome this problem. Is a setup as you wish - sda and sdb as raid1, and partitions only on md0 - bootable in general? Yes, in this case. It makes easy to set a faulty drive in failed state and remove it from RAID in a single step, rather than the alternative which would involve removing multiple /dev/mdX, one for each partition. I could I guess install LVM on top of RAID, but this adds complexity for a functionality (increasing LV sizes) which will not be used in this implementation. Any suggestions welcome. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
Am 20.10.2013 15:13, schrieb Mick: On Sunday 20 Oct 2013 13:57:34 Michael Hampicke wrote: Am 20.10.2013 11:54, schrieb Mick: Any ideas why the Ubuntu installation won't boot? My guess would be, you cannot boot, because if you install grub in /dev/md0. Upon boot the bios cannot find stage1 of the bootloader, which normally lies in the MBR (which also houses the partition table). I see ... so installing the MBR code in the /dev/md0 block device is further down the disk than where BIOS is looking for it and that's why it errors out? That would be my guess. Maybe someone more knowledgeable on how mdadm writes stuff on the disk can jump in and provide additional info. But I'm pretty sure, if you install grub in md0, it's not in that place on the disk where the bios is actually looking for. It seems to me then that I *have* to create normal partitions on /dev/sda /dev/sdb, or I would need a different boot drive. Is there another way to overcome this problem. Maybe create two mds. md1 (sda1, sdb1) is a small boot partition which contains stage2+, the kernel and the initramfs. And md2 (sda2, sdb2) which acts as another block device with partition table, etc... In this setup you could install grub in the mbr of sda and sdb (grub-install /dev/sda...) A quick google on this subject returned no usable results. But I am off now until tomorrow. signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
Michael Hampicke m...@hadt.biz wrote: Am 20.10.2013 15:13, schrieb Mick: On Sunday 20 Oct 2013 13:57:34 Michael Hampicke wrote: Am 20.10.2013 11:54, schrieb Mick: Any ideas why the Ubuntu installation won't boot? My guess would be, you cannot boot, because if you install grub in /dev/md0. Upon boot the bios cannot find stage1 of the bootloader, which normally lies in the MBR (which also houses the partition table). I see ... so installing the MBR code in the /dev/md0 block device is further down the disk than where BIOS is looking for it and that's why it errors out? That would be my guess. Maybe someone more knowledgeable on how mdadm writes stuff on the disk can jump in and provide additional info. But I'm pretty sure, if you install grub in md0, it's not in that place on the disk where the bios is actually looking for. It seems to me then that I *have* to create normal partitions on /dev/sda /dev/sdb, or I would need a different boot drive. Is there another way to overcome this problem. Maybe create two mds. md1 (sda1, sdb1) is a small boot partition which contains stage2+, the kernel and the initramfs. And md2 (sda2, sdb2) which acts as another block device with partition table, etc... In this setup you could install grub in the mbr of sda and sdb (grub-install /dev/sda...) A quick google on this subject returned no usable results. But I am off now until tomorrow. I would suggest trying it by usong the older metadata format. Check the man pages, but I thinl it would be --metadata=0.90 (or similar) during creation. That might put the metadata at the end, rather then at the front. (Or it's the other way round and new metadata does it at the end.) -- Joost Ps. I have never tried it this way (full disk raid for boot device) using linux software raid. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] Re: [O/T] RAID help - now won't boot
On Sunday 20 Oct 2013 15:31:12 jo...@antarean.org wrote: I would suggest trying it by usong the older metadata format. Check the man pages, but I thinl it would be --metadata=0.90 (or similar) during creation. That might put the metadata at the end, rather then at the front. (Or it's the other way round and new metadata does it at the end.) -- Joost Ps. I have never tried it this way (full disk raid for boot device) using linux software raid. Ha! Yes, this made a difference, thanks! With metadata 0.90 I can see the same partitions I set up on /dev/md0, also on /dev/sda and /dev/sdb. The only problem now is that the Ubuntu server CD wants to format /dev/sda2 as swap and fails at that stage. :-/ Not sure how to by-pass this. I may also try metadata=1.0 to see if this makes a difference, which also positions the RAID data superblock at the end of the device: Sub-Version Superblock Position on Device --- - 0.9 At the end of the device 1.0 At the end of the device 1.1 At the beginning of the device 1.2 4K from the beginning of the device -- Regards, Mick signature.asc Description: This is a digitally signed message part.
