[Bug 324997] Re: System startup fails with degraded RAID + encryption
In my original bug report, I used 8.04.1 that had been live upgraded to 8.04.2. Recently, I tried installing the same setup (RAID1 + encryption + LVM) on another computer, this time directly from Ubuntu 8.04.2 Alternate installation disk. Installation went fine, but in the very first boot after installation the computer did not start. I got these messages when I rebooted without quiet splash boot options: md: invalid raid superblock magic on sda raid array is not clean -- starting background reconstruction check root= bootarg cat /proc/cmdline or missing modules, devices: cat /proc/modules ls /dev ALERT! /dev/mapper/lvm_group-lvm_root does not exist I got dropped to busybox. ls /dev/mapper did not reveal any md devices, so the error seemed correct. Then I figured to test cat /proc/mdstat. Everything was fine there, both disks up, resync ongoing... First I could not figure out what was going on as the same setup method had previously worked. Then I realized that I was now using 8.04.2 installation disk, which includes some changes for degraded RAID boot procedure. I had already filed this bug, related to degraded RAID and encryption. And, what do you know, after typing the same fix in Busybox cryptsetup luksOpen /dev/md1 md1_crypt everything was going to be ok again: with ls /dev/mapper I could see my devices. I waited until RAID sync was over (cat /proc/mdstat shows the progress) and then booted with CTRL+D. It worked. I rebooted, and this time LUKS password was queried and system booted correctly. So, what happened is very much related to this same bug: when you have LUKS encryption on top of RAID 1 array, and the array gets degraded either because of a disk is missing or, as in this case, needs to be resynced, the system won't start. Now it seems even more fatal, as the system fails to boot the very first time after installation, even if both disks are present and working. And once again, error messaged do not give hints to correct direction. In this case, the problem could be handled at least partially by printing a message that tells that RAID resync is going on, and you can check its progress with this command, and when it is finished you can restart the computer and then it will boot normally. Currently, the user gets confused and probably thinks that the installation did not work at all, although there's nothing wrong with it. -- System startup fails with degraded RAID + encryption https://bugs.launchpad.net/bugs/324997 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Degraded RAID 1 array with encryption, Bug #324997. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Server Team, which is a subscriber of a duplicate bug. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Degraded RAID 1 array with encryption, Bug #324997. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 324997] [NEW] System startup fails with degraded RAID + encryption
Public bug reported: Binary package hint: mdadm Release: Ubuntu 8.04.2, from alternate installation media. Probably also affects Ubuntu 8.10. Module: mdadm. Possibly affects others as well. Version: 2.6.3. Probably also affects 2.6.7, which is used in 8.10. Situation: I have installed \root and \swap to RAID 1 partitions with encryption and LVM. When one of the RAID disks is missing from the system, a prompt should ask whether I want to start with a degraded RAID setup or not. Answering Yes should start up the system. What happens: This depends on whether quiet splash boot options are in use or not, as follows: - When quiet splash is used, a prompt for LUKS encryption password appears normally, but fails again and again even if the user types the correct password. At this point the user probably thinks that the password is not correct, or that there's something wrong in encryption. Both assumptions are incorrect, as in fact there is nothing wrong with encryption nor the password... Nevertheless, the system fails to start. By pressing CTRL+ALT+F1, one can see these messages on screen, which also lead to wrong direction: Starting up ... Loading, please wait... Setting up cryptographic volume md1_crypt (based on /dev/md1) cryptsetup: cryptsetup failed, bad password or options? cryptsetup: cryptsetup failed, bad password or options? - Without quiet splash boot options, the user can observe system messages on screen during boot process. At a certain point, the missing RAID disk causes a long wait. During this wait, one can see these messages on screen: Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? ... other stuff ... Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? cryptsetup: maximum number of tries exceeded Done. Begin: Waiting for root file system... ... After a few minutes, the system prompts whether the user wants to start with a degraded RAID setup. After answering Yes and another few minutes of waiting, the system presents the command line (Busybox), ie. fails to start. This happens probably because \root is encrypted and needs to be opened with a password, however the password prompt already failed during the long RAID wait period. Note that it is possible to start the system from Busybox by typing cryptsetup luksOpen /dev/md1 md1_crypt, then typing LUKS password, and finally pressing CTRL+D. This proves that the encryption works and the problem is related to degraded RAID, handled by mdadm. Also the system starts properly when all RAID disks are present; the problem only appears with a degraded RAID. More information: For a long time, it was not possible to properly boot with a degraded RAID setup. This bug was never present in Debian, only in Ubuntu. See bug 120375. A solution was presented in 8.10, which was recently backported to 8.04.2. See bug 290885. Apparently, encryption was not dealt with the fix, and thus this bug likely affects all recent Ubuntu releases, including 8.10. I consider this quite important to be fixed, as many RAID 1 users also need to use encryption for protecting their data, especially in the business world. Although there is a workaround for a starting the system with degraded RAID and encryption, the error messages clearly lead to wrong direction (encryption is not the problem), and even if the user knows what's going on it is too complicated to look up the workaround by googling in a stressing situation where a production server fails to start. Suggestions: - When usplash is used, there should be a notification about a possible degraded RAID array *before* LUKS encryption password is prompted, so that the user has a possibility to figure out that the failure to boot is related to RAID disks, not encryption - When a degraded RAID is observed, LUKS password should actually be prompted *after* answering the question for starting the system with degraded RAID ** Affects: mdadm (Ubuntu) Importance: Undecided Status: New -- System startup fails with degraded RAID + encryption https://bugs.launchpad.net/bugs/324997 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Dustin, thank you for your quick answer and tips. It took me a while to test it, as I have an encrypted RAID 1 array with LVM, and things are not that straightforward with that setup. So far I have been using one of the tricks described in this thread earlier (ie. edit /etc/udev/rules.d/85-mdadm.rules: change the --no- degraded to -R, after that sudo update-initramfs -u -k all). It allows me to boot with only one drive, but has that annoying side effect where one of the partitions often starts in degraded mode, even if both drives are in fact present and working. I wanted to get rid of that problem, so I did this: - return 85-mdadm.rules as it used to be, ie. --no-degraded - sudo update-initramfs -u -k all - cat /proc/mdstat and check that all drives are online and sync'ed - upgrade all packages - re-install grub to both drives These are my test results: 1. Restart computer with both disks - everything works OK 2. Restart computer with only one disk - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password 3. Restart computer again with both disks - everything works OK So, first it seemed that the fix does not work at all, as Ubuntu starts only when both disks are present. Then I made some more tests: 4. Restart computer with only one disk - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password - Now press CTRL+ALT+F1, and see these messages: Starting up ... Loading, please wait... Setting up cryptographic volume md1_crypt (based on /dev/md1) cryptsetup: cryptsetup failed, bad password or options? cryptsetup: cryptsetup failed, bad password or options? - After waiting some minutes, I got dropped into the busybox - Something seems to be going wrong with encryption 5. Restart computer with only one disk, without quiet splash boot parameters in /boot/grub/menu.lst - Got these messages: Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? ... other stuff ... Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? cryptsetup: maximum number of tries exceeded Done. Begin: Waiting for root file system... ... - After waiting some minutes, I get the question whether I want to start the system with degraded setup. However, it does not matter what I answer, as the system cannot start since the encryption has already given up trying. I don't know what it was trying to read as a password, because I did not type anything. 6. Restart computer with only one disk, with quiet splash bootdegraded=true boot parameters in /boot/grub/menu.lst - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password - Now press CTRL+ALT+F1, and see these messages: Starting up ... Loading, please wait... Setting up cryptographic volume md1_crypt (based on /dev/md1) cryptsetup: cryptsetup failed, bad password or options? Summary: The fix does not seem to work, in case you have encrypted your RAID disks. To be more specific: after a long wait it does ask whether to start in degraded mode, although the question seems to appear only when booted without quiet splash parameters. I guess it also starts in degraded mode automatically if bootdegraded parameter is set. Nevertheless the system will not start, as this seems to happen too late for encryption has already given up. Question: Is this fix tested with encryption at all? Is it suppose to work with it, or not? I think this is important, as if you have a RAID setup you obviously have some important data and in many cases want to encrypt it, too. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Server Team, which is a subscriber of a duplicate bug. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Ok, I'm answering myself: there is a workaround for getting it to work with LUKS encryption. You can run sudo dpkg-reconfigure mdadm and enable automatic startup with degraded RAID array if you want, or watch the screen and be quick enough to answer Yes when asked to start degraded. Nevertheless, you need to wait again until you're dropped to BusyBox. Then do this: # to enter the passphrase. md1 and the md1_crypt are the same values # you had to put in /target/etc/crypttab at the end of the install cryptsetup luksOpen /dev/md1 md1_crypt # (type your LUKS password, as requested) # continue to boot! hit CTRL+D I found the instructions from here: http://ubuntuforums.org/archive/index.php/t-524513.html Now, if only someone could give a hint on how to make this automatic, so that there would be no need to write anything. It is ok to wait a few minutes, though. Nevertheless, I'm pretty happy now that I can use --no-degraded parameter in 85-mdadm.rules, yet get the system up in case a disk fails. In the rare case of an actual disk failure, writing a one-liner can be tolerated. Thank you everyone who have worked with this issue and helped to get it solved in Hardy. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Server Team, which is a subscriber of a duplicate bug. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Dustin, thank you for your quick answer and tips. It took me a while to test it, as I have an encrypted RAID 1 array with LVM, and things are not that straightforward with that setup. So far I have been using one of the tricks described in this thread earlier (ie. edit /etc/udev/rules.d/85-mdadm.rules: change the --no- degraded to -R, after that sudo update-initramfs -u -k all). It allows me to boot with only one drive, but has that annoying side effect where one of the partitions often starts in degraded mode, even if both drives are in fact present and working. I wanted to get rid of that problem, so I did this: - return 85-mdadm.rules as it used to be, ie. --no-degraded - sudo update-initramfs -u -k all - cat /proc/mdstat and check that all drives are online and sync'ed - upgrade all packages - re-install grub to both drives These are my test results: 1. Restart computer with both disks - everything works OK 2. Restart computer with only one disk - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password 3. Restart computer again with both disks - everything works OK So, first it seemed that the fix does not work at all, as Ubuntu starts only when both disks are present. Then I made some more tests: 4. Restart computer with only one disk - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password - Now press CTRL+ALT+F1, and see these messages: Starting up ... Loading, please wait... Setting up cryptographic volume md1_crypt (based on /dev/md1) cryptsetup: cryptsetup failed, bad password or options? cryptsetup: cryptsetup failed, bad password or options? - After waiting some minutes, I got dropped into the busybox - Something seems to be going wrong with encryption 5. Restart computer with only one disk, without quiet splash boot parameters in /boot/grub/menu.lst - Got these messages: Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? ... other stuff ... Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? Command failed: Not a block device cryptsetup: cryptsetup failed, bad password or options? cryptsetup: maximum number of tries exceeded Done. Begin: Waiting for root file system... ... - After waiting some minutes, I get the question whether I want to start the system with degraded setup. However, it does not matter what I answer, as the system cannot start since the encryption has already given up trying. I don't know what it was trying to read as a password, because I did not type anything. 6. Restart computer with only one disk, with quiet splash bootdegraded=true boot parameters in /boot/grub/menu.lst - Keeps asking Enter password to unlock the disk (md1_crypt): even though I write the correct password - Now press CTRL+ALT+F1, and see these messages: Starting up ... Loading, please wait... Setting up cryptographic volume md1_crypt (based on /dev/md1) cryptsetup: cryptsetup failed, bad password or options? Summary: The fix does not seem to work, in case you have encrypted your RAID disks. To be more specific: after a long wait it does ask whether to start in degraded mode, although the question seems to appear only when booted without quiet splash parameters. I guess it also starts in degraded mode automatically if bootdegraded parameter is set. Nevertheless the system will not start, as this seems to happen too late for encryption has already given up. Question: Is this fix tested with encryption at all? Is it suppose to work with it, or not? I think this is important, as if you have a RAID setup you obviously have some important data and in many cases want to encrypt it, too. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
Ok, I'm answering myself: there is a workaround for getting it to work with LUKS encryption. You can run sudo dpkg-reconfigure mdadm and enable automatic startup with degraded RAID array if you want, or watch the screen and be quick enough to answer Yes when asked to start degraded. Nevertheless, you need to wait again until you're dropped to BusyBox. Then do this: # to enter the passphrase. md1 and the md1_crypt are the same values # you had to put in /target/etc/crypttab at the end of the install cryptsetup luksOpen /dev/md1 md1_crypt # (type your LUKS password, as requested) # continue to boot! hit CTRL+D I found the instructions from here: http://ubuntuforums.org/archive/index.php/t-524513.html Now, if only someone could give a hint on how to make this automatic, so that there would be no need to write anything. It is ok to wait a few minutes, though. Nevertheless, I'm pretty happy now that I can use --no-degraded parameter in 85-mdadm.rules, yet get the system up in case a disk fails. In the rare case of an actual disk failure, writing a one-liner can be tolerated. Thank you everyone who have worked with this issue and helped to get it solved in Hardy. -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
It seems that the issue is now fixed in Intrepid, and also backported to Hardy 8.04.2, which was released a few days ago. However, it is unclear to me if I need to reinstall from 8.04.2 distribution media, or can I fix an existing Hardy installation via software updates? -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Server Team, which is a subscriber of a duplicate bug. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 120375] Re: cannot boot raid1 with only one disk
It seems that the issue is now fixed in Intrepid, and also backported to Hardy 8.04.2, which was released a few days ago. However, it is unclear to me if I need to reinstall from 8.04.2 distribution media, or can I fix an existing Hardy installation via software updates? -- cannot boot raid1 with only one disk https://bugs.launchpad.net/bugs/120375 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
