All, Devs, This affects a normal boot and a boot from the Arch ISO.
I had a odd occurrence today where Arch boot failed and for all practical purposes it appeared there was a serious hardware failure leading to some type of panic/freeze but there was no outward indication or any error displayed. The Arch ISO behaved the same way.
There was apparently a hard, unexplained, shutdown at 11:09:00 CST on this server. The / filesystem is RAID1 (mdadm) on an ext4 filesystem, the bootloader written to each individual disk.
On reboot, the boot proceeded normally through the "Recovering journal", then the "root clean ......." message, and then the small underline blinking cursor below "root clean" showed activity for about 15 seconds, then froze. After several minutes, I shut the server down.
I downloaded a fresh 7/01 Arch install iso, dd'ed it to a USB and booted from it choosing the x86_64 install/maintenance entry. The init image loaded fine, but then the loading of Arch froze, in the same manner, appearing like a hardware freeze.
Waiting several minutes, with nothing but rising CPU temp and fan noise, I shut down and booted back into the install ISO and ran memtest. An hour later with no errors and 60% of Pass 1 done, it was obviously not a memory problem.
Reboot the install ISO and run the hardware check, all drives report good.
Reboot the install ISO again and try the Install/Maintenance option and this time it booted into Arch, there was a flashed message about RAID running in degraded mode. A quick check of /proc/mdstat showed / in resync ~50% complete, all other arrays working fine. Another 90 seconds or so and the resync completed normally. Then a reboot to the system drives worked normally.
So from what I can deduce there was a momentary power problem (UPS has been flawless, but apparently not foolproof). The end of the journal at 11:09:00 was unremarkable, it just ended. So my best guess is the / array was left in an inconsistent state requiring resync, otherwise all arrays were fine.
The question I have is how does the Arch boot process handle this? Or is this just one of the quirky corner cases that there isn't any good way to handle?
Had there been an indication that the / filesystem couldn't be loaded -- that would have made sense and clued me into the fact that / needed investigation. However, apparently when mdadm is conducting a resync, there is no message generated to inform the user to wait, and no timeout whatsoever. The box for all purposes just appears frozen.
Adding to the confusion was the Arch ISO which showed the same behavior (in hindsight it makes sense it would). The ISO automatically assembles the mdadm arrays, which left it in the same state waiting on / with no error and no timeout.
This may just be an odd corner-case, where the both disks in the array are good, not in degraded mode, but in resync. The boot process does not seem to handle that in any way except waiting for / to become available.
I don't have "systemd" in the /etc/mkinitcpio.conf HOOKS array, so this may just apply to a busybox init if that makes any difference (I don't know what the ISO uses). Should this be handled somehow so a warning is generated? Is that something Arch can do or will it take upstream mdadm changes or the like?
Sorry for the length, it got longer than anticipated, but I wanted to make sure I included enough so the scenario made sense.
-- David C. Rankin, J.D.,P.E.
