Am 21.06.26 um 01:23 schrieb Paul Leiber:
Am 20.06.26 um 20:00 schrieb Charles Curley:
On Sat, 20 Jun 2026 18:01:15 +0200
Paul Leiber <[email protected]> wrote:
Somehow, I missed to include the RAID1 information for md0 to the
configuration file (e. g. by entering root@localhost:~# mdadm
--detail --scan /dev/md0 >> /etc/mdadm/mdadm.conf). I am not sure if
this actually is the cause and adding that information would solve
the issue.
My questions are the following:
1. Is my analysis valid in principle? Especially: Could the root
cause for this issue be that mdadm.conf is missing the information
for md0, and could adding that information prevent data loss or
inconsistencies in the future?
I doubt that this is the culprit. the man page for mdadm says, in part:
Assemble
Assemble the components of a previously created array
into an active array. Components can be explicitly given or can
be searched for. mdadm checks that the components do form a
bona fide array, and can, on request, fiddle superblock
information so as to assemble a faulty array.
So mdadm *should* find both devices. But it might not be. And adding
that line will not hurt. I have a similar line in my mdadm.conf.
I built my RAID array up a bit differently that you did yours. You made
your partitions, put LUKS on the partitions, then the RAID on top of
that. I have the partitions, then the RAID array, LUKS on top of that,
then LVM, with file systems on top of the LVs. But I know of no reason
your setup shouldn't work.
I have found that when I have multiple LUKS partitions, giving them
all the same passphrase means I need give only one passphrase to
decrypt on boot.
2. Can I (re)create the RAID1 md0 or (re-)add the missing partition
in an easy way that no or at least not all information is lost? If
yes, how?
Yes. For the gory details see
https://oneuptime.com/blog/post/2026-03-02-how-to-replace-a-failed-disk-in-mdadm-raid-on-ubuntu/view.
In short,
* Fail the offending disk. It looks like this has already happened, but
it shouldn't hurt to do it again.
* Remove the disk from the array.
* Add the disk back in again. This should trigger rebuilding, which
takes a while. During the rebuild, the data should be both readable
and writable. You may monitor with:
cat /proc/mdstat
I just noticed that I didn't manage to make clear that (1) I don't think that
there is one specific failed partition, but that both partitions containing
databases seem to work, but not at the same time, and that (2) I want to keep
the data on the seemingly failed device.
Let me explain with an example: I am using KODI to access my video data from different devices. A couple of months ago, I switched KODI to using a centralized database (containing metadata information on movies, watch status, etc.) in order to maintain only one database instead of a database on each device running KODI. The data is stored on the database VM, running MariaDB, which stores the data on the md software RAID1 (at least that's what was supposed to happen). I spent some time configuring the metadata, e.g. correcting mistakes in the movie titles etc. I then noticed that I mistakenly selected English language to display the movie descriptions. Because of family members not fluent in English, I redid the metadata configuration in German. (It was an annoying work, therefore I remember it well.) Then, some time later, after a reboot of the hypervisor (and the database VM) due to kernel updates, the language of the movie descriptions was displayed in English again. I
attributed this to a corrupt database after the database VM reboot and loaded a database backup from some time ago, where the movie description was still in English. So I did the metadata configuration a third time, again in German. (I guess you can imagine the fun I had.) Then, a couple of days ago, after a reboot of the hypervisor and the database VM, the KODI movie description was displayed in English again. That's when I really started digging, because now it was clear that there were actually two intact, but differing databases. (To be clear: There were some other changes to other databases that also were affected in a similar manner which I don't mention in this example, so this issue is not restricted to the KODI database).
Based on the available data, I attribute this issue to the RAID1 which seems to
select one of two partitions at random when booting the hypervisor. Indications:
- The last update time in the description of the (seemingly) failed device given by mdadm --examine
match the point in time of the switch from one database version ("German") to the other
("English"), therefore I assume that the switch happens at the software RAID level.
- A failure at hardware level doesn't seem likely, because how could there
suddenly be an older version of a database available in a RAID1 if one device
fails and the RAID1 is degraded, and this after entirely rebuilding the
database from a backup? And, mind you, this switch to an older version of the
database didn't happen just once, but at least two times. The data (in English)
simply shouldn't have been available anymore at this point if the RAID1 had
been working as intended.
The most likely explanation to me is that the RAID1 has been running in a
degraded state for some time (unnoticed by me), the database changes (e. g.
from English to German) were stored to just one of the two partitions, and at
some point the RAID1 switched to the other partition after a reboot, containing
intact, but older (e. g. English) data. As a defective hardware doesn't seem
likely, I assume that something in my setup causes this behaviour by md. But of
course, I might be wrong and I am open to other explanations. For example, what
my assumption fails to explain is why the switch only happens from time to
time, and not more often, e.g. after each reboot.
The example you kindly give is for removing a seemingly failed partition (currently dm-30, "German"
database) from a md RAID1, keeping the data on an intact partition (currently dm-31, "English"
database) and than re-fgadding a partition to the RAID1. This is pretty straightforward: the data is kept and
replicated from the valid partition to the freshly added one. However, in my case, the dataset I want to keep
is on the seemingly failed partition not used in the RAID (currently dm-30, "German").
Options I see (besides recreating the RAID1 from scratch and using an available
backup to restore the data, losing some data):
1. I could fail the seemingly intact partition or remove the RAID1 entirely, somehow use
the seemingly failed partition (dm-30, "German") to create a new RAID without
losing the data on it, then add the other partition (dm-31) as a new drive and have the
data replicated. I am not sure if this is possible, therefore my question to this list.
2. Another option is to reboot the hypervisor and hope for a switch of the RAID to the
partition containing the more recent version of the database, then follow your guide. But
I am not really confident that such a "strategy" is the best choice I have at
the moment. Also, I just tried a reboot three times, each time the data in the database
is the wrong, old one.
3. I could also backup the database from the seemingly failed partition in
order to not lose data and then use this backup to recreate the RAID1, but I
would need to mount that partition, which ended in an error when I tried it.
And, of course, I don't want this to happen again, therefore I want to find the
root cause for this situation and fix it. If it is not the missing information
in /etc/mdadm/mdadm.conf, what else could it be?
Sorry for the lengthy posts, I don't know how to describe this situation
clearly in a shorter way.
I managed to rebuild the md RAID1 using the data on the seemingly failed device
(partition_1). First, I did a dd dump of partition_2 (currently in usage) in
order not to lose data. Then, I recreated the RAID1 using the data from
partition_1:
mdadm --stop /dev/md0 # This stops the degraded RAID1
mdadm --assemble --update=uuid /dev/md0 /dev/mapper/partition_1 # This creates
a new RAID1 using the partition_1, a new array UUID is required in order for
--assembly to work
mdadm --manage --add /dev/md0 /dev/mapper/partition_2 # This adds partition_2
to the RAID1, contents of partition_1 are replicated to partition_2
automatically
md is currently replicating the data (German movie descriptions in Kodi, yay!) from partition_1 to
partition_2. I might have to turn partition_2 from "spare" to "active", but
I'll let the replication complete first.
In any case, I set up mdmonitor to alert me if the RAID1 degrades again. That's
something I should have thought of earlier.
We'll see if this issue occurs again. I'll give an update if this is the case.
Thanks to everybody for trying to help me!
Paul
