Hi here is my account of setting up raid-1 over an existing system.
Setting up Raid-1 over an existing system
Brian Denheyer
[EMAIL PROTECTED]
Much of this information can be found in both the "new" raid how-to
http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/
and the disk upgrade mini how-to.
http://metalab.unc.edu/mdw/HOWTO/mini/Hard-Disk-Upgrade.html
I personally find it very instructive to see real world examples with
all of the trifling details, like where files are mounted, what
commands are used to copy things, etc... I hope others who take the
plunge will find this account of my experience helpful.
I did make my mirror device the root partition, although I did not try
to get things to boot from the mirror device.
The how-to authors should feel free to take anything from here which
might be useful or helpful and incorporate it into the how-to's.
Caveats
-------
If you do something wrong you could trash all the data on your
harddisks. Make back-ups. Don't blame me if something goes wrong.
Half the time I wasn't even sure it was going to work. But I have a
4GB DAT tape with all of my whole system on it, so I wasn't worried.
Well, not too worried.
Please be careful and think about what you're doing. I found the
biggest problem to be file confusion. Getting hda/hdb/hdc mixed up.
Modifying /etc/fstab when I meant to modify /new/etc/fstab and other
such nonsense. So take it slow and double-check your work.
Start
-----
I started with 2 20.5GB IBM IDE disk drives.
The BIOS was configured to autodetect the disk drives. This makes it
easy to swap disks since you don't have to go into the BIOS set-up to
redefine the disk type. Whenever you swap disks be sure and set those
jumpers to master or slave as appropriate.
It's also important to set the MODE of the disk drives to NORMAL and
not to LARGE or LBA. I found that this made the geometry detection
more uniform, i.e. the drives were detected with the same geometry.
So now I have :
/dev/hda : my original boot/root disk
/dev/hdb : mirror drive 0
/dev/hdc : mirror drive 1
My first attempt
----------------
My first attempt resulted in a major problem. The boot process would
always stop with errors on the mirror devices. Apparently the number
of sectors in the superblock and physical number of sectors didn't
agree.
I believe the following may have caused this problem, but I don't
really know.
1. The BIOS mode was set to something other than NORMAL which caused
the disk geometries to be read differently even though the disks
were the same.
2. My lack of understanding led me to create filesystems on the
individual partitions (Don't do that !).
So after much thought I decided to give it one more try and started
from scratch. Essentially I followed the "new and improved" raid
how-to.
Patches
-------
Since I didn't see any patches available for 2.2.14, I downgraded my
kernel from 2.2.14 to 2.2.10. I then obtained the
raid0145-19990724-2.2.10 patch file and applied the patch. I'm using
a debian system and it currently provides the latest version of
raidtools (0.90 version) as a package. You'll find raidtools2 in the
unstable part of the distribution.
apt-get install raidtools2
Sadly, many people do not use Debian and will need to fetch the source
for the tools and build.
It's very important that the boot scripts execute raidstart to set-up
the mirror devices once they are up and running. After you have
auto-detection working, you can remove those same scripts.
Apply the patches and build raid support into the kernel.
cd /usr/src/linux
patch -p0 < raid0145-19990724-2.2.10
Rebuild the kernel being sure to add support for RAID-1 and raid
auto-detection. Do the usual kernel build stuff and reboot so that
you are running your shiny, new raid-aware kernel.
I used cfdisk and partitioned the disks for use as :
device mirror device mount-point
------ ------------- -----------
/dev/hd[bc]1 /dev/md0 /
/dev/hd[bc]2 /dev/md1 /usr
/dev/hd[bc]3 /dev/md2 /usr/local
/dev/hd[bc]5 /dev/md3 /home
Ans saved some space for BIG swap.
/dev/hd[bc]6 swap
Stupid mistake alert. Use "cfdisk /dev/hdb" NOT "cfdisk /dev/hdb1".
I kept doing this by accident and couldn't figure out why cfdisk
wouldn't start.
Another Stupid mistake alert. Do NOT install a filesystem on the
partitions you just created.
Make Raid
---------
I then set-up my /etc/raidtab file. Please see the end of this
discussion for the file contents.
IMPORTANT! Be sure and use a persistent-superblock setting of 1 so
that the mirror will be useful the next time you boot.
Make the partitions and then move on to the easy part. Make the raid
devices by using mkraid.
mkraid /dev/md0
mkraid /dev/md1
mkraid /dev/md2
mkraid /dev/md3
Here's a question : how do you speed the re-sync process ?? Re-nicing
didn't seem to work and the big partitions take a LONG time.
The amazing thing is that you can use the partitions while they are
re-syncing. So I decided to install filesystems :
mkfs.ext2 -b 4096 -R stride=4 /dev/md0
mkfs.ext2 -b 4096 -R stride=4 /dev/md1
mkfs.ext2 -b 4096 -R stride=4 /dev/md2
mkfs.ext2 -b 4096 -R stride=4 /dev/md3
I have found that the stride doesn't do didly for performance so you
can omit the stride parameter when installing the filesystem on the
mirror devices.
To be safe I waited until the re-sync process finished. It takes
forever... Just check /proc/mdstat every now and again to see how it's
going.
Once it was finished with the re-sync, do an fsck of the filesystems
to make sure they're clean.
Moving Files
------------
You then have to start moving files from your "old" system to the
mirror devices. The "Hard Disk Upgrade Mini How-To" is VERY helpful
in this regard. Read it carefully, especially the section about LILO.
Here is an example. I want to transfer /usr/local to the mirror
device which is destined for use as /usr/local.
mkdir /newusrlocal
mount /dev/md2 /newusrlocal
cp -a /usr/local/* /newusrlocal
umount /usr/local
umount /newusrlocal
mount /dev/md2 /usr/local
/usr/local is now on the mirror device. Modify the /etc/fstab
appropriately to get the filesystem mounted at boot-time.
I then copied the rest of the files over, set-up my fstab
appropriately so that the filesystems would be mounted on the
appropriate mirror device and re-booted.
PRESTO! Everything worked just fine.
Hungry for challenge I turned my attention towards using the mirror
device for my root filesystem.
LILO : The Challenge
--------------------
LILO scares me. I don't understand _exactly_ what it's doing.
Whenever I change anything having to do with LILO I find myself
holding my breath, hoping the system will boot the way I want. So it
was no small leap of faith for me to attempt to use /dev/md0 as /.
First things first. The partitions which are used for the mirror
_must_ be marked as autodetected partitions (fd instead of 82) using
cfdisk. Please be sure that the partitions are auto-detected properly
_before_ you try to set-up the root partition on your mirrored device.
Once the partitions are auto-deteced you will no longer need the
raidstart stuff in the rc scripts to get raid started.
WARNING: be sure and unmount filesystems and perform raidstop before
you go and change the partition types ! You will be changing the
partition types of the underlying ide partitions, i.e. hdb/c in my
case. Be _sure_ and do a "umount /dev/md?" and "raidstop /dev/md?"
_before_ you change the partition types. Change the types to fd on
_all_ the partitions which are being used as mirror devices.
Once I was convinced auto-detection was working I put a file system on
/dev/md0, mounted it to /new, and then copied over the / files, being
very careful not to copy /usr, /usr/local, /proc, etc...
Once done, you have to be sure and create the directories you did not
copy, like /proc for instance.
WARNING: I use the hda drive to boot from, i.e. hdb/c are the mirrors.
You are NOT going to boot from the mirror with this set-up. I'm too
chicken, and I've been staying up too late as it is, to try patching
LILO and booting from a mirrored device.
Now the moment of truth. Here is the LILO file _exactly_ as I used it
to create the new boot sequence. Please not that /boot is still on my
original disk, i.e. /dev/hda :
verbose=1
boot = /dev/hda
# /sbin/lilo (the installer) copies the Lilo boot record
# from the following file to the MBR location.
install = /boot/boot.b
#
# I wrote a verbose boot menu. Lilo finds it here.
message = /boot/message
# The installer will build the following file. It tells
# the boot-loader where the blocks of the kernels are.
map = /boot/map
compact
prompt
# Wait 5 seconds, then boot
timeout = 50
image = /boot/vmlinuz-2.2.10
label = 2.2.10
root = /dev/hda1
read-only
image = /boot/vmlinuz-2.2.10
label = m2.2.10
root = /dev/md0
read-only
image = /boot/vmlinuz-2.2.14
label = 2.2.14
root = /dev/hda1
read-only
Notice that the only change from what would be considered a "normal"
config file, is the redirection of the root filesystem to /dev/md0.
Please note that the _second_ image is the mirror root partition. So
the default boot, and therefore the escape hatch, is to set-up
/dev/hda1 as the / filesystem. If the mirrored filesystem doesn't
work right, you can fall back to the original root filesystem to fix
things.
Modify the /new/etc/fstab file to reflect the fact that you will be
putting / on /dev/md0. Don't forget to do this and don't modify
/etc/fstab by accident !!
Run lilo. Reboot.
Wen the lilo prompt comes up, press tab and try using "m2.2.10". If
you've done everything correctly, it will work. It did for me :-) If
not, drop back to the original / location, on /dev/hda1, take a deep
breath, and try to figure out what's screwed-up.
Now the trickiy part. Once / is on /dev/md0, you need to modify lilo
again to make /dev/md0 the root filesystem as the default case.
Here's where my lack of LILO knowledge is really a problem. The above
file will no longer work. LILO bails out with an error : "don't know
how to handle device 0x0900" or some such nonsense.
Here is what I did to get around this. It's probably not the correct
way to do it, but it does work. Remember that /dev/hda1 still holds
the original / filesystem. So create a home for it and mount it :
mount /dev/hda1 /mnt
Point all the files in the lilo.conf file to /mnt/boot, like so :
verbose=1
boot = /dev/hda
# /sbin/lilo (the installer) copies the Lilo boot record
# from the following file to the MBR location.
install = /mnt/boot/boot.b
#
# I wrote a verbose boot menu. Lilo finds it here.
message = /mnt/boot/message
# The installer will build the following file. It tells
# the boot-loader where the blocks of the kernels are.
map = /mnt/boot/map
compact
prompt
# Wait 5 seconds, then boot
timeout = 50
image = /mnt/boot/vmlinuz-2.2.10
label = m2.2.10
root = /dev/md0
read-only
image = /mnt/boot/vmlinuz-2.2.10
label = 2.2.10
root = /dev/hda1
read-only
image = /mnt/boot/vmlinuz-2.2.14
label = 2.2.14
root = /dev/hda1
read-only
Notice that the order is switched so that /dev/md0 will be used for /
by default. Now run lilo. It installs and the next time you boot, /
will be on /dev/md0.
Now go get some sleep.
Here is a listing of my raidtab file I used to get things going.
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 16
persistent-superblock 1
device /dev/hdb1
raid-disk 0
device /dev/hdc1
raid-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 16
persistent-superblock 1
device /dev/hdb2
raid-disk 0
device /dev/hdc2
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 16
persistent-superblock 1
device /dev/hdb3
raid-disk 0
device /dev/hdc3
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 16
persistent-superblock 1
device /dev/hdb5
raid-disk 0
device /dev/hdc5
raid-disk 1
