For those of us who install the Redhat , Fedora , ........etc flavors of linux the default install automatically does the LVM and make
the root (/) part of the LVM configuration therefore if you ever get any data corruption or disk issues on the disk holding root without a backup of you /etc/lvm location , you will have to go through the pain below................... Think the Veritas Volume manager& Solaris Volume Manager beat LVM at the way the store meta data on different disk partitions! Regards On Thu, Jan 14, 2010 at 2:31 PM, Noah Sematimba <[email protected]> wrote: > > Hi All, > > I was discussing my recent adventures with LVM2 with a friend who suggested > that I should post my recent experiences on the list perchance someone could > benefit from my experiences. So here goes: > > I had a server that someone else had setup (to my disgust) in the following > way: > > SUNFire X2100 with 2 250GB SATA disks. They created a small /boot partition > on sda1 for the boot stuff then created an LVM RAID 0 volume from the rest > of the space on sda2 and sdb1 which was a partition usign the whole of the > second disk, then then installed Redhat Enterprise Server 4, created two > logical volumes. One for root and another for swap. > > Now somewhere along the way, some "bright" chap realised that the server > had two disks but he was only seeing one in use :-) So the chap formatted > /dev/sdb1 to create it as a separate mount point. Since he did not reboot, > this mistake was never discovered. > > So one fine day, the data on the server grows beyond the capacity of the > 1st disk and LVM2 attempts to extend on to the second disk in the volume. > Predictably this results in data corruption, major LVM errors and the server > crashes and fails to reboot because the LVM metadata is corrupted. > > Now the fun part: There was no backup of this server. > > However the errors that LVM was throwing were actually showing cannot find > physical volume with uuid blah blah... so I actually had a uuid to work > with. > Variously on the web the suggestions were as follows: > > pvcreate -ff --uuid xxxx /dev/sdb1 > vgcreate VolGroup00 /dev/sda2 /dev/sdb1 > vgcfgrestore -f metadata-file VolGroup00 > > However I did not have a backed up copy of the metadata file. Reading a few > things on the web showed me that LVM keeps its metadata files at the > beginning of the disk, in the first 255 sectors following the partition > table in sector 1 of the disk. > So I booted up with the rescue disk and used foremost to try extract the > text of the config files form the raw device using pattern matching > following the suggestions on http://blog.eliasprobst.eu/?p=3 however this > did not work for me probably because as I discovered later, there was no > complete uncorrupted metadata file. However I realised I was being too cute > with my stuff and instead simply did: > > dd if=/dev/sda2 of=/dd.txt count=255 skip=1 bs=512 > > then vi /dd.txt. I found a bunch of binary stuff and snippets of config > files but no complete config file showing that the metadata files were > indeed corrupt. With a bit of cut and paste, here and there, I put together > a working config file. > > However I then run into my next hurdle. I can only assume that because the > disk was already full and the volume group configuration was still > corrupted, it meant that there was no space to put my configuration file > when I attempted to use vgcfgrestore as the logical disks were not available > to move the extra data onto the second disk. > > So I again used dd again to manually overwrite what is on that section of > the raw HDD and put my created metadata file onto the disk. However unless > vgcfgrestore has actually failedand you have no options, this is a very > risky stunt and should be avoided. I only did this stunt after doing a dd of > the entire disk onto another just in case I made a mistake. > > Now when I did vgscan I was able to see > Found volume group "VolGroup00" using metadata type lvm2 > I could also do pvscan and see both hard disks in the volume group. > I then run vgchange VolGroup00 -a y > then lvscan which showed me: > ACTIVE '/dev/VolGroup00/LogVol00' [476.38 GB] inherit > ACTIVE '/dev/VolGroup00/LogVol01' [512.00 MB] inherit > > Voila, now that I could see my volumes, I could mount them and access my > data. However mounting them proved to be a problem initially as the data was > so corrupt that when i mounted them and attempted to list the root directory > using ls, it could not even tell whether the /usr /opt /var were directories > or files and put a ? in that field. That meant that I needed to run fsck on > the filesystem to see if I could get this fixed. > > Running fsck yielded errors and kept bombing out while trying to fix the > first couple of inodes alluding to a corrupted superblock. So I then run > mke2fs -n /dev/VolGroup00/LogVol00 which showed me the alternate superblocks > for the filesystem. I was then able to run fsck -b superblock -y > /dev/VolGroup00/LogVol00 > Remember to run fsck with -y because initially I did not and had to keep > answering y to the fsck prompts however I got tired, cancelled the fsck and > redid it with -y which proved a good decision because the fsck run for over > 36 hours with those prompts flying by so fast ont he screen that I expect > they must have been about 100,000 of those prompts at the least. > > a couple of things to note: > 1. if using LVM, be sure to have a backup of your /etc directory or at > least your /etc/lvm directory. > 2. if you do not have a backup and end up getting a config file from raw > disk, pay attention to the } that close statements in the lvm metadata file. > 3. The usual. Take backups. I wouldn't have needed to go through all of > this if i had backups of my data. I would have simply reinstalled and then > restored from backup. > > _______________________________________________ > LUG mailing list > [email protected] > http://kym.net/mailman/listinfo/lug > %LUG is generously hosted by INFOCOM http://www.infocom.co.ug/ > > The above comments and data are owned by whoever posted them (including > attachments if any). The List's Host is not responsible for them in any way. > --------------------------------------- > >
_______________________________________________ LUG mailing list [email protected] http://kym.net/mailman/listinfo/lug %LUG is generously hosted by INFOCOM http://www.infocom.co.ug/ The above comments and data are owned by whoever posted them (including attachments if any). The List's Host is not responsible for them in any way. ---------------------------------------
