Re: Minor disaster recovery
On Wed, Aug 12, 2009 at 8:01 PM, Ben Scott dragonh...@gmail.com wrote: On Wed, Aug 12, 2009 at 8:16 AM, Tyson Sawyerty...@j3.org wrote: The filesystem on the system drive is (or should be) backed up. Don't try. As Jeff Smith said, the more you tinker, the worse things usually get. If you intend to attempt to recover data from the failed disk, I would suggest making a block level image, like Jeff said. But I'd recommend using the dd_rescue and dd_rhelp utilities to do so. dd_rhelp will supervise dd_rescue, and use it to recovery easily readable blocks first, and then try harder for the remaining blocks. Once you've got the block-level image, you can examine it at your leisure, without worrying about if the drive is about to die for good. First I've heard about dd_rescue and dd_rhelp - is that showing my age? jeff ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Minor disaster recovery
Its been at least 10 years since I have actually done a recovery of this sort. ...back then LILO was king and floppy drives were still in use. I've been lucky enough to not do much sysadmin work in recent years. So... I have a small home server running a not quite up to date version of Ubuntu 8.10. It has an 80G main drive that is the OS and applications. It has an external USB drive that is our primary data storage. It has a 2nd USB drive that is used to backup the rest of the system, a couple of laptops and an N810. We use BackupPC for the backups and occasionally check that it is working and have occasionally recovered a single file. The primary drive has gone on the blink. About 4-5 days ago the primary system drive reported some sort of complaint and the OS remounted it read-only. We rebooted, said, Damn, well we'll have to replace that and went about life. This morning we found the system mostly unresponsive. The caps-lock LED was about the only response we could get out of it. The filesystem on the system drive is (or should be) backed up. We would like to recover the system rather than rebuild to avoid having to figure out all the applications we had installed and figure out how we had them configured. There is a reasonable chance that we can reboot the old drive again, but I have not yet tried. We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? Any suggestions on recovery strategy? One strategy I had in mind is (if the old drive still runs) is to boot the old drive with the new as a secondary. Shut down all extra services. Partition and format the new drive. Copy the filesystem from the old to the new. Install Grub on the new (dont' know how, never done much with Grub) and boot it. Do some sort of restore from BackupPC to restore any libraries for files that have been corrupted. Other suggestions? ...or fill in the details (specifically the install of Grub and moving to the primary position) of what I have outlined? Oh, the drive is a standard sized IDE. Thanks! Ty -- Tyson D Sawyer A strong conviction that something must be done is the parent of many bad measures. - Daniel Webster ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
Tyson Sawyer wrote: Its been at least 10 years since I have actually done a recovery of this sort. ...back then LILO was king and floppy drives were still in use. I've been lucky enough to not do much sysadmin work in recent years. So... I have a small home server running a not quite up to date version of Ubuntu 8.10. It has an 80G main drive that is the OS and applications. It has an external USB drive that is our primary data storage. It has a 2nd USB drive that is used to backup the rest of the system, a couple of laptops and an N810. We use BackupPC for the backups and occasionally check that it is working and have occasionally recovered a single file. The primary drive has gone on the blink. About 4-5 days ago the primary system drive reported some sort of complaint and the OS remounted it read-only. We rebooted, said, Damn, well we'll have to replace that and went about life. This morning we found the system mostly unresponsive. The caps-lock LED was about the only response we could get out of it. The filesystem on the system drive is (or should be) backed up. We would like to recover the system rather than rebuild to avoid having to figure out all the applications we had installed and figure out how we had them configured. There is a reasonable chance that we can reboot the old drive again, but I have not yet tried. We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? Any suggestions on recovery strategy? One strategy I had in mind is (if the old drive still runs) is to boot the old drive with the new as a secondary. Shut down all extra services. Partition and format the new drive. Copy the filesystem from the old to the new. Install Grub on the new (dont' know how, never done much with Grub) and boot it. Do some sort of restore from BackupPC to restore any libraries for files that have been corrupted. Other suggestions? ...or fill in the details (specifically the install of Grub and moving to the primary position) of what I have outlined? Oh, the drive is a standard sized IDE. Thanks! Ty You might want to take this opportunity (;^) to triage your system. If you have installed lot's of apps you will undoubtedly have apps that you never use. This will automatically take them off your system. You should also be able to perform a clean install to a new hard drive in just a relatively few minutes. One way to contaminate a system is to try to pick and choose from the pieces of an old system. As for disk drives the big sellers are still Western Digital and Seagate. You might see Maxtor in a retail store but they are now owned by Seagate. Recently Seagate had serious problems with their disk drives. They responded by reducing the length of their warranty unless you pay extra for their corporate versions. I have had good luck with both Western Digital and Seagate in having warranties honored. A few months back one of my clients had a 2.5 inch Seagate drive fail (had a 3 year warranty) and all I did was generate an automated RMA on the Seagate web site and then print it out, package up the bad drive and ship it to Seagate. About a week later I got new replacement drive. The drive had been in use at the customer site for about two years. Other manufacturers whose products might be in a retail store are Toshiba and Hitachi. I haven't had an undue number of problems with their products but remember that all drives eventually fail. It's not If but a matter of When. One manufacturer that I have read horror stories about is Samsung. The typical problem is that it is difficult to get through to their support and then they tend to deny coverage. I have had DOA's with just about every drive you can buy. About six months ago I bought a 160 GB Western Digital drive at Best Buy. DOA. I exchanged it for another identical model. DOA. At that point I returned it for a refund and informed the Best Buy manager that they probably had a bad run on that model and they might consider pulling them (they did). In the last six months I have installed at least two Western Digital 500 GB IDE hard drives and they have so far worked without incident. YMMV. -Alex ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
On Wednesday 12 August 2009 08:16, Tyson Sawyer wrote: Its been at least 10 years since I have actually done a recovery of this sort... We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? I have had good service from Showtime Computers in Hudson NH, http://www.showtimepc.com/. They are tech-savvy and can give you advice based on failure reports from customers. Any suggestions on recovery strategy? One strategy I had in mind is (if the old drive still runs) is to boot the old drive with the new as a secondary. Shut down all extra services. Partition and format the new drive. Copy the filesystem from the old to the new. Install Grub on the new (dont' know how, never done much with Grub) and boot it. Do some sort of restore from BackupPC to restore any libraries for files that have been corrupted. Folks more expert than I will comment, but I suggest that you install a fresh copy of Ubuntu 8.10 (exact same version as before) on the new drive. All of the Ubuntu files will have creation dates earlier than anything you changed. Take a look to see what that date is. Then copy anything newer on your backup to the replacement disk. Other suggestions? ...or fill in the details (specifically the install of Grub and moving to the primary position) of what I have outlined? I would be leery of recovering the operating system from the failing disk. You would not know if a seldom-used file had a flaw in it. Jim Kuzdrall ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
Step one - duplicate the drive. Do a dd if=/dev/sda of=/dev/sdb (or suitable drive with a new disk). You could also do dd if=/dev/sda of=/var/disk-copy to create a file with the name. One key I've found in recovering bad disks is to minimize the read/writes. Do a copy and work with that. jeff On Wed, Aug 12, 2009 at 5:29 PM, Jim Kuzdrall gnh...@intrel.com wrote: On Wednesday 12 August 2009 08:16, Tyson Sawyer wrote: Its been at least 10 years since I have actually done a recovery of this sort... We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? I have had good service from Showtime Computers in Hudson NH, http://www.showtimepc.com/. They are tech-savvy and can give you advice based on failure reports from customers. Any suggestions on recovery strategy? One strategy I had in mind is (if the old drive still runs) is to boot the old drive with the new as a secondary. Shut down all extra services. Partition and format the new drive. Copy the filesystem from the old to the new. Install Grub on the new (dont' know how, never done much with Grub) and boot it. Do some sort of restore from BackupPC to restore any libraries for files that have been corrupted. Folks more expert than I will comment, but I suggest that you install a fresh copy of Ubuntu 8.10 (exact same version as before) on the new drive. All of the Ubuntu files will have creation dates earlier than anything you changed. Take a look to see what that date is. Then copy anything newer on your backup to the replacement disk. Other suggestions? ...or fill in the details (specifically the install of Grub and moving to the primary position) of what I have outlined? I would be leery of recovering the operating system from the failing disk. You would not know if a seldom-used file had a flaw in it. Jim Kuzdrall ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/ ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
At the risk of being told you so, add some infrastructure to your server. For example, here's what I do: 2 drives in software RAID 1 for the OS (20 GB driver were cheapest when I started). 2+ drives in software RAID 1 or 5 for DATA. Monitor the drives for failure (logwatch sends a daily email) When (not if) a drive fails, remove it, RMA it and replace it before the other drive fails. You can put a larger drive in if it costs less then drive you're replacing. You'll have downtime when you remove/install drives, but that's it. You shouldn't care about downtime on a home system. Of course, this won't help you now. ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
On Wed, Aug 12, 2009 at 8:16 AM, Tyson Sawyerty...@j3.org wrote: The filesystem on the system drive is (or should be) backed up. You need to explain that using a lot more words. :) (e.g., how you back it up, using what software, how often, what you backup, how you test the backups, when your last backup was made, what's changed since then, etc.) We would like to recover the system rather than rebuild to avoid having to figure out all the applications we had installed and figure out how we had them configured. As Jim Kuzdrall said, I would avoid trying to run a system copied from a drive that's known to be faulty. You might copy a corrupt file. If you have a known-good backup of the system, from before the disk started to fail, restoring *that* to a new disk and running is okay. (I don't think Ubuntu keeps checksums of installed files by default (anyone know?). So there's no way to verify the integrity of the installed system (like rpm --verify --all). And even with package manager checksums, that won't help you with corrupt data files, config files, or files installed outside of the package manager (make install).) There is a reasonable chance that we can reboot the old drive again, but I have not yet tried. Don't try. As Jeff Smith said, the more you tinker, the worse things usually get. If you intend to attempt to recover data from the failed disk, I would suggest making a block level image, like Jeff said. But I'd recommend using the dd_rescue and dd_rhelp utilities to do so. dd_rhelp will supervise dd_rescue, and use it to recovery easily readable blocks first, and then try harder for the remaining blocks. Once you've got the block-level image, you can examine it at your leisure, without worrying about if the drive is about to die for good. We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? All hard disk drive brands are about equal. All big stores are about equal. Staples, OfficeMax, Wal-Mart, etc. Or find a local guy. For buying commodity parts, parts is parts, but some find it nice to give business to the little guy, and having a good relationship with a local tech dealer is a useful thing for a home user. ... boot the old drive with the new as a secondary. ... Copy the filesystem from the old to the new. I wouldn't recommend that no matter what. If I was going to copy a system from one disk to another, I'd put both drives in the system, boot from CD, and copy that way. Copying a running system is best avoided. But I wouldn't try to copy the install, for reasons given above. If I had a *complete* and *known-good* backup of the old system, I'd boot from CD, then restore that backup to the new disk. I might then try and recovery anything that's changed since the last backup. If I wasn't sure about my backup, or it wasn't complete: I would install a clean copy of my OS on to the new drive, from a known-good CD. Get up and running with a working system. Then I'd copy and/or restore stuff from backups and/or the old drive. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
Ben Scott wrote: On Wed, Aug 12, 2009 at 8:16 AM, Tyson Sawyerty...@j3.org wrote: The filesystem on the system drive is (or should be) backed up. You need to explain that using a lot more words. :) (e.g., how you back it up, using what software, how often, what you backup, how you test the backups, when your last backup was made, what's changed since then, etc.) We would like to recover the system rather than rebuild to avoid having to figure out all the applications we had installed and figure out how we had them configured. As Jim Kuzdrall said, I would avoid trying to run a system copied from a drive that's known to be faulty. You might copy a corrupt file. If you have a known-good backup of the system, from before the disk started to fail, restoring *that* to a new disk and running is okay. (I don't think Ubuntu keeps checksums of installed files by default (anyone know?). So there's no way to verify the integrity of the installed system (like rpm --verify --all). And even with package manager checksums, that won't help you with corrupt data files, config files, or files installed outside of the package manager (make install).) There is a reasonable chance that we can reboot the old drive again, but I have not yet tried. Don't try. As Jeff Smith said, the more you tinker, the worse things usually get. If you intend to attempt to recover data from the failed disk, I would suggest making a block level image, like Jeff said. But I'd recommend using the dd_rescue and dd_rhelp utilities to do so. dd_rhelp will supervise dd_rescue, and use it to recovery easily readable blocks first, and then try harder for the remaining blocks. Once you've got the block-level image, you can examine it at your leisure, without worrying about if the drive is about to die for good. We will attempt to find a replacement drive today. We live in Brookline, NH and work in Bedford, MA. Any suggestions on stores or drive brands? All hard disk drive brands are about equal. Pretty much. What is different is how easy it is to get support/replacement. It's easy with Western Digital and Seagate. Other vendors manufacture a lot more than disk drives. Their web pages can be nearly impossible to navigate and their support might be downright useless. I would suggest that you investigate a manufacturer's support by visiting their web site(s). If you can't even find the support contact information you're going to be in tough shape if you need help. My favorite, from a recent experience, is AMD. I had an out of the box defective processor and there was literally no way to contact their support. Fortunately I had bought the part from Newegg and they did their usual excellent job of handling my problem (RMA and bought a different processor model). All big stores are about equal. Staples, OfficeMax, Wal-Mart, etc. Or find a local guy. For buying commodity parts, parts is parts, but some find it nice to give business to the little guy, and having a good relationship with a local tech dealer is a useful thing for a home user. The local guy is almost always buying his parts from the same place you might buy the parts but his/her advantage would be getting a better price and more expeditious service. -Alex P.S. The markup on parts is usually very low unless it's somehow a proprietary part in which case you should be prepared to open your wallet wide. ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
On Wed, Aug 12, 2009 at 12:31 PM, Alex Hewitthewitt_t...@comcast.net wrote: Pretty much. What is different is how easy it is to get support/replacement. It's easy with Western Digital and Seagate. Other vendors manufacture a lot more than disk drives. Good point. Products may be commodities, but customer service can vary a lot. I can't say I've ever had much trouble filing an hard disk OEM warranty claim, but I haven't had to file many. (It happens that my experience has mostly been with out-of-warranty disks, or disks covered by a reseller warranty, e.g., Dell.) The markup on parts is usually very low unless it's somehow a proprietary part ... I think that varies. I've certainly encountered plenty of local shops which mark their parts up considerably. Their stance is basically that they're providing service and support, while Staples or whoever is just selling a box. I don't entirely disagree with that stance, either, but one should be aware of it either way. -- Ben ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
Re: Minor disaster recovery
Guys, Thanks for the help. I wasn't able to go shopping today so rebuild is going to take a little longer than I had hoped. Too bad I have a day job. Thanks for the suggestions for dd'ing the existing drive. If I go with a fresh Ubuntu install, I shouldn't need it as I expect that I have a good backup. As I said in the original post, we have BackupPC running to an external drive. It runs daily. I've been getting around to updating my data drives to a RAID configuration, but now I'm putting a priority on the main system drive also after considering your comments. ...not a novel idea, but the comments bring focus onto good points. I realize that recovery from the failing disk could result in some corrupt files, that is why that method included a restore from the backup once the system was running. The motivation for starting from the image of the failing drive is that it is exactly the same OS version as the backup I'd be restoring from. I have no idea how to get exactly the same version of Ubuntu from a fresh install. However, the system drive contains very little custom data. I suspect I'll be best off just doing a fresh, clean install and then digging config files out of the backup. Cheers! Ty -- Tyson D Sawyer A strong conviction that something must be done is the parent of many bad measures. - Daniel Webster ___ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
