I really have to agree with Dave on this. I can think of alot of ways to recover from this, and unless you have money to burn, you really don't need a consultant to help you do it. But if you still feel differently, and haven't found anyone for the job, i'll gladly take up the flag. Contact me off list for that.
I will make an additional note here. If this is such a critical production server, then why were you making changes when you lacked recent backups? Or possibly more importantly, why were you making changes when you weren't 100% certain what you were doing? Whenever i'm making changes that I have the slightest suspicion may break the box, i always do a dry run on a test box first. glibc & gcc are prolly the worst offenders, as once you break either, you may be toast. --- Dave Anselmi <[EMAIL PROTECTED]> wrote: > Sorry for cross posting, but maybe this will be interesting on both. > > Does anyone know of any lists that discuss reliability and disaster > recovery? I have some ideas below, but they are from my experience > only. > > George Kasica wrote: > [...] > > > What I'm asking is: Is there ANYONE out there that know how to do > this > > and would be willing to do the job FOR PAY in the very near future? > > REBUILDING THE BOX FROM SCRATCH IS NOT AN OPTION. Though I do do a > > full tape backup nightly, the time between the attempts and noticing > > they broke something is beyond the length of tapes I have here... > > It would be cheaper for you, I think, to fix it yourself. If you > can't > have the box down, get another one and build a system on it (you can > start > with backup tapes if you think that's easier). Copy over all your > production stuff and test it out. When it's ready, stick it into the > network and pull the old one. The switch should be done during a low > traffic period, and if you're careful about how you do it (and how > your > network is arranged), your customers won't notice. > > If you don't have time to do this yourself, then I guess you're into > paying > someone. But I would not try to fix the problem in place. gcc is one > thing but when you mess with glibc you risk breaking what's running > and > your customers will notice. I don't think I'd trust even an expert to > do > this, since he's unlikely to know all the details of your system. > > Here are some suggestions to avoid this problem in the future: > > You should have a test box in addition to your production box. They > should > be identical hardware and software (you probably don't need gcc or > other > development tools on the production box, though). When you need to > upgrade > a package, build it on the test box in such a way that you get a tar > file - > there's usually some sort of distribution make target for this sort of > thing. > > Install the package by untarring, both on the test box and the > production > box. To be even safer, get yourself a development box and do the > building > there. That way the process you use on the test box and the > production box > will be identical, and you'll know what to expect. > > Test out your new package *thouroughly* before you put it on the > production > box. You have to make sure that config and data files get updated at > the > same time. Ideally, you'll be able to back out changes in your test > system > and start over so that if there's a problem you can fix it and redo > the > whole process. Your upgrade procedure needs to be straightforward so > you > don't mess it up on the production system. > > Bigger packages, like glibc, will need more testing than something > like > apache. But you should only upgrade the base system when it's > necessary to > support a new package you need. This system should be stable, not > bleeding > edge. ===== ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lonni J. Friedman [EMAIL PROTECTED] Linux Step-by-step help: http://netllama.ipfox.com . __________________________________________________ Do You Yahoo!? Send your FREE holiday greetings online! http://greetings.yahoo.com _______________________________________________ Linux-users mailing list Archives, Digests, etc at http://linux.nf/mailman/listinfo/linux-users
