Doug: We put the system into admin mode (I think) and then make the copies. (I'll have to verify that!) And, yes, if you have to recover, you will need to run pvfs2-fsck with the replaced databases. You don't need to be in admin mode, because the servers won't come up anyway at this point.
If I am recalling pvfs2-fsck correctly, it tries to put all of the metadata back in sync; it can't look at the underlying system files in the bstream directory and reverse-engineer the metadata. Thus, you would have to have at least some non-corrupted databases still alive since the last backup in order to recover corrupted databases. If something catastrophic happened to your system and all of the databases were corrupted, then all you can do is restore from your backups and move ahead from there! Becky -- Becky Ligon PVFS Developer Clemson University 864-656-3865 > Becky, > > We will definitely be backing up the db files more frequently. Do you > do anything special for your backups, or just copy the files? Would > it be safer to have a file system snapshot, then copy the files for > backup? If we were to attempt a recovery by falling back to slightly > older copies of the bdb files, are there steps beyond setting mode to > 'admin' and running pvfs2-fsck recommended? Sorry for all the > questions, the docs are a little circumspect about procedures this > sort of thing. > > Per your other messages to the list, when we upgrade the pvfs-2.8.2 we > will building against the latest Berkeley DB from Oracle. > > Doug > > > At Tue, 23 Mar 2010 11:39:31 -0400 (EDT), > Becky Ligon wrote: >> >> Doug: >> >> We are also running in a production mode. So, we have jobs that backup >> the berkeley databases every so often. This is not ideal by any means; >> the filesystem should REALLY be in a quiesced state before the backups >> are >> taken. Until we have better redundancy in the system, this will have to >> do. >> >> Becky >> -- >> Becky Ligon >> PVFS Developer >> Clemson University >> 864-656-3865 >> >> > Doug Johnson wrote: >> >> At Mon, 22 Mar 2010 16:22:21 -0500, >> >> Phil Carns wrote: >> >>> [1 <text/plain; ISO-8859-1 (7bit)>] >> >>> Yeah, that's a problem. Each of those databases uses a different >> sort >> >>> order, so you'll need to try again with a slightly different >> >>> modification to BDB. I updated the documentation in trunk and >> attached >> >>> the new version to this email. I haven't had an opportunity to test >> it >> >>> yet, but it's a pretty simple change. >> >>> >> >> >> >> I've tried the db_load with PINT_trove_dbpf_ds_attr_compare, >> >> unfortunately the same error is encountered on startup. I also >> >> attempted using '-R' with db_dump, but same result. >> >> >> >> We've contacted the larger users to give them a heads up, but I will >> >> be creating a new file system today. I plan to keep the old >> >> directories for further recovery attempts, we have a multi-day outage >> >> scheduled next month and we could try some additional recovery steps >> >> (plus we're upgrading to 2.8.2 at that time.) Thanks for your help >> >> with this problem. >> > >> > Ok. Thanks for the status update, and sorry to hear that the recovery >> > didn't work out better. >> > >> > -Phil >> > >> > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
