I'm not sure what you're doing with your --verify...

It *sounds* like you want a full CRC style check of the *current*
files after the backup is complete. (i.e. File X gets updated with a
delta, and you want to verify that file X is the same both on the
source and destination locations/drives.)

Yes, although it's more of an internal consistency check within the
rdiff-backup repository itself. I'm looking for a way to quickly
verify the integrity my entire rdiff-backup repository.

In my scenario the repository is synced to an external USB drive that
gets rotated each day (i.e. each day I put yesterday's drive in
storage and bring a different drive out of storage to use for the next
backup). I use rsync to transfer my rdiff-backup repository (which
gets updated daily) to the USB drive. Then I run rdiff-backup -- verify- at-time to verify that the files on the USB drive are not corrupt. But
lately this has been taking too long.

Does that make sense?

Yes, and the USB connection may explain the longish verify times,
since it's somewhat slow, compared to a SATA drive connected directly
to the controller...

USB probably does have something to do with how long it takes. But on the other hand yafic can do a full verify in 1/4 of the time on the same drive with the same data, etc. So maybe rdiff-backup could be made to be faster?

But I see that you want to verify the "local" RDiff repository to the
"of-line" one.

I'm not sure what you mean by this statement... I want to do an internal consistency check on my rdiff-backup repository after it's been rsync'd to the USB disk. I need to be sure that the data on the USB disk is valid. I am doing the verify on the USB drive because that is the last place that the data will be copied before it goes into secure storage (for up to a month, but normally just a few days). Maybe an outline of my data flow will help you to understand what I'm trying to accomplish.

First the hardware:
- Xserve with raid array - this is being backed up with rdiff-backup
- Firewire 800 drive attached to Xserve - staging location for rdiff- backup repository, gets a new revision each night
- Mac Mini - remote backup "server"
- USB 2.0 drive attached to Mac Mini - gets a copy of the rdiff-backup repo from the Firewire 800 drive on the Xserve

Now the data flow:
- Xserve runs rdiff-backup from raid array to local firewire drive
- Xserve runs rdiff-backup --verify-at-time 0B on local firewire drive to verify integrity of most recent revision (this step may not be necessary) - Mac Mini runs rsync to copy rdiff-backup repo from Xserve firewire 800 drive to local USB drive - Mac Mini would now like to verify the integrity of the rdiff-backup repository that it just rsync'd to the USB drive

During this last step I would rather not tie up any resources on the Xserve. Instead, I want to do a fully local (to Mac Mini) verification of the rdiff-backup repository. This verification should let me know if any link in the (hardware) chain is failing: is the firewire 800 staging drive failing? is the USB drive failing?

Not sure how to do that - I'd guess you could do it with some other
tools - not storing the hashes - just a full compare each time. (How
big is the repository? [I think you said, but I don't recall.]

100 GB mirror + 80 GB of rdiff data. So almost 200 GB

---
But I'd guess your "local" repository isn't on the same disks as the
data, right?

Right.

If so, then it's probably not a huge deal if it takes 20 hours to
check the local repository against the remote. [Though I guess all
that disk channel activity might impact other disk through-put too...]

The drive will be moved to a secure location, so it needs to happen as quickly as possible. If we have a disaster (fire, etc.) a backup doesn't do us much good if the most recent snapshot is still online being verified (and hence consumed by the fire).

(Add a controller? Dunno...)

I use a similar system and I don't verify the local repository to the
remote, though perhaps I should. (I trust rsync to make sure they're
the same...since it's not just copying the files - it's doing hash
matches like RDiff...)

Even if rsync verifies that they're the same this is only a false sense of security since the staging repo (the source that rsync copied from) could be corrupt and you'll never know it. This corruption could be sneaking into old revisions which you don't bother to verify because it takes too long. There needs to be some way to verify that ALL of the data is fully intact after it's been copied... --verify-at- time almost gets there, but not quite. It could get you there if you have lots of time to do a verify-at-time for each revision in the repo, but I'm guessing that would be prohibitively expensive in most cases.

BTW, is this on a windows platform? (Curious...) Ah, probably not
since yafic isn't... :)

Nope. All machines are running Mac OS. I have aspirations to add some Windows machines at some point, but that's not likely until I get a faster verify.

~ Daniel



_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Reply via email to