Re: Verifying backups

[Ronald F. Guilmette]

In message <560cca51.5cwvptqce1nflu+u%per...@pluto.rain.com>, 
per...@pluto.rain.com (Perry Hutchison) wrote:

>Just because rsync is an awesome hammer, it does not necessarily follow
>that every problem involving backups closely resembles a nail :)

An excellent and very apropos point.

>Since your source and backup are both local, I suspect using rsync as
>a comparison tool is overkill.  (It may even be overkill for making
>the backups in the first place.)

In theory, I might be able to use cpio -p rather than rsync to make my
backups, however for some reason that I can't remember anymore, when
I originally set up my system for making backups, I looked at that
possibility and concluded that rsync was preferable to cpio -p.  I
wish that I could remember why.  (It might have had to do with better
or more complete handling of ``weird stuff'' like extended file
attributes, ACLs, device nodes, or other such oddities)

>Had you considered "diff -q -r"?

No, actually.  thanks for the suggestion!

>(I'm using gnu diff; other variants might have different command options.)

The diff I have here on FreeBSD (9.1) seem to be GNU, so this will work.


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560ce706@sanitarium.net>, 
Kevin Korb  wrote:

>Yes, when it comes to local copies cp is significantly faster than
>rsync.  Without --link-dest there isn't much advantage to using rsync
>for backups.  The only thing you get beyond cp -au is --delete.

I just now remembered the (forehead slap) bloody obvious reason I decided
to use rsync to make and maintain my backup drive(s).

Yes, it theory I could have used something simpler... cp -R or else
maybe cpio -p... but those just copy everything blindly.  For my
backups, I only need/want to have the NEW and/or MODIFIED files
copied to the backup drive.  (And also, of course, I need to have
files that have been deleted on the main drive be deleted also on
the backup drive.)

Rsync does everything I want as far as making and maintaining backups.
I could also have used FreeBSD backup & restore programs, but for
reasons I can't really remember anymore, I concluded that rsync was
the better option.


Regards,
rfg


P.S.  I have no idea what the -u option for cp is supposed to do.
I guess that must be a Linux-ism.  The FreeBSD man page for cp doesn't
mention any such thing as a -u option.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c4a51.4040...@sanitarium.net>, 
Kevin Korb  wrote:

>First off, --fileflags --force-change are not in my man rsync so I
>don't know what those are.

These are probably (Free)BSD specific.  Here's what the man page says:

--fileflags preserve file-flags (aka chflags)
--force-change  affect user/system immutable files/dirs

>Second, you should look into using either ZFS subvolume snapshots or
>rsync --link-dest to maintain multiple backups.

Thank you, but I have no real interest in switching to ZFS just now.

>Now, for your actual question...
>Add --itemize-changes to your standard command line.  -v is almost
>entirely useless without it anyway.

Thanks!  I certainly did not know about that option!

>To figure out and fix what is corrupt you have 2 paths:
>
>1. Add --checksum for a single pass.  This will take forever as rsync
>checksums everything even things it shouldn't expect to match (even
>things that are only on one end!).  Anything that checksum finds that
>rsync wouldn't have otherwise found would have a 'c' but not a 't'.

I don't understand what you mean about 'c' and 't'.  Are you talking
about what will appear in the (itemized) change log?  I am guessing so.

>2.  Add --ignore-times for a single pass.

NOW I am REALLY confused!  I don't understand at all what the functional
difference is between these two options:

-c, --checksum  skip based on checksum, not mod-time & size
-I, --ignore-times  don't skip files that match size and time

Could you please explain?  Both of these options would seem to me to have
exactly the same/identical effect.

>...  Normally this doesn't take
>as long as --checksum.  However, since you are using an external USB
>device which means you are also using --whole-file...

No, I am *not* using the --whole-file option.  Indeed, up until this
moment, I didn't even know that option existed!  I thank you for
bringing it to my attention.  Now I plan to be using --whole-file
in future when making all of my backups.  (Because most of the files
I back up are binary media files, I think that the delta algorithm
is not really saving me that much in terms of run-time.)

Having said that however, I need also to say that your comment (just
above) is, for me, puzzling and downright cryptic.  Yes, I am doing
my backups to an external USB-connected device.  But what has that
fact got to do with the --whole-file option?  I see no obvious
connection at all.

>... this will probably be even slower than --checksum.

Why would using the --whole-file option during my attempts to verify
my backup files cause things to run even slower?  (This is not at
all obvious.)

>Either way, you need to get your system writing files correctly.

Oh yes, clearly.  I believe (for now) that simply having proper cooling
applied to the external drive in question may be sufficient to prevent
corruption of files put on the drive in futire.  (I did run one of the
LONG built-in drive self-test passes on this drive after I found that
some files on it had been corrupted, but results from that were 100% OK.
So I think the drive perhaps just got a bit flaky during a time when I
*did not* have an small utility fan right next to it and pointing at it.
I _do_ have that now.)

Thank you for all of your detailed responses.


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Verifying backups

[Ronald F. Guilmette]

For some time now I've been using rsync on FreeBSD to make my system
backups.  Recently, I accidentally rm'd some files from one directory
and I had to go and fetch copies off of my backup drive.  After I had
done so, I found that about 1/5 of them were corrupted.  (They were
all .jpg files, by the way.)

To be clear, I most certainly *do not* attribute the apparent corruption
to rsync.  My backup drive is itself slightly dubious... a retired
(so-called ``refurbished'') WD Black 1TB drive which the S.M.A.R.T.
info showed already had well more than 30,000 hours on it before I
ever laid hands on it.  Also, and separately, I've been trying to use
it... and others like it... in one of these open-on-the-top slot
loading external/powered SATA to USB3 adapters.  As I have learned
recently, 2.5" drives are generally no problem to use with such
adapters, but unless you have a utility fan pointing right at it,
3.5" drives that are placed into one of those adapters can pretty
quickly get REALLY hot... a fact which, taken alone, may actually be
the root cause of the file corruption on my backup drive.

So anyway, I have a 1TB backup drive now that is chock full of
files that I placed on it previously (using rsync)... the majority
of which, by volume, are binary media files (i.e. mp3 songs, JPEGs,
movies, etc.) and now I'd like to carefully check them all to see
which ones may have gotten corrupted.

So, my question:  How best to do this?

Looking at the rsync man page, I see a couple of options that may
perhaps be relevant, specifically:

-c, --checksum  skip based on checksum, not mod-time & size
-n, --dry-run   perform a trial run with no changes made

Can I use these to, in effect, check all of my backup files for integrity?

Please note that the options I have been using to make my backups are
as follows:

-v -t -axHAXS --delete --fileflags --force-change

I don't imagine that either the -n or -c options will interact badly with
any of those, correct?


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c7e98.3090...@sanitarium.net>, 
Kevin Korb  wrote:

>Remove the -n and look at the results.  You would be copying the one
>dir into the two dir instead of copying the contents of the one dir
>into the two dir.

AHHH!  OK.  Yes.  My bad.

I keep on forgetting how critical to the semantics those trailing
slashes are when it comes to rsync.

Thank you greatly.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

Kevin Korb ,

I thank you greatly for your attempts to educate me, however as we get
deeper into discussing more and more different rsync options, I feel
that I am actually just getting more confused and frustrated.  I've been
sitting here, trying all sorts of different combinations and permutations
of the various options we've discussed, and that you've told me about,
but so far none of the combinations is producing the (seemingly simple)
effect that I wanted.

So, at the risk of trying your patience, may I please request that we
begin again.  Let's start from scratch, and perhaps you can guide me to
the simplest possible solution to the problem I first came here with...
hopefully a solution which is so simple that even *I* can't screw it up.

So, let's consider the following simple hypothetical example...

We have two directories within the current directory.  They are named
./one/ and ./two/

Within each of these two directories, there is a set of ordinary files.
The set of filenames present within each of the two directories is
identical.  Let's just say that each one contains the following
three ordinary files:

001.jpg
002.jpg
003.jpg

How may I use rsync to obtain a list of _just_ those files within these
two directories that the cmp command would describe as being different
between the two directories?  What is the exact command line I should use?
(And within the output from the given command, what must I look for,
exactly, that designates files that are different, as opposed to those
whose content is the same?)


Regards,
rfg


P.S.  I really do hope that I can get this to work with rsync.  I do
prefer not reinventing the wheel, but it is starting to seem simpler
to me if I were to just write a Perl script that would walk two directory
hierarchies, in parallel, and just repeatedly invoke the cmp command on
all of the regular files found therein.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c660f.5000...@sanitarium.net>, 
Kevin Korb  wrote:

>Just add --itemize-changes and --checksum to what you were doing
>before and know that it will take a long time.

I'm still not getting to where I need to be.  Maybe you can explain
what has gone wrong in this very simple example:

% mkdir one two
% echo hello > one/hello
% ln one/hello two/hello
% echo different0 > one/foo
% echo different1 > two/foo
% rsync -n -v --itemize-changes -checksum -a one two

Here is the output generated by that last command:

sending incremental file list
cd+ one/
>f+ one/foo
>f+ one/hello


I fail to see how this helps me to know that in this case the files
one/hello and two/hello are byte-for-byte identical AND also that
the contents of the files one/foo and two/foo are in fact different.

Where is the clear sign of a difference between the two "foo" files?


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c79ff.5010...@sanitarium.net>, 
Kevin Korb  wrote:

>Because you are making two/one.  Change to:
>rsync -n -v --itemize-changes -checksum -a one/ two/

OK, I tried it with your suggested command line, and yes, that produces
rather more substantially useful results.  However...

Perhaps I am just a bit thick, but I really don't have any idea what
you mean what you say that I am "making two/one" when I do it the
other way.  Can you clarify that comment and enlighten me (please)?


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html