Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

FWIW, the one time I had corruption in my backups the problem was a
bad DIMM randomly flipping bits.  I now insist on ECC RAM.

On 03/07/2016 03:51 PM, Henri Shustak wrote:
> Just chiming in slightly off topic.
> 
> As a first step if you are going to be backing up files to some
> media with a computer it would be a really good idea to ensure,
> that the hardware being used is not faulty. I am not saying that
> your hardware is faulty. However, it would be worth checking this
> somehow. Check the drive media for bad blocks, check that all the
> cables are working well. Ensure the mother board of the system is
> in good working order etc.
> 
> As a second step if you are going to be performing backups (with a
> file system based tool such as rsync) to any kind of file system in
> future, I would strongly suggest checking the file system is in a
> good state on a regular basis. File system corruption is capable of
> cause all sorts of problems for backup systems which rely upon the
> file system like rsync.
> 
> Hope this helps.
> 
> 
>
> 
This email is protected by LBackup, an open source backup solution
> http://www.lbackup.org
> 
> On 2/10/2015, at 9:48 AM, Ronald F. Guilmette
>  wrote:
> 
>> 
>> In message <560ce706@sanitarium.net>, Kevin Korb
>>  wrote:
>> 
>>> Yes, when it comes to local copies cp is significantly faster
>>> than rsync.  Without --link-dest there isn't much advantage to
>>> using rsync for backups.  The only thing you get beyond cp -au
>>> is --delete.
>> 
>> I just now remembered the (forehead slap) bloody obvious reason I
>> decided to use rsync to make and maintain my backup drive(s).
>> 
>> Yes, it theory I could have used something simpler... cp -R or
>> else maybe cpio -p... but those just copy everything blindly.
>> For my backups, I only need/want to have the NEW and/or MODIFIED
>> files copied to the backup drive.  (And also, of course, I need
>> to have files that have been deleted on the main drive be deleted
>> also on the backup drive.)
>> 
>> Rsync does everything I want as far as making and maintaining
>> backups. I could also have used FreeBSD backup & restore
>> programs, but for reasons I can't really remember anymore, I
>> concluded that rsync was the better option.
>> 
>> 
>> Regards, rfg
>> 
>> 
>> P.S.  I have no idea what the -u option for cp is supposed to
>> do. I guess that must be a Linux-ism.  The FreeBSD man page for
>> cp doesn't mention any such thing as a -u option.
>> 
>> -- Please use reply-all for most replies to avoid omitting the
>> mailing list. To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlbeHlUACgkQVKC1jlbQAQdMAgCgzF9fT2juKqF+A0kkVr+Ofby/
s5gAnRS1LmJKiHqWVWCpArBBp8jymSuo
=VoO/
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Henri Shustak]

Just chiming in slightly off topic.

As a first step if you are going to be backing up files to some media with a 
computer it would be a really good idea to ensure, that the hardware being used 
is not faulty. I am not saying that your hardware is faulty. However, it would 
be worth checking this somehow. Check the drive media for bad blocks, check 
that all the cables are working well. Ensure the mother board of the system is 
in good working order etc.

As a second step if you are going to be performing backups (with a file system 
based tool such as rsync) to any kind of file system in future, I would 
strongly suggest checking the file system is in a good state on a regular 
basis. File system corruption is capable of cause all sorts of problems for 
backup systems which rely upon the file system like rsync.

Hope this helps.


This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 2/10/2015, at 9:48 AM, Ronald F. Guilmette  wrote:

> 
> In message <560ce706@sanitarium.net>, 
> Kevin Korb  wrote:
> 
>> Yes, when it comes to local copies cp is significantly faster than
>> rsync.  Without --link-dest there isn't much advantage to using rsync
>> for backups.  The only thing you get beyond cp -au is --delete.
> 
> I just now remembered the (forehead slap) bloody obvious reason I decided
> to use rsync to make and maintain my backup drive(s).
> 
> Yes, it theory I could have used something simpler... cp -R or else
> maybe cpio -p... but those just copy everything blindly.  For my
> backups, I only need/want to have the NEW and/or MODIFIED files
> copied to the backup drive.  (And also, of course, I need to have
> files that have been deleted on the main drive be deleted also on
> the backup drive.)
> 
> Rsync does everything I want as far as making and maintaining backups.
> I could also have used FreeBSD backup & restore programs, but for
> reasons I can't really remember anymore, I concluded that rsync was
> the better option.
> 
> 
> Regards,
> rfg
> 
> 
> P.S.  I have no idea what the -u option for cp is supposed to do.
> I guess that must be a Linux-ism.  The FreeBSD man page for cp doesn't
> mention any such thing as a -u option.
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Yes, when it comes to local copies cp is significantly faster than
rsync.  Without --link-dest there isn't much advantage to using rsync
for backups.  The only thing you get beyond cp -au is --delete.

Also, when it comes to static data like media files I like to keep an
md5 file around with checksums for all the files.  That way I can
easily verify the files later on either the primary or backup storage
and more importantly if they differ I can tell which one is correct.

The cfv utility is great for file verifying.  The -n option on it
tells it to rename incorrect files.  Then you can just do a simple
rsync pass to delete the incorrectly named files and put back the now
missing files.

On 10/01/2015 01:53 AM, Perry Hutchison wrote:
> "Ronald F. Guilmette"  wrote:
> 
>> P.S.  I really do hope that I can get this to work with rsync. I
>> do prefer not reinventing the wheel, but it is starting to seem 
>> simpler to me if I were to just write a Perl script that would 
>> walk two directory hierarchies, in parallel, and just repeatedly 
>> invoke the cmp command on all of the regular files found
>> therein.
> 
> Just because rsync is an awesome hammer, it does not necessarily
> follow that every problem involving backups closely resembles a
> nail :)
> 
> Since your source and backup are both local, I suspect using rsync
> as a comparison tool is overkill.  (It may even be overkill for
> making the backups in the first place.)  Had you considered "diff
> -q -r"?
> 
> $ mkdir one two $ echo hello > one/hello $ ln one/hello two/hello $
> echo different0 > one/foo $ echo different1 > two/foo $ diff -q -r
> one two Files one/foo and two/foo differ
> 
> or, if you prefer
> 
> $ diff -q -r -s one two Files one/foo and two/foo differ Files
> one/hello and two/hello are identical
> 
> (I'm using gnu diff; other variants might have different command
> options.)
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYM5wYACgkQVKC1jlbQAQdfRgCgkcyjw09g677+T+BUsdDWFb3c
DwAAoL2q8+P/c8oolfNq9WEbhlIL2SvW
=8K4f
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560cca51.5cwvptqce1nflu+u%per...@pluto.rain.com>, 
per...@pluto.rain.com (Perry Hutchison) wrote:

>Just because rsync is an awesome hammer, it does not necessarily follow
>that every problem involving backups closely resembles a nail :)

An excellent and very apropos point.

>Since your source and backup are both local, I suspect using rsync as
>a comparison tool is overkill.  (It may even be overkill for making
>the backups in the first place.)

In theory, I might be able to use cpio -p rather than rsync to make my
backups, however for some reason that I can't remember anymore, when
I originally set up my system for making backups, I looked at that
possibility and concluded that rsync was preferable to cpio -p.  I
wish that I could remember why.  (It might have had to do with better
or more complete handling of ``weird stuff'' like extended file
attributes, ACLs, device nodes, or other such oddities)

>Had you considered "diff -q -r"?

No, actually.  thanks for the suggestion!

>(I'm using gnu diff; other variants might have different command options.)

The diff I have here on FreeBSD (9.1) seem to be GNU, so this will work.


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

(GNU) cp -au is exactly equal to rsync -au.  It won't copy files that
are already up to date.  It just doesn't have an equivalence to
- --delete.  Therefore, when doing local copies it is often faster to do
a cp -au followed by an rsync --delete so that rsync is only bothering
with the deletions.

On 10/01/2015 04:48 PM, Ronald F. Guilmette wrote:
> 
> In message <560ce706@sanitarium.net>, Kevin Korb
>  wrote:
> 
>> Yes, when it comes to local copies cp is significantly faster
>> than rsync.  Without --link-dest there isn't much advantage to
>> using rsync for backups.  The only thing you get beyond cp -au is
>> --delete.
> 
> I just now remembered the (forehead slap) bloody obvious reason I
> decided to use rsync to make and maintain my backup drive(s).
> 
> Yes, it theory I could have used something simpler... cp -R or
> else maybe cpio -p... but those just copy everything blindly.  For
> my backups, I only need/want to have the NEW and/or MODIFIED files 
> copied to the backup drive.  (And also, of course, I need to have 
> files that have been deleted on the main drive be deleted also on 
> the backup drive.)
> 
> Rsync does everything I want as far as making and maintaining
> backups. I could also have used FreeBSD backup & restore programs,
> but for reasons I can't really remember anymore, I concluded that
> rsync was the better option.
> 
> 
> Regards, rfg
> 
> 
> P.S.  I have no idea what the -u option for cp is supposed to do. I
> guess that must be a Linux-ism.  The FreeBSD man page for cp
> doesn't mention any such thing as a -u option.
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYNniEACgkQVKC1jlbQAQcMzwCfcb6FBPrnfE+EXWbUIoy3+GNN
OiwAoKDju0x7yEhMGVkyV9q2kxsGI+6D
=Ftfa
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560ce706@sanitarium.net>, 
Kevin Korb  wrote:

>Yes, when it comes to local copies cp is significantly faster than
>rsync.  Without --link-dest there isn't much advantage to using rsync
>for backups.  The only thing you get beyond cp -au is --delete.

I just now remembered the (forehead slap) bloody obvious reason I decided
to use rsync to make and maintain my backup drive(s).

Yes, it theory I could have used something simpler... cp -R or else
maybe cpio -p... but those just copy everything blindly.  For my
backups, I only need/want to have the NEW and/or MODIFIED files
copied to the backup drive.  (And also, of course, I need to have
files that have been deleted on the main drive be deleted also on
the backup drive.)

Rsync does everything I want as far as making and maintaining backups.
I could also have used FreeBSD backup & restore programs, but for
reasons I can't really remember anymore, I concluded that rsync was
the better option.


Regards,
rfg


P.S.  I have no idea what the -u option for cp is supposed to do.
I guess that must be a Linux-ism.  The FreeBSD man page for cp doesn't
mention any such thing as a -u option.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Perry Hutchison]

"Ronald F. Guilmette"  wrote:

> P.S.  I really do hope that I can get this to work with rsync.
> I do prefer not reinventing the wheel, but it is starting to seem
> simpler to me if I were to just write a Perl script that would
> walk two directory hierarchies, in parallel, and just repeatedly
> invoke the cmp command on all of the regular files found therein.

Just because rsync is an awesome hammer, it does not necessarily follow
that every problem involving backups closely resembles a nail :)

Since your source and backup are both local, I suspect using rsync as
a comparison tool is overkill.  (It may even be overkill for making
the backups in the first place.)  Had you considered "diff -q -r"?

$ mkdir one two
$ echo hello > one/hello
$ ln one/hello two/hello
$ echo different0 > one/foo
$ echo different1 > two/foo
$ diff -q -r one two
Files one/foo and two/foo differ

or, if you prefer

$ diff -q -r -s one two
Files one/foo and two/foo differ
Files one/hello and two/hello are identical

(I'm using gnu diff; other variants might have different command options.)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c4a51.4040...@sanitarium.net>, 
Kevin Korb  wrote:

>First off, --fileflags --force-change are not in my man rsync so I
>don't know what those are.

These are probably (Free)BSD specific.  Here's what the man page says:

--fileflags preserve file-flags (aka chflags)
--force-change  affect user/system immutable files/dirs

>Second, you should look into using either ZFS subvolume snapshots or
>rsync --link-dest to maintain multiple backups.

Thank you, but I have no real interest in switching to ZFS just now.

>Now, for your actual question...
>Add --itemize-changes to your standard command line.  -v is almost
>entirely useless without it anyway.

Thanks!  I certainly did not know about that option!

>To figure out and fix what is corrupt you have 2 paths:
>
>1. Add --checksum for a single pass.  This will take forever as rsync
>checksums everything even things it shouldn't expect to match (even
>things that are only on one end!).  Anything that checksum finds that
>rsync wouldn't have otherwise found would have a 'c' but not a 't'.

I don't understand what you mean about 'c' and 't'.  Are you talking
about what will appear in the (itemized) change log?  I am guessing so.

>2.  Add --ignore-times for a single pass.

NOW I am REALLY confused!  I don't understand at all what the functional
difference is between these two options:

-c, --checksum  skip based on checksum, not mod-time & size
-I, --ignore-times  don't skip files that match size and time

Could you please explain?  Both of these options would seem to me to have
exactly the same/identical effect.

>...  Normally this doesn't take
>as long as --checksum.  However, since you are using an external USB
>device which means you are also using --whole-file...

No, I am *not* using the --whole-file option.  Indeed, up until this
moment, I didn't even know that option existed!  I thank you for
bringing it to my attention.  Now I plan to be using --whole-file
in future when making all of my backups.  (Because most of the files
I back up are binary media files, I think that the delta algorithm
is not really saving me that much in terms of run-time.)

Having said that however, I need also to say that your comment (just
above) is, for me, puzzling and downright cryptic.  Yes, I am doing
my backups to an external USB-connected device.  But what has that
fact got to do with the --whole-file option?  I see no obvious
connection at all.

>... this will probably be even slower than --checksum.

Why would using the --whole-file option during my attempts to verify
my backup files cause things to run even slower?  (This is not at
all obvious.)

>Either way, you need to get your system writing files correctly.

Oh yes, clearly.  I believe (for now) that simply having proper cooling
applied to the external drive in question may be sufficient to prevent
corruption of files put on the drive in futire.  (I did run one of the
LONG built-in drive self-test passes on this drive after I found that
some files on it had been corrupted, but results from that were 100% OK.
So I think the drive perhaps just got a bit flaky during a time when I
*did not* have an small utility fan right next to it and pointing at it.
I _do_ have that now.)

Thank you for all of your detailed responses.


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

reply inline...

On 09/30/2015 05:18 PM, Ronald F. Guilmette wrote:
> 
> In message <560c4a51.4040...@sanitarium.net>, Kevin Korb
>  wrote:
> 
>> First off, --fileflags --force-change are not in my man rsync so
>> I don't know what those are.
> 
> These are probably (Free)BSD specific.  Here's what the man page
> says:
> 
> --fileflags preserve file-flags (aka chflags) 
> --force-change  affect user/system immutable files/dirs

Interesting, these aren't on my FreeBSD system.

>> Second, you should look into using either ZFS subvolume snapshots
>> or rsync --link-dest to maintain multiple backups.
> 
> Thank you, but I have no real interest in switching to ZFS just
> now.
> 
>> Now, for your actual question... Add --itemize-changes to your
>> standard command line.  -v is almost entirely useless without it
>> anyway.
> 
> Thanks!  I certainly did not know about that option!
> 
>> To figure out and fix what is corrupt you have 2 paths:
>> 
>> 1. Add --checksum for a single pass.  This will take forever as
>> rsync checksums everything even things it shouldn't expect to
>> match (even things that are only on one end!).  Anything that
>> checksum finds that rsync wouldn't have otherwise found would
>> have a 'c' but not a 't'.
> 
> I don't understand what you mean about 'c' and 't'.  Are you
> talking about what will appear in the (itemized) change log?  I am
> guessing so.

You will when you run sync with --itemize-changes.  It prepends a
string to the file name list telling you what is different about a
file. A c without a t means that the file had a different checksum but
not a different timestamp which shouldn't happen.

> 
>> 2.  Add --ignore-times for a single pass.
> 
> NOW I am REALLY confused!  I don't understand at all what the
> functional difference is between these two options:
> 
> -c, --checksum  skip based on checksum, not mod-time &
> size -I, --ignore-times  don't skip files that match size
> and time

- --checksum == copy only things that have a different checksum and
ignore everything else

- --ignore-times == assume everything is wrong and re-delta-copy it all.
 If you aren't using --whole-file then only the different parts of the
file is copied.  If you are using --whole-file then it just means
re-copy everything.

> 
> Could you please explain?  Both of these options would seem to me
> to have exactly the same/identical effect.
> 
>> ...  Normally this doesn't take as long as --checksum.  However,
>> since you are using an external USB device which means you are
>> also using --whole-file...
> 
> No, I am *not* using the --whole-file option.  Indeed, up until
> this moment, I didn't even know that option existed!  I thank you
> for bringing it to my attention.  Now I plan to be using
> --whole-file in future when making all of my backups.  (Because
> most of the files I back up are binary media files, I think that
> the delta algorithm is not really saving me that much in terms of
> run-time.)
> 
> Having said that however, I need also to say that your comment
> (just above) is, for me, puzzling and downright cryptic.  Yes, I am
> doing my backups to an external USB-connected device.  But what has
> that fact got to do with the --whole-file option?  I see no
> obvious connection at all.

If your source and target are both local then --whole-file is forced.
 This is a performance feature as delta-transferring a file locally is
a ton of extra work for no benefit at all.

With --whole-file the local copy process is:
read file from source
write file to target

Without --whole-file the process would be:
read file from source and hash the pieces of it
read file from target and hash the pieces of it
compare the hashes
if difference
  if --inplace
write file to target beginning at difference
  else
write entire file to target



> 
>> ... this will probably be even slower than --checksum.
> 
> Why would using the --whole-file option during my attempts to
> verify my backup files cause things to run even slower?  (This is
> not at all obvious.)
> 
>> Either way, you need to get your system writing files correctly.
> 
> Oh yes, clearly.  I believe (for now) that simply having proper
> cooling applied to the external drive in question may be sufficient
> to prevent corruption of files put on the drive in futire.  (I did
> run one of the LONG built-in drive self-test passes on this drive
> after I found that some files on it had been corrupted, but results
> from that were 100% OK. So I think the drive perhaps just got a bit
> flaky during a time when I *did not* have an small utility fan
> right next to it and pointing at it. I _do_ have that now.)
> 
> Thank you for all of your detailed responses.
> 
> 
> Regards, rfg
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853

Verifying backups

[Ronald F. Guilmette]

For some time now I've been using rsync on FreeBSD to make my system
backups.  Recently, I accidentally rm'd some files from one directory
and I had to go and fetch copies off of my backup drive.  After I had
done so, I found that about 1/5 of them were corrupted.  (They were
all .jpg files, by the way.)

To be clear, I most certainly *do not* attribute the apparent corruption
to rsync.  My backup drive is itself slightly dubious... a retired
(so-called ``refurbished'') WD Black 1TB drive which the S.M.A.R.T.
info showed already had well more than 30,000 hours on it before I
ever laid hands on it.  Also, and separately, I've been trying to use
it... and others like it... in one of these open-on-the-top slot
loading external/powered SATA to USB3 adapters.  As I have learned
recently, 2.5" drives are generally no problem to use with such
adapters, but unless you have a utility fan pointing right at it,
3.5" drives that are placed into one of those adapters can pretty
quickly get REALLY hot... a fact which, taken alone, may actually be
the root cause of the file corruption on my backup drive.

So anyway, I have a 1TB backup drive now that is chock full of
files that I placed on it previously (using rsync)... the majority
of which, by volume, are binary media files (i.e. mp3 songs, JPEGs,
movies, etc.) and now I'd like to carefully check them all to see
which ones may have gotten corrupted.

So, my question:  How best to do this?

Looking at the rsync man page, I see a couple of options that may
perhaps be relevant, specifically:

-c, --checksum  skip based on checksum, not mod-time & size
-n, --dry-run   perform a trial run with no changes made

Can I use these to, in effect, check all of my backup files for integrity?

Please note that the options I have been using to make my backups are
as follows:

-v -t -axHAXS --delete --fileflags --force-change

I don't imagine that either the -n or -c options will interact badly with
any of those, correct?


Regards,
rfg

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

First off, --fileflags --force-change are not in my man rsync so I
don't know what those are.

Second, you should look into using either ZFS subvolume snapshots or
rsync --link-dest to maintain multiple backups.

Now, for your actual question...
Add --itemize-changes to your standard command line.  -v is almost
entirely useless without it anyway.

To figure out and fix what is corrupt you have 2 paths:

1. Add --checksum for a single pass.  This will take forever as rsync
checksums everything even things it shouldn't expect to match (even
things that are only on one end!).  Anything that checksum finds that
rsync wouldn't have otherwise found would have a 'c' but not a 't'.

2.  Add --ignore-times for a single pass.  Normally this doesn't take
as long as --checksum.  However, since you are using an external USB
device which means you are also using --whole-file this will probably
be even slower than --checksum.

Either way, you need to get your system writing files correctly.

On 09/30/2015 04:19 PM, Ronald F. Guilmette wrote:
> 
> For some time now I've been using rsync on FreeBSD to make my
> system backups.  Recently, I accidentally rm'd some files from one
> directory and I had to go and fetch copies off of my backup drive.
> After I had done so, I found that about 1/5 of them were corrupted.
> (They were all .jpg files, by the way.)
> 
> To be clear, I most certainly *do not* attribute the apparent
> corruption to rsync.  My backup drive is itself slightly dubious...
> a retired (so-called ``refurbished'') WD Black 1TB drive which the
> S.M.A.R.T. info showed already had well more than 30,000 hours on
> it before I ever laid hands on it.  Also, and separately, I've been
> trying to use it... and others like it... in one of these
> open-on-the-top slot loading external/powered SATA to USB3
> adapters.  As I have learned recently, 2.5" drives are generally no
> problem to use with such adapters, but unless you have a utility
> fan pointing right at it, 3.5" drives that are placed into one of
> those adapters can pretty quickly get REALLY hot... a fact which,
> taken alone, may actually be the root cause of the file corruption
> on my backup drive.
> 
> So anyway, I have a 1TB backup drive now that is chock full of 
> files that I placed on it previously (using rsync)... the majority 
> of which, by volume, are binary media files (i.e. mp3 songs,
> JPEGs, movies, etc.) and now I'd like to carefully check them all
> to see which ones may have gotten corrupted.
> 
> So, my question:  How best to do this?
> 
> Looking at the rsync man page, I see a couple of options that may 
> perhaps be relevant, specifically:
> 
> -c, --checksum  skip based on checksum, not mod-time &
> size -n, --dry-run   perform a trial run with no
> changes made
> 
> Can I use these to, in effect, check all of my backup files for
> integrity?
> 
> Please note that the options I have been using to make my backups
> are as follows:
> 
> -v -t -axHAXS --delete --fileflags --force-change
> 
> I don't imagine that either the -n or -c options will interact
> badly with any of those, correct?
> 
> 
> Regards, rfg
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYMSlEACgkQVKC1jlbQAQeNCQCff5zt9XGv94L0uNCtZn6Yf1L5
WqEAoOit1g8CazYDIwp9YDjMDPMhU4Tr
=ekEj
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c7e98.3090...@sanitarium.net>, 
Kevin Korb  wrote:

>Remove the -n and look at the results.  You would be copying the one
>dir into the two dir instead of copying the contents of the one dir
>into the two dir.

AHHH!  OK.  Yes.  My bad.

I keep on forgetting how critical to the semantics those trailing
slashes are when it comes to rsync.

Thank you greatly.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

Kevin Korb ,

I thank you greatly for your attempts to educate me, however as we get
deeper into discussing more and more different rsync options, I feel
that I am actually just getting more confused and frustrated.  I've been
sitting here, trying all sorts of different combinations and permutations
of the various options we've discussed, and that you've told me about,
but so far none of the combinations is producing the (seemingly simple)
effect that I wanted.

So, at the risk of trying your patience, may I please request that we
begin again.  Let's start from scratch, and perhaps you can guide me to
the simplest possible solution to the problem I first came here with...
hopefully a solution which is so simple that even *I* can't screw it up.

So, let's consider the following simple hypothetical example...

We have two directories within the current directory.  They are named
./one/ and ./two/

Within each of these two directories, there is a set of ordinary files.
The set of filenames present within each of the two directories is
identical.  Let's just say that each one contains the following
three ordinary files:

001.jpg
002.jpg
003.jpg

How may I use rsync to obtain a list of _just_ those files within these
two directories that the cmp command would describe as being different
between the two directories?  What is the exact command line I should use?
(And within the output from the given command, what must I look for,
exactly, that designates files that are different, as opposed to those
whose content is the same?)


Regards,
rfg


P.S.  I really do hope that I can get this to work with rsync.  I do
prefer not reinventing the wheel, but it is starting to seem simpler
to me if I were to just write a Perl script that would walk two directory
hierarchies, in parallel, and just repeatedly invoke the cmp command on
all of the regular files found therein.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c660f.5000...@sanitarium.net>, 
Kevin Korb  wrote:

>Just add --itemize-changes and --checksum to what you were doing
>before and know that it will take a long time.

I'm still not getting to where I need to be.  Maybe you can explain
what has gone wrong in this very simple example:

% mkdir one two
% echo hello > one/hello
% ln one/hello two/hello
% echo different0 > one/foo
% echo different1 > two/foo
% rsync -n -v --itemize-changes -checksum -a one two

Here is the output generated by that last command:

sending incremental file list
cd+ one/
>f+ one/foo
>f+ one/hello


I fail to see how this helps me to know that in this case the files
one/hello and two/hello are byte-for-byte identical AND also that
the contents of the files one/foo and two/foo are in fact different.

Where is the clear sign of a difference between the two "foo" files?


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Remove the -n and look at the results.  You would be copying the one
dir into the two dir instead of copying the contents of the one dir
into the two dir.

On 09/30/2015 08:28 PM, Ronald F. Guilmette wrote:
> 
> In message <560c79ff.5010...@sanitarium.net>, Kevin Korb
>  wrote:
> 
>> Because you are making two/one.  Change to: rsync -n -v
>> --itemize-changes -checksum -a one/ two/
> 
> OK, I tried it with your suggested command line, and yes, that
> produces rather more substantially useful results.  However...
> 
> Perhaps I am just a bit thick, but I really don't have any idea
> what you mean what you say that I am "making two/one" when I do it
> the other way.  Can you clarify that comment and enlighten me
> (please)?
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYMfpgACgkQVKC1jlbQAQci5wCg1rsRspQARC2ho/9/qwlT1KNY
ZmIAoMHhYMxens58/l991YGIjVLW/ttl
=wsDi
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Because you are making two/one.  Change to:
rsync -n -v --itemize-changes -checksum -a one/ two/

On 09/30/2015 07:22 PM, Ronald F. Guilmette wrote:
> rsync -n -v --itemize-changes -checksum -a one two

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYMef8ACgkQVKC1jlbQAQdi2wCgibL+jshwMtzxSwMMAJYmmftz
hJMAn2DLjXF3WKBE+3eZyLArDeyYRJII
=/vV+
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Ronald F. Guilmette]

In message <560c79ff.5010...@sanitarium.net>, 
Kevin Korb  wrote:

>Because you are making two/one.  Change to:
>rsync -n -v --itemize-changes -checksum -a one/ two/

OK, I tried it with your suggested command line, and yes, that produces
rather more substantially useful results.  However...

Perhaps I am just a bit thick, but I really don't have any idea what
you mean what you say that I am "making two/one" when I do it the
other way.  Can you clarify that comment and enlighten me (please)?


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Verifying backups

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Just add --itemize-changes and --checksum to what you were doing
before and know that it will take a long time.

On 09/30/2015 06:42 PM, Ronald F. Guilmette wrote:
> 
> Kevin Korb ,
> 
> I thank you greatly for your attempts to educate me, however as we
> get deeper into discussing more and more different rsync options, I
> feel that I am actually just getting more confused and frustrated.
> I've been sitting here, trying all sorts of different combinations
> and permutations of the various options we've discussed, and that
> you've told me about, but so far none of the combinations is
> producing the (seemingly simple) effect that I wanted.
> 
> So, at the risk of trying your patience, may I please request that
> we begin again.  Let's start from scratch, and perhaps you can
> guide me to the simplest possible solution to the problem I first
> came here with... hopefully a solution which is so simple that even
> *I* can't screw it up.
> 
> So, let's consider the following simple hypothetical example...
> 
> We have two directories within the current directory.  They are
> named ./one/ and ./two/
> 
> Within each of these two directories, there is a set of ordinary
> files. The set of filenames present within each of the two
> directories is identical.  Let's just say that each one contains
> the following three ordinary files:
> 
> 001.jpg 002.jpg 003.jpg
> 
> How may I use rsync to obtain a list of _just_ those files within
> these two directories that the cmp command would describe as being
> different between the two directories?  What is the exact command
> line I should use? (And within the output from the given command,
> what must I look for, exactly, that designates files that are
> different, as opposed to those whose content is the same?)
> 
> 
> Regards, rfg
> 
> 
> P.S.  I really do hope that I can get this to work with rsync.  I
> do prefer not reinventing the wheel, but it is starting to seem
> simpler to me if I were to just write a Perl script that would walk
> two directory hierarchies, in parallel, and just repeatedly invoke
> the cmp command on all of the regular files found therein.
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlYMZg8ACgkQVKC1jlbQAQfjgQCdGmGrIm3eol++rnB+fPGXApin
jUEAnAsNTycPDsM4k+Kox5J2RSH4zldP
=QfjT
-END PGP SIGNATURE-

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html