Re: [Eug-lug] rsync problems

[Bob Miller]

Rob Hudson wrote:

> I'm helping a friend set up a sort of backup solution.  He's got a
> co-located Linux web server, and a Linux file server in his office
> behind a firewall.  I set up an rsync script to synchronize (pull) all
> the web files down at night.  The problem is that it's dying about 25%
> of the way through.  From the log error output it appears that since
> rsync can sometimes take a bit of time, something happens to the
> network connection during the sync and drops the connection.  This is
> the closest explanation I could find:
> http://www.mail-archive.com/[EMAIL PROTECTED]/msg14422.html

Maybe the NAT box is dropping the connection due to inactivity.
Try enabling ssh keepalives.  On the client:

$ echo ServerAliveInterval 300 >> ~/.ssh/config

or...

$ man ssh_config# (-:

-- 
Bob Miller  K
[EMAIL PROTECTED]
___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Alan]

On Mon, March 3, 2008 9:29 am, Rob Hudson wrote:
> Hi LUG,
 I set up an rsync script to synchronize (pull) all
> the web files down at night.  The problem is that it's dying about 25%
> of the way through.  From the log error output it appears that since
> rsync can sometimes take a bit of time, something happens to the
> network connection during the sync and drops the connection.

If you're using an SSH connection, Keep-Alives can help a lot with that.
Also, I recommend you check out:

It's been around for a while, and a lot of the rough edges are smoothed out.
No sense in reinventing the wheel if you don't have to.

-ajb


___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Hal Pomeranz]

> > Are you using the "--partial" option for rsync?
> 
> No, but I just read the man page on it.

You should also check out the "--link-dest" option, which is a nice
way of doing "incremental" style backups where you only backup the
files that have been modified since the last backup.

-- 
Hal Pomeranz, Founder/CEO  Deer Run Associates  [EMAIL PROTECTED]
Network Connectivity and Security, Systems Management, Training
___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Garl Grigsby]

Rob Hudson wrote:

Hi LUG,

I'm helping a friend set up a sort of backup solution.  He's got a
co-located Linux web server, and a Linux file server in his office
behind a firewall.  I set up an rsync script to synchronize (pull) all
the web files down at night.  The problem is that it's dying about 25%
of the way through.  From the log error output it appears that since
rsync can sometimes take a bit of time, something happens to the
network connection during the sync and drops the connection.  This is
the closest explanation I could find:
http://www.mail-archive.com/[EMAIL PROTECTED]/msg14422.html

So, what's a good alternate backup strategy?

Goals are:
* Mirror the directories on remote server periodically.
* Would be nice to only pull down differences to save bandwidth but at
this point it's not a priority.
  
Rob,
   What error message are you getting and what command line options are 
you using? Does it stop on the same file each time or after the same 
amount of transfer each time? (i.e. is there any discernible  pattern?).


Garl

___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Ben Barrett]

Rsync --partial still needs to iterate all the files/paths you specify, of
course, so it will still use some system resources... I'd suggest doing some
system load-testing (for rsync differences), and also gaining a sense of how
much file change happens ("ls -lt"?) as well as how much actually matters.
For instance, in the common case of a webapp, would it be beneficial to
ignore the app's temp files, or crucial to include them?  Will you be
wanting DB snapshots to accompany filesystem snapshots -- is a goal to be
able to restore a complete system to a particular point in time?  A common &
desirable use-case is to pull out a single file, so consider the overhead of
un-tarballing if you go that route.  In most business applications, storage
space is cheap and time is costly; while in most home and some hobbyist
environments, time is cheaper than storage|bandwidth|processing, although
the new-product push wants to sell us our time back, it seems :)

Ben

PS - It might matter quite a bit about system load, during the rsync,
depending on the remote host's situation (virtual host/slice on
cluster/colo'd iron/etc).  Consider nicing the rsync since it can run
slowly... also consider using unionfs, for a variety of reasons, and whether
you'll want to lock or snapshot the rsyncing portion of the FS from changes
for the duration of the backup -- might not matter, but OTOH your backups
could end up fairly useless, depending...


On Mon, Mar 3, 2008 at 10:41 AM, Rob Hudson <[EMAIL PROTECTED]> wrote:

> On 3/3/08, Ben Barrett <[EMAIL PROTECTED]> wrote:
> > Are you using the "--partial" option for rsync?
>
> No, but I just read the man page on it.
>
> >  Sticking with rsync for now... do you have any reasons other that
> >  network timeouts to desire a different solution?
>
> No... if it would work, this would be the solution we're looking for.
>
> >  The best approach I've seen using rsync for backups uses a wrapper to
> >  setup a lock-file of sorts,
> >  kicked off [repeatedly] by a cron job... the wrapper script will not
> >  try to do 2 rsyncs at once, in short.
>
> Hmmm, with partial it seems I could set up an hourly cron job (or so)
> rather than a big nightly.  If things are pulled down little by
> little, that might reduce the chance for a bad network disconnects.
> And if it did disconnect, it would just pick up where it left off last
> time.
>
> My other idea was to use tar and do a big monthly tar, then do nightly
> tars of the differences (archival tar?) and scp those down.  I'm not
> sure which might be preferred.
>
> Thanks,
> Rob
> ___
> EUGLUG mailing list
> euglug@euglug.org
> http://www.euglug.org/mailman/listinfo/euglug
>
___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Rob Hudson]

On 3/3/08, Ben Barrett <[EMAIL PROTECTED]> wrote:
> Are you using the "--partial" option for rsync?

No, but I just read the man page on it.

>  Sticking with rsync for now... do you have any reasons other that
>  network timeouts to desire a different solution?

No... if it would work, this would be the solution we're looking for.

>  The best approach I've seen using rsync for backups uses a wrapper to
>  setup a lock-file of sorts,
>  kicked off [repeatedly] by a cron job... the wrapper script will not
>  try to do 2 rsyncs at once, in short.

Hmmm, with partial it seems I could set up an hourly cron job (or so)
rather than a big nightly.  If things are pulled down little by
little, that might reduce the chance for a bad network disconnects.
And if it did disconnect, it would just pick up where it left off last
time.

My other idea was to use tar and do a big monthly tar, then do nightly
tars of the differences (archival tar?) and scp those down.  I'm not
sure which might be preferred.

Thanks,
Rob
___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug

Re: [Eug-lug] rsync problems

[Ben Barrett]

Are you using the "--partial" option for rsync?
The best approach I've seen using rsync for backups uses a wrapper to
setup a lock-file of sorts,
kicked off [repeatedly] by a cron job... the wrapper script will not
try to do 2 rsyncs at once, in short.
Sticking with rsync for now... do you have any reasons other that
network timeouts to desire a different solution?

Ben

PS - you may also want to consider "--partial-dir" and
"--delay-updates" options... and have a wrapper script setup so that
you can confirm when the xfer is in fact complete.  Consider recording
changes via a versioning system on the receiving end, each time a
transfer is noted at complete.


On Mon, Mar 3, 2008 at 9:29 AM, Rob Hudson <[EMAIL PROTECTED]> wrote:
> Hi LUG,
>
>  I'm helping a friend set up a sort of backup solution.  He's got a
>  co-located Linux web server, and a Linux file server in his office
>  behind a firewall.  I set up an rsync script to synchronize (pull) all
>  the web files down at night.  The problem is that it's dying about 25%
>  of the way through.  From the log error output it appears that since
>  rsync can sometimes take a bit of time, something happens to the
>  network connection during the sync and drops the connection.  This is
>  the closest explanation I could find:
>  http://www.mail-archive.com/[EMAIL PROTECTED]/msg14422.html
>
>  So, what's a good alternate backup strategy?
>
>  Goals are:
>  * Mirror the directories on remote server periodically.
>  * Would be nice to only pull down differences to save bandwidth but at
>  this point it's not a priority.
>
>  Thanks,
>  Rob
>  ___
>  EUGLUG mailing list
>  euglug@euglug.org
>  http://www.euglug.org/mailman/listinfo/euglug
>
___
EUGLUG mailing list
euglug@euglug.org
http://www.euglug.org/mailman/listinfo/euglug