Re: Options for a "I'm done" flag file

2015-04-29 Thread Mark 

You could try increasing the timeout delay, rather than resume.
rsync will tolerate quite long network dropouts and still carry on.
I have managed to keep an internet transfer of up to 100Gb alive for two 
weeks.


I didn't find --partial to be much use for very large scale transfers 
due to the very cpu intensive checksum process.


By large scale I have rsync'd several Petabytes of backup files up to 
500Gb size over the last five years with good success.



On 29/04/2015 2:49 a.m., Simon Hobson wrote:

As an aside to this, part of the problem I've been having is the transfer 
timing out/getting interrupted during a particular large file (1G, new file, 
2-3 hours if it works).

So I've been experimenting with --partial and --partial-dir=.rsync-partial 
which weren't working. It appears to work at first - if the transfer is 
interrupted, the partial file is correctly saved in the named directory.
Then if I run the script again, it deletes the partial file before starting 
again.

I found that I needed to also specify --delete-delay to avoid deleting the 
partial file before it's used.

Is this "known", because it isn't implied (as I read it) by the --partial-dir 
section in the man page ?



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-29 Thread Mark 

For a push job.

Run the rsync for the files,
if the exit code is 0, create the flag file and then rsync just that 
file on its own.





On 28/04/2015 10:38 p.m., Simon Hobson wrote:

As part of my backup system, I use Rsync to keep a copy of each server on one 
central backup server. This backup server then uses StoreBackup to keep 
multiple iterations of each clone directory.
So that the StoreBackup archives don't keep adding "redundant" and misleading 
backups, I update a flag file with the current date/time before doing the Rsync update, 
and test to see if this file is newer than the one in the latest StoreBackup backup. If 
it isn't, then I skip the StoreBackup for that server.

The end result is that if a system is down or out of communication (one or two are at 
sites that can be offline for days), then the list of backups in StoreBackup will reflect 
that. Eg, if the system did a sync on the 1st, but not on 2nd - 5th, then there will be 
no backups for 2nd-5th, and when looking later I won't be "fooled" into 
thinking that I have a backup from (say) the 4th.

Where this breaks down is if the sync fails part way through. The flag file has 
already been synced, so I have multiple backups which aren't actually complete.
I actually have this at the moment. Just put a small system on a customer site, 
it has a database that creates 1GB journal files (not that it handles anything 
like that volume of data), and at the moment their connectivity is a bit flakey.

My first thought was "do the flag file last", but a quick search confirms what 
I thought - that there isn't an option for this.

So, does anyone have any suggestions how I might reasonably easily get the 
ability for my script to see if the previous sync completed ?




--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-28 Thread Simon Hobson 
As an aside to this, part of the problem I've been having is the transfer 
timing out/getting interrupted during a particular large file (1G, new file, 
2-3 hours if it works).

So I've been experimenting with --partial and --partial-dir=.rsync-partial 
which weren't working. It appears to work at first - if the transfer is 
interrupted, the partial file is correctly saved in the named directory.
Then if I run the script again, it deletes the partial file before starting 
again.

I found that I needed to also specify --delete-delay to avoid deleting the 
partial file before it's used.

Is this "known", because it isn't implied (as I read it) by the --partial-dir 
section in the man page ?

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-28 Thread Simon Hobson 
Lorenz Weber  wrote:

> rsync -avH ${all_gubbins} / user@remote.machine:/dest/ && ssh 
> user@remote.machine touch /etc/donefile

No SSH access between them, only rsync. Besides, it would add the overhead of 
managing ssh access (users and keys) as well as Rsync.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-28 Thread Lorenz Weber 
Or

rsync -avH ${all_gubbins} / user@remote.machine:/dest/ && ssh 
user@remote.machine touch /etc/donefile

so your client touches a file on your server (that sounds so wrong...)

Am 28.04.2015 um 13:36 schrieb Simon Hobson:
> Michael Johnson - MJ  wrote:
> 
>> rsync -av /src/ /dst/ && touch /dst/done
> 
> A, knew I'd miss some detail.
> All the syncs are pushed to the backup server.
> 
> But that does give me an idea. I guess I could do that on the source, then 
> sync the flag file over.
> rsync -avH ${other_gubbins} / user@remote.machine:/dest/ &&
>   touch /etc/donefile &&
>   rsync -av ${some_other_gubbins} /etc/donefile user@remote.machine:/dest/
> 
> That (or some variation of it) could work.
> 

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-28 Thread Simon Hobson 
Michael Johnson - MJ  wrote:

> rsync -av /src/ /dst/ && touch /dst/done

A, knew I'd miss some detail.
All the syncs are pushed to the backup server.

But that does give me an idea. I guess I could do that on the source, then sync 
the flag file over.
rsync -avH ${other_gubbins} / user@remote.machine:/dest/ &&
  touch /etc/donefile &&
  rsync -av ${some_other_gubbins} /etc/donefile user@remote.machine:/dest/

That (or some variation of it) could work.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Options for a "I'm done" flag file

2015-04-28 Thread Michael Johnson - MJ 
rsync -av /src/ /dst/ && touch /dst/done

That should do it as the touch only happens if rsync exits with a code of
0.  If you need to consider other non zero exit code, it is still doable,
just a bit more shell code.

There are surely other options as well, but this is probably the most
simple.
On Apr 28, 2015 3:47 AM, "Simon Hobson"  wrote:

> As part of my backup system, I use Rsync to keep a copy of each server on
> one central backup server. This backup server then uses StoreBackup to keep
> multiple iterations of each clone directory.
> So that the StoreBackup archives don't keep adding "redundant" and
> misleading backups, I update a flag file with the current date/time before
> doing the Rsync update, and test to see if this file is newer than the one
> in the latest StoreBackup backup. If it isn't, then I skip the StoreBackup
> for that server.
>
> The end result is that if a system is down or out of communication (one or
> two are at sites that can be offline for days), then the list of backups in
> StoreBackup will reflect that. Eg, if the system did a sync on the 1st, but
> not on 2nd - 5th, then there will be no backups for 2nd-5th, and when
> looking later I won't be "fooled" into thinking that I have a backup from
> (say) the 4th.
>
> Where this breaks down is if the sync fails part way through. The flag
> file has already been synced, so I have multiple backups which aren't
> actually complete.
> I actually have this at the moment. Just put a small system on a customer
> site, it has a database that creates 1GB journal files (not that it handles
> anything like that volume of data), and at the moment their connectivity is
> a bit flakey.
>
> My first thought was "do the flag file last", but a quick search confirms
> what I thought - that there isn't an option for this.
>
> So, does anyone have any suggestions how I might reasonably easily get the
> ability for my script to see if the previous sync completed ?
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html