Re: [gentoo-user] rotating backup script

[Rich Freeman]

On Tue, Mar 7, 2017 at 12:04 AM,   wrote:
> I was looking at this rotating backup script
>
> source:
> https://community.spiceworks.com/topic/34970-how-to-create-rotating-backups-of-files
>

If you're looking for rotating backup solution based on rsync I'd take
a look at rsnapshot.  It is in the Gentoo repo, and it has been
completely dependable in my experience.

You get the same kind of storage you'd expect with plain rsync (just a
replica directory tree that you can freely read, cp from, etc).
However, it does stuff like ensure backups are complete before
rotating them in, handling the rotation itself, and also linking
incremental backups with hard-links to reduce storage.  It isn't quite
de-duplication but it gets you 90% of the benefit vs having a bunch of
complete backup directories that each take up the capacity of a full
backup.  Basically it copies the last backup using hard links, then
rsyncs over that.  The result is what looks like a bunch of full
backups but without all the space use.

If you're looking for something even more space-efficient that is
still based on rsync but which does not store its files as a simple
replica directory tree then look at duplicity, which is also in the
Gentoo repo.  It stores the data in compact archive files like most
other backup solutions, but it uses librsync to do most of the heavy
lifting.  If 1kb changes inside the middle of a 1TB file it only
stores the extra 1kb, just as rsync would only transfer the 1kb.  It
also has a bunch of backend options, including a few cloud services
like S3.  For remote backups it is pretty smart about how it stores
metadata vs data so that if the local cache gets out of sync it can
just re-fetch the metadata from the cloud without having to retrieve
the actual backup data, and it supports gpg.

I personally use both right now.  High-value files are saved using
duplicity to an S3 backend, fully gpg encrypted.  Since I like to mess
with zfs/btrfs I also keep a full replica of those drives locally on
ext4 using rsnapshot.  This is very easy to restore should btrfs eat
my data (and I've made use of it twice).

Rolling your own can certainly be an educational experience, but IMO
it is unnecessary.  Of course, be sure to test recovery no matter how
you end up setting everything up.

-- 
Rich

Re: [gentoo-user] rotating backup script

[thelma]

On 03/06/2017 11:35 PM, Jean-Christophe Bach wrote:
> Hello,
> 
>> I was looking at this rotating backup script
>>
>> source:
>> https://community.spiceworks.com/topic/34970-how-to-create-rotating-backups-of-files
>>
>> --backup script
>> BACKUPDIR=`date +%A`
>> OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES 
>>   --delete --backup --backup-dir=/$BACKUPDIR -a"
>>
>> export PATH=$PATH:/bin:/usr/bin:/usr/local/bin
>>
>> # the following line clears the last weeks incremental directory
>> [ -d $HOME/emptydir ] || mkdir $HOME/emptydir
>> rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
>> rmdir $HOME/emptydir
>>
>> # now the actual transfer
>> rsync $OPTS $BDIR $BSERVER::$USER/current
>> ---end backup script
>>
>> Can anybody explain why they they "...clear the last weeks incremental 
>> directory"?
> 
> Probably because BACKUPDIR is set to the name of the day (`date +`A`).
> Today is Tuesday and a backup is done, next week the backup will be
> overwritten because BACKUPDIR will also be Tuesday. Therefore there will
> only be 7 directories.
> 
>> Doesn't "rsync --deleate" option take take care of this?
> 
> --delete removes extraneous files. Combined with the previous thing, it
> ensures the backup rotation.
> 
[snip]

Yes, the script clear states there will be 7-backups.
But I see no reason to remove/clear old backup from previous week since
"--delete" will remove all the file on destination that are not in source.

That is just a redundant step, I think.

--
Thelma

Re: [gentoo-user] rotating backup script

[Jean-Christophe Bach]

Hello,

> I was looking at this rotating backup script
> 
> source:
> https://community.spiceworks.com/topic/34970-how-to-create-rotating-backups-of-files
> 
> --backup script
> BACKUPDIR=`date +%A`
> OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES 
>   --delete --backup --backup-dir=/$BACKUPDIR -a"
> 
> export PATH=$PATH:/bin:/usr/bin:/usr/local/bin
> 
> # the following line clears the last weeks incremental directory
> [ -d $HOME/emptydir ] || mkdir $HOME/emptydir
> rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
> rmdir $HOME/emptydir
> 
> # now the actual transfer
> rsync $OPTS $BDIR $BSERVER::$USER/current
> ---end backup script
> 
> Can anybody explain why they they "...clear the last weeks incremental 
> directory"?

Probably because BACKUPDIR is set to the name of the day (`date +`A`).
Today is Tuesday and a backup is done, next week the backup will be
overwritten because BACKUPDIR will also be Tuesday. Therefore there will
only be 7 directories.

> Doesn't "rsync --deleate" option take take care of this?

--delete removes extraneous files. Combined with the previous thing, it
ensures the backup rotation.

> Does it have something to do with Windows? 

Hu? What is Windows? I do not know what are those alternative OSs ;)

Regards,

JC


signature.asc
Description: PGP signature

[gentoo-user] rotating backup script

[thelma]

I was looking at this rotating backup script

source:
https://community.spiceworks.com/topic/34970-how-to-create-rotating-backups-of-files

--backup script
BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES 
  --delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER::$USER/current
---end backup script

Can anybody explain why they they "...clear the last weeks incremental 
directory"?

Doesn't "rsync --deleate" option take take care of this?

Does it have something to do with Windows? 

-- 
Thelma