Re: Backups with rsync [Was: Is my root partition dying?]

[Chris Samuel via luv-main]

On Mon, 21 Dec 2015 11:25:29 AM Trent W. Buck via luv-main wrote:

> Or see btrfs/ZFS snapshots.  These operate per block rather than per
> inode, so should be more effective and less failure-prone in edge cases
> (hint, there's an upper limit on hard link count).

Yeah, I use both rsnapshot backups on ext4 and btrfs snapshots (both on 
external drives) and where I rsync with --inplace --no-whole-file for the 
btrfs side.

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Russell Coker via luv-main]

On Mon, 21 Dec 2015 11:25:29 AM Trent W. Buck via luv-main wrote:
> One thing rsnapshot does reasonably well is faking multiple tape
> rotations within the snapshot set.  e.g. you say "1 yearly, 2 monthlies,
> and 7 dailies", and it works out which snapshots to expire.
> I don't know how to do that in btrfs/zfs land.

Just write a shell script to delete the ones you don't want.

I have written scripts for ZFS and BTRFS to have a specified number of 
snapshots stored on a less than daily basis (hourly or every 15 minutes) and a 
specified number of daily snapshots.

You could write scripts to have yearly, monthly, etc.  It's just a bit of 
extra shell scripting.  However the benefits are less (in most cases the vast 
majority of data that's been around for a couple of months will be around for 
several years).  Disk space is all that limits you.

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Adam Bolte via luv-main]

On Mon, Dec 21, 2015 at 11:25:29AM +1100, Trent W. Buck via luv-main wrote:
> The active ingredient of rsnapshot is "rsync --link-dest",
> if you want to roll your own rsnapshot replacement.
> 
> rsnapshot config mandates annoying literal tabs, has a directory
> structure that's not directly compatible with samba's Shadow Copy
> modules, and doesn't correctly handle a file being deleted, then readded
> (i.e. pass --link-dest to ALL snapshots, not just the last one).

That's the way my scripts work, except I was not aware you could now
have multiple --link-dest arguments. Thanks for pointing that out.

The only downside to using --link-dest is that changes to the backed
up file permissions and ownership can get lost and should be tracked
separately if needed. At least, I vaguely recall noticing that being a
problem when I write the scripts years ago.

-Adam



signature.asc
Description: Digital signature
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Trent W. Buck via luv-main]

Erik Christiansen via luv-main
 writes:

> On 18.12.15 22:45, Chris Samuel via luv-main wrote:
>> On Thu, 17 Dec 2015 11:06:10 PM Erik Christiansen via luv-main wrote:
>>
>>> I'm not grokking the benefit of doing the rsync _and_ a "cp -al".
>>
>> The idea is to have snapshots over time, rather than a single
>> snapshot.  For instance as implemented in: http://rsnapshot.org/
>
> [...] I didn't realise was that the links appear to be between
> unchanged files across backup versions, to save space.
>
> It's around 8 years since I used an HP tape carousel for daily + weekly
> backups at work. Not doing that any more, so have devolved to just a
> couple of rotating snapshots, with no need to link between them to save
> space.

The active ingredient of rsnapshot is "rsync --link-dest",
if you want to roll your own rsnapshot replacement.

rsnapshot config mandates annoying literal tabs, has a directory
structure that's not directly compatible with samba's Shadow Copy
modules, and doesn't correctly handle a file being deleted, then readded
(i.e. pass --link-dest to ALL snapshots, not just the last one).

Or see btrfs/ZFS snapshots.  These operate per block rather than per
inode, so should be more effective and less failure-prone in edge cases
(hint, there's an upper limit on hard link count).


One thing rsnapshot does reasonably well is faking multiple tape
rotations within the snapshot set.  e.g. you say "1 yearly, 2 monthlies,
and 7 dailies", and it works out which snapshots to expire.
I don't know how to do that in btrfs/zfs land.

___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Erik Christiansen via luv-main]

On 18.12.15 22:45, Chris Samuel via luv-main wrote:
> On Thu, 17 Dec 2015 11:06:10 PM Erik Christiansen via luv-main wrote:
> 
> > I'm not grokking the benefit of doing the rsync _and_ a "cp -al". I just
> > include -aH in my rsync options, the -H to preserve hard links. Seems to
> > work.
> 
> The idea is to have snapshots over time, rather than a single snapshot.  For 
> instance as implemented in:
> 
> http://rsnapshot.org/

Thanks Chris, that was not in doubt. What I didn't realise was that the
links appear to be between unchanged files across backup versions, to
save space. (Whereas my local concern has only been to ensure that the
backup is not distorted by failure to preserve the fs structure.)

It's around 8 years since I used an HP tape carousel for daily + weekly
backups at work. Not doing that any more, so have devolved to just a
couple of rotating snapshots, with no need to link between them to save
space. 

Erik
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Chris Samuel via luv-main]

On Thu, 17 Dec 2015 11:06:10 PM Erik Christiansen via luv-main wrote:

> I'm not grokking the benefit of doing the rsync _and_ a "cp -al". I just
> include -aH in my rsync options, the -H to preserve hard links. Seems to
> work.

The idea is to have snapshots over time, rather than a single snapshot.  For 
instance as implemented in:

http://rsnapshot.org/

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Russell Coker via luv-main]

On Thu, 17 Dec 2015 11:06:10 PM Erik Christiansen via luv-main wrote:
> On 17.12.15 21:33, Russell Coker via luv-main wrote:
> > There are a variety of backup systems that start with rsync and manage
> > trees of links.  It's not difficult to write your own, rsync the
> > files, run "cp -al" to make a copy with hard links and use today's
> > date in the directory name, and then delete backup directories that
> > are too old.
> 
> I'm not grokking the benefit of doing the rsync _and_ a "cp -al". I just
> include -aH in my rsync options, the -H to preserve hard links. Seems to
> work.

rsync copies all the files.

Then you do something like the following to preserve a version of that tree 
before you do the next rsync.
cp -al current 2015-12-18

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Re: Backups with rsync [Was: Is my root partition dying?]

[Adam Bolte via luv-main]

On 17/12/15 23:06, Erik Christiansen via luv-main wrote:
> On 17.12.15 21:33, Russell Coker via luv-main wrote:
>> There are a variety of backup systems that start with rsync and manage
>> trees of links.  It's not difficult to write your own, rsync the
>> files, run "cp -al" to make a copy with hard links and use today's
>> date in the directory name, and then delete backup directories that
>> are too old.
> 
> I'm not grokking the benefit of doing the rsync _and_ a "cp -al". I just
> include -aH in my rsync options, the -H to preserve hard links. Seems to
> work.

At my workplace, we use this rsync option:

--link-dest=DIR hardlink to files in DIR when unchanged

where DIR would be the most recent successful backup directory.

I wrote aboute 500 lines of Bash for my workplace years ago which is
basically an rsync wrapper that also reads hosts and directories to
backup from a config file, and does some basic directory management to
make it easy to browse backups and see what you are restoring.

Today after reading this thread, I asked and was granted permission to
make the git repo public. It's a bit hackish, but it has served us
well and might be worth a look.

https://github.com/sitepoint/abs

Since it backs up hosts on the network using rsync over SSH via pubkey
authentication, we use the rrsync wrapper script (typically installed
at /usr/share/doc/rsync/scripts/rrsync.gz in the rsync package), with
authorized_keys files like:

command="/usr/local/sbin/rrsync -ro /" ssh-rsa  

and

PermitRootLogin forced-commands-only

in the /etc/ssh/sshd_config option on targeted hosts to improve
security.

Cheers,
Adam


signature.asc
Description: Digital signature
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main

Backups with rsync [Was: Is my root partition dying?]

[Erik Christiansen via luv-main]

On 17.12.15 21:33, Russell Coker via luv-main wrote:
> There are a variety of backup systems that start with rsync and manage
> trees of links.  It's not difficult to write your own, rsync the
> files, run "cp -al" to make a copy with hard links and use today's
> date in the directory name, and then delete backup directories that
> are too old.

I'm not grokking the benefit of doing the rsync _and_ a "cp -al". I just
include -aH in my rsync options, the -H to preserve hard links. Seems to
work.

It's almost enough to make one wonder why it's a little bit fiddly to make
rsync "just shuddup & copy the listed bits of the filesystem _as_is_, so
they can be restored unaltered."

> What I'm doing for backups at the moment is to rsync to files on a
> BTRFS filesystem and then create a snapshot.  If I want to retrieve a
> file that was deleted then I just copy it from a suitably old
> snapshot.

A bit more basic here. Just a rsync -Hauv to one of several flash
sticks, then a diff -qr to confirm the copy's OK, and show any deletions
which should be done, usually 0-3 per backup. But then my precious data
is diminutive by most standards. Most important is that they come with
me off-site.

Erik
___
luv-main mailing list
luv-main@luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main