Re: Recommendation: Backup system

[Richard Hector]

On 07/10/16 23:19, Jonathan Dowland wrote:
> I don't know whether dirvish does something to improve matters, but with
> hard link trees, if you have lots of little files (such as Maildir archives
> of busy mailing lists like LKML), the amount of space consumed by the file
> system metadata to represent the trees is very high, getting towards the
> size used for the files themselves. This is compounded on some filesystems
> (the ext family amongst them) that only ever increase the amount of space
> for storing dirent metadata, never decrease it, so if you have a Maildir
> which has high churn, and you move from having a large amount of mail to
> a small amount, the storage allocated for metadata for the directory will
> remain large.

The metadata you're talking about is merely the directory entries,
right? I'm still searching, but haven't yet found info on how much space
a directory entry takes up. Any pointers would be most welcome :-)

I'm on xfs, btw.

Regardless, in the trivial case where nothing has changed between
backups, the directories in the new backup must take up exactly the same
space as the ones in the old backup, where both point to the same files.
So whether there's one set of links or several, it's still the case that
for small files, the directory entries will take proportionately more
space relative to the file - that's inevitable.

I think I'm right in saying that in the dirvish (or similar) use case,
the number of directory entries in a given directory will never decrease
anyway - except in the odd case where I discover I've backed up
something I shouldn't have (eg a huge or confidential file), and gone
through purging it from all the backups.

Richard




signature.asc
Description: OpenPGP digital signature

Re: Recommendation: Backup system

[Dave Thayer]

On Mon, Oct 10, 2016 at 10:55:44PM +0100, Jonathan Dowland wrote:
> On Mon, Oct 10, 2016 at 09:32:02AM -0400, Celejar wrote:
> > Am I being unreasonable? You are certainly more of an expert than I - I
> > suppose you find that it is quality software, and better than
> > rsnapshot, despite basically being dead?
>  
> Lack of bug and development activity is certainly a worrying sign of a program
> with problems. But development activity naturally dies off if a program does
> what it is supposed to do, so it's not always a certainty that it is doomed.
> It remains a concern for me, but regardless the tool has worked very well.
> 

It looks like rdiff-backup has a new maintainer as of February 2016, sol1. See
. There may be some progress forthcoming
after all. 

dt

-- 
Dave Thayer / Denver, Colorado USA / d...@thayer-boyle.com 
 Whenever you read a good book, it's like the author is right there, in
 the room talking to you, which is why I don't like to read good books.
 - Jack Handey "Deep Thoughts"

Re: Recommendation: Backup system

[Jonathan Dowland]

On Mon, Oct 10, 2016 at 09:32:02AM -0400, Celejar wrote:
> Interesting, thanks. I've been using rsnapshot for years, and am
> basically satisfied with it, although the performance when run on my
> T61 laptop (backing up to a (slow) USB external disk) is indeed painful
> (I do have largeish Maildirs). [Interestingly, when run on my ARM
> (Kirkland) NAS, pulling from the laptop over ethernet / wifi,
> performance seems much better ...]

The first machine where I really had performance problems
with rsnapshot was an ARM-powered Thecus N2100, but that was a pretty
weak CPU from what I recall.

> One thing I really like about rsnapshot is how the backups are all
> stored just as the files themselves, without any special formats, and
> can therefore be inspected / restored from using just the ordinary
> filesystem tools.

That is a nice property, yes. the rdiff-backup format is not quite this,
but is well specified outside of the code (from the last time I looked)
to give me confidence I could pull my files out by hand if I needed to.

I'm not sure if I mentioned it in this thread or not, but I actually began a
third party tool to parse rdiff-backup format backups[1], and there exists
another third party tool to do the same thing[2].

> A number of years ago I looked at rdiff-backup, but dropped it due to
> suspicions of the code quality: the project seemed to be dead, and even
> significant bugs were being ignored:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623336
> https://lists.debian.org/debian-user/2012/10/msg00182.html
> 
> I see that you, too, have a report that's been ignored for more than 5
> years ;)

Yes, that's true. The bug that I remember reporting (haven't looked it back up
to be sure) was that reverting a partial (failed) backup requires some disk
space and so fails if the backup device is full - this is a big problem in
theory, but in practice I'm running my backup jobs as non-root, so there's
always the 5% or so reserved space for root users. So in this situation I
can have the roll-back occur as root.

> Am I being unreasonable? You are certainly more of an expert than I - I
> suppose you find that it is quality software, and better than
> rsnapshot, despite basically being dead?
 
Lack of bug and development activity is certainly a worrying sign of a program
with problems. But development activity naturally dies off if a program does
what it is supposed to do, so it's not always a certainty that it is doomed.
It remains a concern for me, but regardless the tool has worked very well.

[1] https://jmtd.net/software/rdifffs/
[2] https://github.com/rbrito/rdiff-backup-fs

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Recommendation: Backup system

[Celejar]

On Mon, 3 Oct 2016 10:48:57 +0100
Jonathan Dowland  wrote:

> On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> > Make a long story short:
> > Have you guys a recommendation for me?
> > Is there a specific application you use for your backups guys?
> 
> rdiff-backup[1]. I don't know what your NAS is, whether it's an off-the-shelf
> thing or a DIY system, but I tend to initiate the backups on my NAS (which is
> a Debian-powered mini itx PC). You can initiate from the clients instead if
> you prefer.
> 
> rdiff-backup is, conceptually, quite similar to the level of detail you are
> working at with your script, so it should be easy to get working. I'd also
> consider looking at Obnam[2], and I'd avoid rsnapshot if I were you (it
> falls over with large quantities of files, such as mailboxes, and rdiff-backup
> basically does the same job but better).

Interesting, thanks. I've been using rsnapshot for years, and am
basically satisfied with it, although the performance when run on my
T61 laptop (backing up to a (slow) USB external disk) is indeed painful
(I do have largeish Maildirs). [Interestingly, when run on my ARM
(Kirkland) NAS, pulling from the laptop over ethernet / wifi,
performance seems much better ...]

One thing I really like about rsnapshot is how the backups are all
stored just as the files themselves, without any special formats, and
can therefore be inspected / restored from using just the ordinary
filesystem tools.

A number of years ago I looked at rdiff-backup, but dropped it due to
suspicions of the code quality: the project seemed to be dead, and even
significant bugs were being ignored:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623336
https://lists.debian.org/debian-user/2012/10/msg00182.html

I see that you, too, have a report that's been ignored for more than 5
years ;)

Am I being unreasonable? You are certainly more of an expert than I - I
suppose you find that it is quality software, and better than
rsnapshot, despite basically being dead?

Celejar

Re: Recommendation: Backup system

[Jonathan Dowland]

On Wed, Oct 05, 2016 at 03:06:58PM +1300, Richard Hector wrote:
> Can you elaborate on the performance issues? I'm using dirvish for my
> maildirs (dovecot imap server), without noticeable problems.

I don't know whether dirvish does something to improve matters, but with
hard link trees, if you have lots of little files (such as Maildir archives
of busy mailing lists like LKML), the amount of space consumed by the file
system metadata to represent the trees is very high, getting towards the
size used for the files themselves. This is compounded on some filesystems
(the ext family amongst them) that only ever increase the amount of space
for storing dirent metadata, never decrease it, so if you have a Maildir
which has high churn, and you move from having a large amount of mail to
a small amount, the storage allocated for metadata for the directory will
remain large.

On a low-powered device like an embedded NAS, performing operations like
delete (such as during a backup rotation) on a large hardlink tree can be
very slow and require a lot of CPU time.

> My current challenge is to back up windows boxes - if I can get rsync to
> work (maybe DeltaCopy? Not sure if that will work how I want), I guess
> I'll be stuck doing a local rsync of a smbfs mount ... unless someone
> has a better suggestion.

Windows has an rsync-like tool called Robocopy, but I'd skip straight to
Backup PC for backing up Windows machines.

-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Recommendation: Backup system

[Richard Hector]

On 05/10/16 16:03, Daniel Bareiro wrote:
> Hi, Richard.
> 
> On 04/10/16 23:06, Richard Hector wrote:
> 
>> My current challenge is to back up windows boxes - if I can get
>> rsync to work (maybe DeltaCopy? Not sure if that will work how I
>> want), I guess I'll be stuck doing a local rsync of a smbfs mount
>> ... unless someone has a better suggestion.
> 
> Some time ago I started a thread [1] on the list for this topic.
> The Cygwin SSH server seems to run smoothly although I have not
> yet implemented it on a daily basis.

Ah, thanks. I think I even read it at the time, but didn't have the
immediate need, so didn't take so much in :-)

It depends a bit on how involved the setup is - the Windows machines
aren't mine (they're my parents'), and I don't want anything too
confusing for their regular Windows helper ... or for me, for that
matter; I don't touch Windows very often :-)

Richard

Re: Recommendation: Backup system

[Daniel Bareiro]

Hi, Richard.

On 04/10/16 23:06, Richard Hector wrote:

> My current challenge is to back up windows boxes - if I can get rsync to
> work (maybe DeltaCopy? Not sure if that will work how I want), I guess
> I'll be stuck doing a local rsync of a smbfs mount ... unless someone
> has a better suggestion.

Some time ago I started a thread [1] on the list for this topic. The
Cygwin SSH server seems to run smoothly although I have not yet
implemented it on a daily basis.


Kind regards,
Daniel

[1] https://lists.debian.org/debian-user/2016/08/threads.html#00040



signature.asc
Description: OpenPGP digital signature

Re: Recommendation: Backup system

[Richard Hector]

On 04/10/16 01:45, Markus Grunwald wrote:
> Hello Teemu,
>
>>> rsync, whilst an awesome piece of software, is not, on its own, a
>>> backup system.
>>
>> Yes. With some scripting I think "rsync" with "--link-dest" is quite
>> ideal for incremental backups. Unchanged files are created as hard
>> links for the previous backup files. Every backup generation is just
>> a normal and complete file system tree.
>
> I haven't followed this thread closely, but:
>
> To everybody who ponders using rsync for backup, I strongly suggest a
> closer look at "dirvish". I'm using it to backup servers, laptops,
> raspberries. It is not hard to configure, uses rsync with hard links
> for "incremental" backups and keeps older versions of the backups.

That's what I use too. I use it with rsync over ssh as root, with the
remote set up for root to run forced commands only, and in order to back
up different trees independently, I use different ssh keys, each with
its own forced command. That's a bit fiddly to set up, but seems to work ok.

On 04/10/16 04:06, Jonathan Dowland wrote:
> That's basically a less convenient rsnapshot, with all the caveats
> (bad performance for large trees of files like mailboxes)

Can you elaborate on the performance issues? I'm using dirvish for my
maildirs (dovecot imap server), without noticeable problems.

One change I do make is to enable 'dateext' in my logrotate config, so I
don't end up with endless duplicates of my logfiles due to the names
changing.

My current challenge is to back up windows boxes - if I can get rsync to
work (maybe DeltaCopy? Not sure if that will work how I want), I guess
I'll be stuck doing a local rsync of a smbfs mount ... unless someone
has a better suggestion.

Richard

Re: Recommendation: Backup system

[Jarle Aase]

I have moved from simple scripts to simple scripts with zbackup in them :)

Then I rsync the zbackup directories from different machines to my 
central  backup disks (and distribute from there to cloud storage and 
off-site disks).


zbackup supports deduplication and encryption, and is really a nice 
piece of software.



Jarle


Den 01. okt. 2016 12:37, skrev mo:

Hi Debian users :)

Information:
Distributor ID:Debian
Description:Debian GNU/Linux 8.6 (jessie)
Release:8.6
Codename:jessie

As the title say i'm in search for a backup application/system.
Currently i manage my backups with a little script that i wrote... but 
it does not really serve my needs anymore.
I want to be able to make backups on my main PC and also on my server, 
the backups i would then store on my NAS.


Make a long story short:
Have you guys a recommendation for me?
Is there a specific application you use for your backups guys?

Btw: I dont mind configuring or playing around with new applications, 
every recommendation is welcome ;)



Here is my current backup script (Which is run by cron daily):
#!/bin/bash

TO_BACKUP="/home /etc /var/log"
BACKUP_DIR="/var/backup"
BACKUP_ARCHIVE="backup-`date +%d_%m_%Y-%H:%M`.tar"
TAR_OPTIONS='-cpf'

delete_old_backup() {
if [ -f ${BACKUP_DIR}/backup*.tar ]; then
rm -rf $BACKUP_DIR/backup*
fi
}

create_new_backup() {
tar $TAR_OPTIONS ${BACKUP_DIR}/$BACKUP_ARCHIVE $TO_BACKUP
}

main() {
delete_old_backup
create_new_backup
}

main

Greets
mo

Re: Recommendation: Backup system

[Jonathan Dowland]

On Mon, Oct 03, 2016 at 02:59:19PM +0300, Teemu Likonen wrote:
> Yes. With some scripting I think "rsync" with "--link-dest" is quite
> ideal for incremental backups. Unchanged files are created as hard links
> for the previous backup files. Every backup generation is just a normal
> and complete file system tree.

That's basically a less convenient rsnapshot, with all the caveats (bad
performance for large trees of files like mailboxes)


-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Recommendation: Backup system

[Markus Grunwald]

Hello Teemu,

> > rsync, whilst an awesome piece of software, is not, on its own, a
> > backup system.
>
> Yes. With some scripting I think "rsync" with "--link-dest" is quite
> ideal for incremental backups. Unchanged files are created as hard links
> for the previous backup files. Every backup generation is just a normal
> and complete file system tree.

I haven't followed this thread closely, but:

To everybody who ponders using rsync for backup, I strongly suggest a
closer look at "dirvish". I'm using it to backup servers, laptops,
raspberries. It is not hard to configure, uses rsync with hard links
for "incremental" backups and keeps older versions of the backups.

--
Markus Grunwald

Fragen zur Mail? http://www.the-grue.de/mail_und_co
http://www.the-grue.de/~markus/markus_grunwald.gpg

Re: Recommendation: Backup system

[Teemu Likonen]

Jonathan Dowland [2016-10-03 10:48:57+01] wrote:

> rsync, whilst an awesome piece of software, is not, on its own, a
> backup system.

Yes. With some scripting I think "rsync" with "--link-dest" is quite
ideal for incremental backups. Unchanged files are created as hard links
for the previous backup files. Every backup generation is just a normal
and complete file system tree.

-- 
/// Teemu Likonen   - .-..    //
// PGP: 4E10 55DC 84E9 DFF6 13D7 8557 719D 69D3 2453 9450 ///


signature.asc
Description: PGP signature

Re: Recommendation: Backup system

[Jonathan Dowland]

On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> Make a long story short:
> Have you guys a recommendation for me?
> Is there a specific application you use for your backups guys?

rdiff-backup[1]. I don't know what your NAS is, whether it's an off-the-shelf
thing or a DIY system, but I tend to initiate the backups on my NAS (which is
a Debian-powered mini itx PC). You can initiate from the clients instead if
you prefer.

rdiff-backup is, conceptually, quite similar to the level of detail you are
working at with your script, so it should be easy to get working. I'd also
consider looking at Obnam[2], and I'd avoid rsnapshot if I were you (it
falls over with large quantities of files, such as mailboxes, and rdiff-backup
basically does the same job but better).

rsync, whilst an awesome piece of software, is not, on its own, a backup
system.

When evaluating or building a backup system, consider all of the points in
"The Tao of Backup"[3]. Finally, remember that RAID is not backup[4]. (JWZ's
advice about a 3rd backup disk here is particularly good. I presently use
rsync for my 3rd backup drive.)

Penultimately, I'll echo what another poster has said, backuppc[5] is very good,
especially if you have windows clients (I realise you don't), although the
backend storage (last time I looked) was a somewhat awkward format that I
think would be painful to extract stuff from in an emergency situation should
the BackupPC software itself fail. As for Bacula, I *think* it's more geared
towards an enterprise-level, perhaps using tape backups, like Amanda. I know
you can use Bacula w/o tapes (Amanda too) but that's kind-of where it came
from and it still "feels" like a tape-oriented system to me (others are welcome
to jump to Bacula's defence here)

Finally, and although Tao of Backup covers this I'll repeat it: test restores.
Test how easy restores are for any tool of choice. Restores, restore performance
and restore UI is infinitely more important than the backup process.

[1] http://rdiff-backup.nongnu.org
[2] http://obnam.org/
[3] http://www.taobackup.com
[4] https://www.jwz.org/doc/backups.html
[5] http://backuppc.sourceforge.net


-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Recommendation: Backup system

[Bob Weber]

On 10/02/2016 08:50 AM, rhkra...@gmail.com wrote:
>> Am 01.10.2016 um 23:06 schrieb Bob Weber:
>>> Like I said backuppc uses incremental and full backups.  The web
>>> interface lets you browse any backup (inc or full) and you see all the
>>> files backed up.  I set the incremental for each day up to a week.  So I
>>> have up to 7 of them.  The full can kept for for however long you want.
>>> I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
>>> that covers almost a year.
> I am not the op, but backuppc sounded pretty nice, so yesterday I tried to 
> install it on both my Wheezy and Jessie systems.  It didn't work (with 
> different failures) on either system--I won't give much detail for now, but 
> I'd 
> just ask a few questions:
>
>* what system (what version of Debian) are you using?
>
>* should I expect that it will properly configure a web server (on the 
> Wheezy system it talked about apache2, iirc), or must I have a properly 
> configured web server before installing backuppc?
>
> Some cryptic notes on the failures:
>
> On wheezy, I thought the installation completed successfully--it ran 
> something 
> it called a script, and, in or after the script it gave me a url to log in to 
> manage backuppc along with a username and password.  When I tried to go to 
> that URL, using either http or https on either of my browsers, it gave me a 
> 404 error.
>
> On jessie, it apparently did not complete the installation, it told me it 
> could not run that initial script.
>
> Suggestions?
>
>

I am running stable 8.5 on my backup machine.  It is a 386 install.  Backuppc is
version backuppc/stable,now 3.3.0-2 i386 [installed]

It uses apache2 (dont know if it will use other web servers) and has put the
following in the apache  /etc/apache2/conf-enabled/backuppc.conf file:

Alias /backuppc /usr/share/backuppc/cgi-bin/


AllowOverride None
Allow from all

# Uncomment the line below to ensure that nobody can sniff importanti
# info from network traffic during editing of the BackupPC config or
# when browsing/restoring backups.
# Requires that you have your webserver set up for SSL (https) access.
#SSLRequireSSL

Options ExecCGI FollowSymlinks
AddHandler cgi-script .cgi
DirectoryIndex index.cgi

AuthUserFile /etc/backuppc/htpasswd
AuthType basic
AuthName "BackupPC admin"
require valid-user



So I would assume that there should be a working install of apache2 before
backuppc is installed.  The only install problem I remember (this was quite a
while ago) was that I wanted the backups put in a directory mounted on a
separate partition.  Even though there was a setting for the backup directory in
backuppc the directory is hard coded to  "/var/lib/backuppc" in some of the
installed backuppc programs.  So to use another location you have to symbolic
link /var/lib/backuppc to that directory before install (or mount the partition
on /var/lib/backuppc).  So if you want to use another directory delete
/var/lib/backuppc and make the link (link -s /var/lib/backuppc  /somewhere-else
  ormount /dev/sdxx /var/lib/backuppc).  Then run "apt-get install
--reinstall backuppc" and hopefully things will be setup correctly at that new
location.  Since I access the backup server on my local net I don't use https. 

I use rsync over ssh to connect to linux servers so the data transferred is over
a secure tunnel. So my backup of a remote vm is secure.  The backuppc user
should have public key that is placed in the authorized_keys file of the clients
that are to be backed up. http://backuppc.sourceforge.net/faq/ssh.html explains
this procedure.

Hope this helps.  I have run backuppc over 10 years at several locations where I
have worked and at home.  It just seems to run.

...Bob

Re: Recommendation: Backup system

[rhkramer]

> Am 01.10.2016 um 23:06 schrieb Bob Weber:
> > Like I said backuppc uses incremental and full backups.  The web
> > interface lets you browse any backup (inc or full) and you see all the
> > files backed up.  I set the incremental for each day up to a week.  So I
> > have up to 7 of them.  The full can kept for for however long you want.
> > I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
> > that covers almost a year.

I am not the op, but backuppc sounded pretty nice, so yesterday I tried to 
install it on both my Wheezy and Jessie systems.  It didn't work (with 
different failures) on either system--I won't give much detail for now, but I'd 
just ask a few questions:

   * what system (what version of Debian) are you using?

   * should I expect that it will properly configure a web server (on the 
Wheezy system it talked about apache2, iirc), or must I have a properly 
configured web server before installing backuppc?

Some cryptic notes on the failures:

On wheezy, I thought the installation completed successfully--it ran something 
it called a script, and, in or after the script it gave me a url to log in to 
manage backuppc along with a username and password.  When I tried to go to 
that URL, using either http or https on either of my browsers, it gave me a 
404 error.

On jessie, it apparently did not complete the installation, it told me it 
could not run that initial script.

Suggestions?

Re: Recommendation: Backup system

[mo]

Am 02.10.2016 um 12:55 schrieb Dan Purgert:

mo wrote:

Am 02.10.2016 um 02:47 schrieb Dan Purgert:

mo wrote:

Maybe this is a little OT, but what kind of backup strategy would you
guys recommend? (Any advice Gene? :) )


If it *must* survive, 3-2-1 is the way to go.

3 copies (Original, Backup, and Backup of the Backup)
2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
1 stored offsite


Except for the offsite storage (Just for my private data) this sounds
perfect for my needs. Good thing is that i have two free HDD's here with
2TB each, so this would work ;)


Yeah, I don't have a spare copy offsite either -- but I don't
(currently) have any electronic data that would be absolutely
devastating if I lost it.  If / when I have kids or something, then baby
pics among other things will likely have that status.


But i guess i should consider it, loosing all the data is quite hard 
(Had that a while ago).



[snip]
One question Dan: What do u use to encrypt your files? also openssl?


I just use PGP.



I need to delve into PGP a little more ;) (Currently i use the openssl 
command to do my encrpytion on the backup archives)

Re: Recommendation: Backup system

[Dan Purgert]

mo wrote:
> Am 02.10.2016 um 02:47 schrieb Dan Purgert:
>> mo wrote:
>>> Maybe this is a little OT, but what kind of backup strategy would you
>>> guys recommend? (Any advice Gene? :) )
>>
>> If it *must* survive, 3-2-1 is the way to go.
>>
>> 3 copies (Original, Backup, and Backup of the Backup)
>> 2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
>> 1 stored offsite
>
> Except for the offsite storage (Just for my private data) this sounds 
> perfect for my needs. Good thing is that i have two free HDD's here with 
> 2TB each, so this would work ;)

Yeah, I don't have a spare copy offsite either -- but I don't
(currently) have any electronic data that would be absolutely
devastating if I lost it.  If / when I have kids or something, then baby
pics among other things will likely have that status.

> [snip]
> One question Dan: What do u use to encrypt your files? also openssl?

I just use PGP.

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Re: Recommendation: Backup system

[mo]

Am 02.10.2016 um 02:47 schrieb Dan Purgert:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

mo wrote:

Maybe this is a little OT, but what kind of backup strategy would you
guys recommend? (Any advice Gene? :) )


If it *must* survive, 3-2-1 is the way to go.

3 copies (Original, Backup, and Backup of the Backup)
2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
1 stored offsite


Except for the offsite storage (Just for my private data) this sounds 
perfect for my needs. Good thing is that i have two free HDD's here with 
2TB each, so this would work ;)



That being said, most of my stuff isn't of the "absolutely must survive"
nature, so I just have it on external HDDs.  If something is
particularly important, it's on HDDs and also somewhere like google
drive / dropbox (although, encrypted when at those places).


I currently use dropbox myself to store my conf files (gzipped and then 
encrypted with aes) in case something bad should ever happen to my NAS, 
so that i can at least restore my servers quickly ;)


One question Dan: What do u use to encrypt your files? also openssl?


-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJX8FjGAAoJEI4R3fMSeaKBX9gH/i6qw2GOY/LSE5haz8uv6u/X
GNT6m44c6olrnzGtKFzEPae3ELTTAT/t47yet2noAR+AS9aJzhtR86l3JfQfu4ug
0XplxY/GkkrEpIeMFrqKy2hc3u++PoEHEBkQut46x6QAw/85ieKs2tsbmfeyUuF7
y5gFbGlowcy3NtSdGIMR6qzn67I+DaO2veTMV3Z+/aR+fAegZtvv/t3CCi1dsmRW
kEWIuPyGR//Foy/vkQsakpHJnl9AvJREB5/T+zRX3pkBXKkzQrIfe80eJYAm5Z8w
rhvZc9lLbO/bVcSvLwRKqYtDmWRY4AeVrqmCitiuXf54KjwZY5Lh16LVq+KfCiQ=
=YzZ4
-END PGP SIGNATURE-



Greets

mo

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 23:06 schrieb Bob Weber:

Like I said backuppc uses incremental and full backups.  The web
interface lets you browse any backup (inc or full) and you see all the
files backed up.  I set the incremental for each day up to a week.  So I
have up to 7 of them.  The full can kept for for however long you want.
I currently keep 12 weekly, 8 bi-weekly and 4 monthly full backups so
that covers almost a year.


That would serve my needs well also :) (For the moment it is just my 
private data - But either way, data should always be backed up ;) )

Thanks for the info ;)


There is another solution you might like called rsnapshot.  I use it to
backup just my root directory on my desktop before I do updates.  That
way if something goes wrong I can boot into a rescue cd and restore the
system to the state before the update.  I just can't afford to have my
desktop to break.  rsnapshot uses rsync so it can backup any computer
that has rsync.  It uses hard links so duplicate files are only stored
once.  You specify how many backups you want to keep and rsnapshot
deletes older ones over that max before adding the new one.  That way
you always have backups (assuming you set the count greater that 1) that
will be there even if there is a transfer error.  This is similar to
your script but is very versatile.


I heard about it but never used it so far, i should give that a try for 
sure :)




*...Bob*
On 10/01/2016 04:42 PM, mo wrote:

Maybe this is a little OT, but what kind of backup strategy would you
guys recommend? (Any advice Gene? :) )






Greets

mo

Re: Recommendation: Backup system

[Gene Heskett]

On Saturday 01 October 2016 13:54:29 Clive Menzies wrote:

> On 01/10/16 18:40, Gene Heskett wrote:
> > On Saturday 01 October 2016 12:39:58 Clive Menzies wrote:
> >> Quick question. Are your backups incremental or complete every
> >> night?
> >
> > This is probably better explained in the manpages. Amanda has the
> > concept of doing a full backup of everything in its disklist
> > according to the days you give it. Amanda will then shuffle the
> > schedule such that those full backups are done at random in that
> > cycle, with an eye toward equalizing, as much as possible, the
> > amount of data saved during each run. It does this by advanceing the
> > the level 0's as it will not let a given file go beyond that cycle
> > before a full copy is made again. Level 0 is a full copy of a file,
> > level 1 is whats been changed in that file since the last level 0.
> > Level 2 is whats been changed since level 1, ad infinitum but most
> > useage never gets past level 2.  I have the drive I use for amanda
> > setup as 30 virtual tapes, using one a day, then recycle. With 4
> > machines feeding amanda, that 1Tb drive stays at around 46% used:
> >
> > gene@coyote:~$ df /amandatapes
> > Filesystem 1K-blocks  Used Available Use% Mounted on
> > /dev/sdc2  960798056 410555684 501429932  46% /amandatapes
> >
> > I stayed on the devel branch of amanda, playing the part of the
> > canary in the coal mine for years while it was being heavily
> > developed, but apparently that grant ran out so not a lot has
> > changed in around 5 years. So while I have a self made, slightly
> > newer version on this machine as master, v3.3.7p1, the slaves are
> > all running 3.3.1 clients from the wheezy repo.  And it Just
> > Works(TM).
> >
> > I would be the first to point out that my way is NOT for archival
> > storage due to this 30 day and recycle setup. I could extend it to
> > 60 days on this drive I suppose, but this is not a business.
> >
> > For a business, I would include the price of the drive as a CODB,
> > fill it up and put it on the shelf at a remote location so it
> > doesn't all go up in smoke when the place burns, thereby giving me
> > the ability to recover something 5+ years old, or for however long
> > one has had that setup running.
> >
> > That $100 or less a month for a new commodity drive is far less of a
> > CODB than the archival storage of tapes would be over a 10 year
> > period. And you would have to add in that the tape drive(s) would be
> > out of service for about a month each annually while they spend the
> > holidays in Oklahoma City getting fresh heads and rubber at about a
> > kilobuck each for the rebuild.  Thats been the track record here in
> > my usage of tapes. The hard drives all have been 10x (or) more
> > dependable.
> >
> > And all it takes is getting rid of the idea that one must do a full
> > backup on Friday nights. Yes, amanda can do that, but do it as a
> > separate configuration else you will drive the poor girl out of her
> > mind when she finds out all her carefully worked out plans have all
> > gone aglay.
> >
> > And don't forget that in ones long term business plans, the
> > technology changes with time and there will come a time when you
> > will have nothing in the house that can read todays 1Tb sata hard
> > drive.  So having a storage location to save the old tech that can
> > read those drives should be part of that long term plan also.  And
> > be damned hard headed about it.
>
> Thanks Gene
>
> A dozen years ago, we found a couple posts on incremental backups
> using rsync and adapted it to provide 6 months of rolling incremental
> backups. We've been running this setup ever since - automated nightly,
> incremental backups. I posted our notes on the installation earlier in
> this thread.
>
> All important stuff is kept on the servers but it would be good to
> backup the laptops/workstations too and maybe Amanda is the answer :-)
> We don't install GUIs on our servers; can this be managed from
> individual workstations?
>
> Regards
>
> Clive

The closest you'll get to a gui is using your fav text editor to 
configure it. If building your own, the example directory has copies of 
all that, very liberally commented.  That may also be in the server deb, 
but if I looked, I've forgotten what I found. Heck most of the time I 
can't remember if I had breakfast this morning...  Synaptic, once the 
amanda server has been installed, will be more than happy to show you 
what is available, and where it was put.  There is also a mailing list, 
at amanda.org with very knowledgable people to explain the fine details. 

As for the other machines here that I backup every night, I only worry 
about the directories that contain linuxcnc or rtapi stuff, all the rest 
of it is no farther away than the install dvd from 
www.linuxcnc.org/downloads if I can't find the written dvd.  The amount 
of data from each of those machines is maybe 25 megabytes each worst 
case per nightly run, 

Re: Recommendation: Backup system

[Dan Purgert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

mo wrote:
> Maybe this is a little OT, but what kind of backup strategy would you 
> guys recommend? (Any advice Gene? :) )

If it *must* survive, 3-2-1 is the way to go.

3 copies (Original, Backup, and Backup of the Backup)
2 different media types (such as HDD(Orig) + HDD(Bk1) + Optical(Bk2))
1 stored offsite

That being said, most of my stuff isn't of the "absolutely must survive"
nature, so I just have it on external HDDs.  If something is
particularly important, it's on HDDs and also somewhere like google
drive / dropbox (although, encrypted when at those places).

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJX8FjGAAoJEI4R3fMSeaKBX9gH/i6qw2GOY/LSE5haz8uv6u/X
GNT6m44c6olrnzGtKFzEPae3ELTTAT/t47yet2noAR+AS9aJzhtR86l3JfQfu4ug
0XplxY/GkkrEpIeMFrqKy2hc3u++PoEHEBkQut46x6QAw/85ieKs2tsbmfeyUuF7
y5gFbGlowcy3NtSdGIMR6qzn67I+DaO2veTMV3Z+/aR+fAegZtvv/t3CCi1dsmRW
kEWIuPyGR//Foy/vkQsakpHJnl9AvJREB5/T+zRX3pkBXKkzQrIfe80eJYAm5Z8w
rhvZc9lLbO/bVcSvLwRKqYtDmWRY4AeVrqmCitiuXf54KjwZY5Lh16LVq+KfCiQ=
=YzZ4
-END PGP SIGNATURE-

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Re: Recommendation: Backup system

[Bob Weber]

Like I said backuppc uses incremental and full backups.  The web interface lets
you browse any backup (inc or full) and you see all the files backed up.  I set
the incremental for each day up to a week.  So I have up to 7 of them.  The full
can kept for for however long you want.  I currently keep 12 weekly, 8 bi-weekly
and 4 monthly full backups so that covers almost a year.

There is another solution you might like called rsnapshot.  I use it to backup
just my root directory on my desktop before I do updates.  That way if something
goes wrong I can boot into a rescue cd and restore the system to the state
before the update.  I just can't afford to have my desktop to break.  rsnapshot
uses rsync so it can backup any computer that has rsync.  It uses hard links so
duplicate files are only stored once.  You specify how many backups you want to
keep and rsnapshot deletes older ones over that max before adding the new one. 
That way you always have backups (assuming you set the count greater that 1)
that will be there even if there is a transfer error.  This is similar to your
script but is very versatile.


*...Bob*
On 10/01/2016 04:42 PM, mo wrote:
> Maybe this is a little OT, but what kind of backup strategy would you guys
> recommend? (Any advice Gene? :) )
>
>

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 21:37 schrieb Glenn English:



On Oct 1, 2016, at 10:22 AM, Gene Heskett  wrote:

On Saturday 01 October 2016 08:40:35 Mark Fletcher wrote:


I know Gene is a fan of Amanda, I have it on my list to try it out
myself based on positive remarks he has made about it in the past.


Yeah. Amanda's a good solution. I use it with tape.

There are a couple advantages to this:

Amanda's been around for a very long time (in computer years). So, like Debian 
stable, the bugs tend to have been worked out.

Amanda does its backups using plain old standard *nix software, dump or tar. So 
if you have a really nasty data loss, you can restore from a backup without 
dealing with data in a format built by the backup program. It's not a trivial 
job, I'm told, but if you're really stuck...

The tapes aren't disks, and every backup is done to a different and simple 
storage device with few moving parts. (The reliability of a disk RAID1 reduces 
this advantage significantly.)

OK, there are disadvantages too: Amanda's not easy to configure, and tapes and 
tape drives are very expensive and very slow.

But I like the advantages more than the disadvantages. It does a really good 
and reliable job.



Amanda is beginning to be more and more interesting to me :)

Re: Recommendation: Backup system

[mo]

Maybe this is a little OT, but what kind of backup strategy would you 
guys recommend? (Any advice Gene? :) )

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 14:20 schrieb Dan Purgert:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

mo wrote:

As the title say i'm in search for a backup application/system.
Currently i manage my backups with a little script that i wrote... but
it does not really serve my needs anymore.
I want to be able to make backups on my main PC and also on my server,
the backups i would then store on my NAS.


There's always rsync from the hosts to the NAS box.


rsync is quite handy for that, indeed ;)
(I actually do that atm - i transfer the data over via rsync to my NAS)


Only thing that I see "wrong" with your script is that:
 - you're deleting the backups (what happens when you delete something,
   and don't catch it til after the next backup run?)


I forgot to mention that i have a script which transfers the current 
backup over if the given backup is not yet saved, sorry, i kind left 
that out. :P



 - you're backing up everything, even if it hasn't changed.


That's indeed a problem atm... That's mainly why i'm on the search for a 
more professional approach so to speak. Honestly, i have not yet planned 
a specific strategy...





-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJX76m+AAoJEI4R3fMSeaKBwWQH/RiG+HH83WcyGT5wOhGM344j
UTOXN+VS7UC0jOsZG+MYrNJkAE8U9qwAYahi/63X+TFmSa2rFRmw6rv0pfvfBeiP
mItlVsjqyeR00tFc+zmAPWKnOZgela6kO6sdM7w2/GZTTThDL/CF/a+BP7YuOyid
QQEM5HLcSqbF1fiIStKkP2bTBZq0C8aSioy69Tg0nWES3vou7WxkXESUyP5ZEjp7
k9WQZj8JHZvJsJlfzRfJWSswkoR5Rj6OjsEjbbgFmoIHjndQ+ivhlKoJH+/6oGIM
txTwx8LzlucuKeYNr9bFQkV74ws8VGQBQPi1JBqvhXQEzW7qlbGg6ri3ADMrfq0=
=jx9W
-END PGP SIGNATURE-



Thanks Dan ;)

Greets

mo

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 18:22 schrieb Gene Heskett:

On Saturday 01 October 2016 08:40:35 Mark Fletcher wrote:


On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:

Hi Debian users :)

Information:
Distributor ID: Debian
Description:Debian GNU/Linux 8.6 (jessie)
Release:8.6
Codename:   jessie

As the title say i'm in search for a backup application/system.
Currently i manage my backups with a little script that i wrote...
but it does not really serve my needs anymore.
I want to be able to make backups on my main PC and also on my
server, the backups i would then store on my NAS.

Make a long story short:
Have you guys a recommendation for me?
Is there a specific application you use for your backups guys?


I know Gene is a fan of Amanda, I have it on my list to try it out
myself based on positive remarks he has made about it in the past.

Mark


Yeppers! It runs in the wee hours of the night here, for an hour or so.
Currently backing up this machine, and 3 more on my little home network
here, using its own unique, distribute the nightly load to equalize as
much a it can given its list of what to back up with nightly backups
totaling 11 to 14 Gb per night.

When I first started using it, I had a DDS2 changer, but it wasn't very
dependable, and at only 4Gb per tape, limited what I could do.  About
the time 500Gb Hd's came out, amanda had worked out a way to use virtual
tapes on a hard drive, so I converted. That has the advantage that
should I need to recover something, its random access instead of
sequential, so I can get back anything I need in just a few minutes,
most of which is spent studying the recovery docs because I've forgotten
how to do it. ;-)

With tapes, I could easily be a half a day recovering the same file
because each level of a backup has to be read from the start of the tape
until its found.  If I need something whose last good backup was a level
3, it would have to back up and find the level0, which is the last full
backup, then find the level 1 and merge any changes, wash, rinse, and
repeat.  A hard drive based system can do all that in seconds.


I guess i'm too young for tapes... or let's put it that way: I've never 
used them. (Why would you need tapes at home anyway :D )



And HD's have become much more dependable than tape, along with the
methods of warning the user that the drive is failing, and that alone
beats tape all the way into the trash bin.


True, the oldest HDD here (which is a 500GB drive) is running since 7 
years straight without a problem.



I was rather worried about the drive I use for amanda's v-tapes when I
saw almost 3 years ago that smartctl said it had 25 Reallocated sectors.
It still says 25 10 seconds ago.  That drive, now a 1Tb drive
as /dev/sdc, now has 58357 Power up hours on it. I don't care what you
may have paid for a tape library, it cannot survive that long, when this
HD has done that for a $100 bill at the time I bought it.  And I can put
this HD in a shirt pocket.  The tape library would need a refrigerator
rated 2 wheel dolly to move it, and a similar second dolly to move its
tapes.

Whats not to like?

I've had far more trouble dealing with tar changes as its been updated
over the years than I've had with amanda itself. All have been fixable
in a day or two once you can post the breakage on the tar list. amanda
uses tar to do the bare metal work.  A wrapper for tar in that sense,
and I then wrap amanda with a bash script that fixes the always a day
late record keeping that you can add to the v-tape image by making a
copy of amanda's database and writing it to the V-tape amanda just used,
so I can lose the main drive and have to start with a new install on a
fresh drive. With amanda I would install from the repo's on the new
drive. Its 2 or 3 steps, but in an hours time I can have this working
wheezy system with all its dross, put back on a new drive.

Cheers, Gene Heskett



Thanks for the info Gene ;)

Re: Recommendation: Backup system

[Glenn English]

> On Oct 1, 2016, at 11:54 AM, Clive Menzies  wrote:
> 
> We don't install GUIs on our servers; can this be managed from individual 
> workstations?

I'm not sure, but I think Amanda was written before GUIs existed :-)

-- 
Glenn English

Re: Recommendation: Backup system

[Glenn English]

> On Oct 1, 2016, at 10:22 AM, Gene Heskett  wrote:
> 
> On Saturday 01 October 2016 08:40:35 Mark Fletcher wrote:
> 
>> I know Gene is a fan of Amanda, I have it on my list to try it out
>> myself based on positive remarks he has made about it in the past.

Yeah. Amanda's a good solution. I use it with tape.

There are a couple advantages to this:

Amanda's been around for a very long time (in computer years). So, like Debian 
stable, the bugs tend to have been worked out.

Amanda does its backups using plain old standard *nix software, dump or tar. So 
if you have a really nasty data loss, you can restore from a backup without 
dealing with data in a format built by the backup program. It's not a trivial 
job, I'm told, but if you're really stuck...

The tapes aren't disks, and every backup is done to a different and simple 
storage device with few moving parts. (The reliability of a disk RAID1 reduces 
this advantage significantly.)

OK, there are disadvantages too: Amanda's not easy to configure, and tapes and 
tape drives are very expensive and very slow. 

But I like the advantages more than the disadvantages. It does a really good 
and reliable job.

-- 
Glenn English

Re: Recommendation: Backup system

[Clive Menzies]

On 01/10/16 18:40, Gene Heskett wrote:

On Saturday 01 October 2016 12:39:58 Clive Menzies wrote:


Quick question. Are your backups incremental or complete every night?

This is probably better explained in the manpages. Amanda has the concept
of doing a full backup of everything in its disklist according to the
days you give it. Amanda will then shuffle the schedule such that those
full backups are done at random in that cycle, with an eye toward
equalizing, as much as possible, the amount of data saved during each
run. It does this by advanceing the the level 0's as it will not let a
given file go beyond that cycle before a full copy is made again.
Level 0 is a full copy of a file, level 1 is whats been changed in that
file since the last level 0. Level 2 is whats been changed since level
1, ad infinitum but most useage never gets past level 2.  I have the
drive I use for amanda setup as 30 virtual tapes, using one a day, then
recycle. With 4 machines feeding amanda, that 1Tb drive stays at around
46% used:

gene@coyote:~$ df /amandatapes
Filesystem 1K-blocks  Used Available Use% Mounted on
/dev/sdc2  960798056 410555684 501429932  46% /amandatapes

I stayed on the devel branch of amanda, playing the part of the canary in
the coal mine for years while it was being heavily developed, but
apparently that grant ran out so not a lot has changed in around 5
years. So while I have a self made, slightly newer version on this
machine as master, v3.3.7p1, the slaves are all running 3.3.1 clients
from the wheezy repo.  And it Just Works(TM).

I would be the first to point out that my way is NOT for archival storage
due to this 30 day and recycle setup. I could extend it to 60 days on
this drive I suppose, but this is not a business.

For a business, I would include the price of the drive as a CODB, fill it
up and put it on the shelf at a remote location so it doesn't all go up
in smoke when the place burns, thereby giving me the ability to recover
something 5+ years old, or for however long one has had that setup
running.

That $100 or less a month for a new commodity drive is far less of a CODB
than the archival storage of tapes would be over a 10 year period. And
you would have to add in that the tape drive(s) would be out of service
for about a month each annually while they spend the holidays in
Oklahoma City getting fresh heads and rubber at about a kilobuck each
for the rebuild.  Thats been the track record here in my usage of tapes.
The hard drives all have been 10x (or) more dependable.

And all it takes is getting rid of the idea that one must do a full
backup on Friday nights. Yes, amanda can do that, but do it as a
separate configuration else you will drive the poor girl out of her mind
when she finds out all her carefully worked out plans have all gone
aglay.

And don't forget that in ones long term business plans, the technology
changes with time and there will come a time when you will have nothing
in the house that can read todays 1Tb sata hard drive.  So having a
storage location to save the old tech that can read those drives should
be part of that long term plan also.  And be damned hard headed about
it.


Thanks Gene

A dozen years ago, we found a couple posts on incremental backups using 
rsync and adapted it to provide 6 months of rolling incremental backups. 
We've been running this setup ever since - automated nightly, 
incremental backups. I posted our notes on the installation earlier in 
this thread.


All important stuff is kept on the servers but it would be good to 
backup the laptops/workstations too and maybe Amanda is the answer :-) 
We don't install GUIs on our servers; can this be managed from 
individual workstations?


Regards

Clive

--
Clive Menzies
http://freecriticalthinking.org

Re: Recommendation: Backup system

[Gene Heskett]

On Saturday 01 October 2016 12:39:58 Clive Menzies wrote:

> On 01/10/16 17:22, Gene Heskett wrote
>
> > Yeppers! It runs in the wee hours of the night here, for an hour or
> > so. Currently backing up this machine, and 3 more on my little home
> > network here, using its own unique, distribute the nightly load to
> > equalize as much a it can given its list of what to back up with
> > nightly backups totaling 11 to 14 Gb per night.
>
> Hi Gene
>
> Quick question. Are your backups incremental or complete every night?
>
> Regards
>
> Clive

This is probably better explained in the manpages. Amanda has the concept 
of doing a full backup of everything in its disklist according to the 
days you give it. Amanda will then shuffle the schedule such that those 
full backups are done at random in that cycle, with an eye toward 
equalizing, as much as possible, the amount of data saved during each 
run. It does this by advanceing the the level 0's as it will not let a 
given file go beyond that cycle before a full copy is made again.
Level 0 is a full copy of a file, level 1 is whats been changed in that 
file since the last level 0. Level 2 is whats been changed since level 
1, ad infinitum but most useage never gets past level 2.  I have the 
drive I use for amanda setup as 30 virtual tapes, using one a day, then 
recycle. With 4 machines feeding amanda, that 1Tb drive stays at around 
46% used:

gene@coyote:~$ df /amandatapes
Filesystem 1K-blocks  Used Available Use% Mounted on
/dev/sdc2  960798056 410555684 501429932  46% /amandatapes

I stayed on the devel branch of amanda, playing the part of the canary in 
the coal mine for years while it was being heavily developed, but 
apparently that grant ran out so not a lot has changed in around 5 
years. So while I have a self made, slightly newer version on this 
machine as master, v3.3.7p1, the slaves are all running 3.3.1 clients 
from the wheezy repo.  And it Just Works(TM).

I would be the first to point out that my way is NOT for archival storage 
due to this 30 day and recycle setup. I could extend it to 60 days on 
this drive I suppose, but this is not a business.

For a business, I would include the price of the drive as a CODB, fill it 
up and put it on the shelf at a remote location so it doesn't all go up 
in smoke when the place burns, thereby giving me the ability to recover 
something 5+ years old, or for however long one has had that setup 
running.

That $100 or less a month for a new commodity drive is far less of a CODB 
than the archival storage of tapes would be over a 10 year period. And 
you would have to add in that the tape drive(s) would be out of service 
for about a month each annually while they spend the holidays in 
Oklahoma City getting fresh heads and rubber at about a kilobuck each 
for the rebuild.  Thats been the track record here in my usage of tapes.  
The hard drives all have been 10x (or) more dependable.

And all it takes is getting rid of the idea that one must do a full 
backup on Friday nights. Yes, amanda can do that, but do it as a 
separate configuration else you will drive the poor girl out of her mind 
when she finds out all her carefully worked out plans have all gone 
aglay.

And don't forget that in ones long term business plans, the technology 
changes with time and there will come a time when you will have nothing 
in the house that can read todays 1Tb sata hard drive.  So having a 
storage location to save the old tech that can read those drives should 
be part of that long term plan also.  And be damned hard headed about 
it.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: Recommendation: Backup system

[Dan Purgert]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

mo wrote:
> As the title say i'm in search for a backup application/system.
> Currently i manage my backups with a little script that i wrote... but 
> it does not really serve my needs anymore.
> I want to be able to make backups on my main PC and also on my server, 
> the backups i would then store on my NAS.

There's always rsync from the hosts to the NAS box.

Only thing that I see "wrong" with your script is that:
 - you're deleting the backups (what happens when you delete something,
   and don't catch it til after the next backup run?)
 - you're backing up everything, even if it hasn't changed.



-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJX76m+AAoJEI4R3fMSeaKBwWQH/RiG+HH83WcyGT5wOhGM344j
UTOXN+VS7UC0jOsZG+MYrNJkAE8U9qwAYahi/63X+TFmSa2rFRmw6rv0pfvfBeiP
mItlVsjqyeR00tFc+zmAPWKnOZgela6kO6sdM7w2/GZTTThDL/CF/a+BP7YuOyid
QQEM5HLcSqbF1fiIStKkP2bTBZq0C8aSioy69Tg0nWES3vou7WxkXESUyP5ZEjp7
k9WQZj8JHZvJsJlfzRfJWSswkoR5Rj6OjsEjbbgFmoIHjndQ+ivhlKoJH+/6oGIM
txTwx8LzlucuKeYNr9bFQkV74ws8VGQBQPi1JBqvhXQEzW7qlbGg6ri3ADMrfq0=
=jx9W
-END PGP SIGNATURE-

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: 05CA 9A50 3F2E 1335 4DC5  4AEE 8E11 DDF3 1279 A281

Re: Recommendation: Backup system

[Clive Menzies]

On 01/10/16 17:22, Gene Heskett wrote

Yeppers! It runs in the wee hours of the night here, for an hour or so.
Currently backing up this machine, and 3 more on my little home network
here, using its own unique, distribute the nightly load to equalize as
much a it can given its list of what to back up with nightly backups
totaling 11 to 14 Gb per night.


Hi Gene

Quick question. Are your backups incremental or complete every night?

Regards

Clive

--
Clive Menzies
http://freecriticalthinking.org

Re: Recommendation: Backup system

[Gene Heskett]

On Saturday 01 October 2016 08:40:35 Mark Fletcher wrote:

> On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> > Hi Debian users :)
> >
> > Information:
> > Distributor ID: Debian
> > Description:Debian GNU/Linux 8.6 (jessie)
> > Release:8.6
> > Codename:   jessie
> >
> > As the title say i'm in search for a backup application/system.
> > Currently i manage my backups with a little script that i wrote...
> > but it does not really serve my needs anymore.
> > I want to be able to make backups on my main PC and also on my
> > server, the backups i would then store on my NAS.
> >
> > Make a long story short:
> > Have you guys a recommendation for me?
> > Is there a specific application you use for your backups guys?
>
> I know Gene is a fan of Amanda, I have it on my list to try it out
> myself based on positive remarks he has made about it in the past.
>
> Mark

Yeppers! It runs in the wee hours of the night here, for an hour or so. 
Currently backing up this machine, and 3 more on my little home network 
here, using its own unique, distribute the nightly load to equalize as 
much a it can given its list of what to back up with nightly backups 
totaling 11 to 14 Gb per night.

When I first started using it, I had a DDS2 changer, but it wasn't very 
dependable, and at only 4Gb per tape, limited what I could do.  About 
the time 500Gb Hd's came out, amanda had worked out a way to use virtual 
tapes on a hard drive, so I converted. That has the advantage that 
should I need to recover something, its random access instead of 
sequential, so I can get back anything I need in just a few minutes, 
most of which is spent studying the recovery docs because I've forgotten 
how to do it. ;-)

With tapes, I could easily be a half a day recovering the same file 
because each level of a backup has to be read from the start of the tape 
until its found.  If I need something whose last good backup was a level 
3, it would have to back up and find the level0, which is the last full 
backup, then find the level 1 and merge any changes, wash, rinse, and 
repeat.  A hard drive based system can do all that in seconds.

And HD's have become much more dependable than tape, along with the 
methods of warning the user that the drive is failing, and that alone 
beats tape all the way into the trash bin.

I was rather worried about the drive I use for amanda's v-tapes when I 
saw almost 3 years ago that smartctl said it had 25 Reallocated sectors.  
It still says 25 10 seconds ago.  That drive, now a 1Tb drive 
as /dev/sdc, now has 58357 Power up hours on it. I don't care what you 
may have paid for a tape library, it cannot survive that long, when this 
HD has done that for a $100 bill at the time I bought it.  And I can put 
this HD in a shirt pocket.  The tape library would need a refrigerator 
rated 2 wheel dolly to move it, and a similar second dolly to move its 
tapes.

Whats not to like?

I've had far more trouble dealing with tar changes as its been updated 
over the years than I've had with amanda itself. All have been fixable 
in a day or two once you can post the breakage on the tar list. amanda 
uses tar to do the bare metal work.  A wrapper for tar in that sense, 
and I then wrap amanda with a bash script that fixes the always a day 
late record keeping that you can add to the v-tape image by making a 
copy of amanda's database and writing it to the V-tape amanda just used, 
so I can lose the main drive and have to start with a new install on a 
fresh drive. With amanda I would install from the repo's on the new 
drive. Its 2 or 3 steps, but in an hours time I can have this working 
wheezy system with all its dross, put back on a new drive.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: Recommendation: Backup system

[Mark Fletcher]

On Sat, Oct 01, 2016 at 10:46:07AM -0400, Bob Weber wrote:
> I use backuppc.  It is web browser based setup and usage.  It takes 
> incremental
> and full backups that can remain as long as you want or have space for.  It 
> can
> browse files by name or in a version mode where you can see the date where a
> file changed and restore an earlier version if you want (or to a separate 
> download directory).  It compresses files for space and only keeps one copy 
> of a
> file's data if it is located in different directories or servers (using hard
> links as needed).  It can even backup user data for windows users (samba).  I
> use the rsync transfer for Linux machines and even with windows running 
> Cygwin.
> 
> I currently backup 8 computers going back almost 1 year.  I even backup a vm 
> at
> digital ocean.  Backuppc reports this:
> 
> 144 full backups of total size 8951.56GB (prior to pooling and compression),
> 57 incr backups of total size 57.13GB (prior to pooling and compression).
> Pool is 358.94GB comprising 1903010 files and 4369 directories (as of 10/1 
> 01:09).
> 
> So 8951GB is compressed or pooled into just 358 GB! 

Wow, that is impressive!

Mark

Re: Recommendation: Backup system

[mo]

Has someone experience with Bacula?
I heard good things about it, although i never looked into it... maybe 
someone has and can give me his report on it :)

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 16:46 schrieb Bob Weber:

I use backuppc.  It is web browser based setup and usage.  It takes
incremental and full backups that can remain as long as you want or have
space for.  It can browse files by name or in a version mode where you
can see the date where a file changed and restore an earlier version if
you want (or to a separate  download directory).  It compresses files
for space and only keeps one copy of a file's data if it is located in
different directories or servers (using hard links as needed).  It can
even backup user data for windows users (samba).  I use the rsync
transfer for Linux machines and even with windows running Cygwin.


I like the web interface part i have to say (Just browsed their website :) )
I will look into it ;)
(I have currently no windows machines here, but that might change in the 
future since i plan to deplay Windows Server 2012 R2)


Thanks for the hint ;)


I currently backup 8 computers going back almost 1 year.  I even backup
a vm at digital ocean.  Backuppc reports this:

144 full backups of total size 8951.56GB (prior to pooling and compression),
57 incr backups of total size 57.13GB (prior to pooling and compression).
Pool is 358.94GB comprising 1903010 files and 4369 directories (as of
10/1 01:09).

So 8951GB is compressed or pooled into just 358 GB!


The compression is quite nice, i might have at least 100GB to backup 
here. (Maybe a little more depending on what i want to save)



*...Bob*


Greets

mo

Re: Recommendation: Backup system

[Bob Weber]

I use backuppc.  It is web browser based setup and usage.  It takes incremental
and full backups that can remain as long as you want or have space for.  It can
browse files by name or in a version mode where you can see the date where a
file changed and restore an earlier version if you want (or to a separate 
download directory).  It compresses files for space and only keeps one copy of a
file's data if it is located in different directories or servers (using hard
links as needed).  It can even backup user data for windows users (samba).  I
use the rsync transfer for Linux machines and even with windows running Cygwin.

I currently backup 8 computers going back almost 1 year.  I even backup a vm at
digital ocean.  Backuppc reports this:

144 full backups of total size 8951.56GB (prior to pooling and compression),
57 incr backups of total size 57.13GB (prior to pooling and compression).
Pool is 358.94GB comprising 1903010 files and 4369 directories (as of 10/1 
01:09).

So 8951GB is compressed or pooled into just 358 GB! 

*...Bob*
On 10/01/2016 05:37 AM, mo wrote:
> Hi Debian users :)
>
> Information:
> Distributor ID:Debian
> Description:Debian GNU/Linux 8.6 (jessie)
> Release:8.6
> Codename:jessie
>
> As the title say i'm in search for a backup application/system.
> Currently i manage my backups with a little script that i wrote... but it does
> not really serve my needs anymore.
> I want to be able to make backups on my main PC and also on my server, the
> backups i would then store on my NAS.
>
> Make a long story short:
> Have you guys a recommendation for me?
> Is there a specific application you use for your backups guys?
>
> Btw: I dont mind configuring or playing around with new applications, every
> recommendation is welcome ;)
>
>
> Here is my current backup script (Which is run by cron daily):
> #!/bin/bash
>
> TO_BACKUP="/home /etc /var/log"
> BACKUP_DIR="/var/backup"
> BACKUP_ARCHIVE="backup-`date +%d_%m_%Y-%H:%M`.tar"
> TAR_OPTIONS='-cpf'
>
> delete_old_backup() {
> if [ -f ${BACKUP_DIR}/backup*.tar ]; then
> rm -rf $BACKUP_DIR/backup*
> fi
> }
>
> create_new_backup() {
> tar $TAR_OPTIONS ${BACKUP_DIR}/$BACKUP_ARCHIVE $TO_BACKUP
> }
>
> main() {
> delete_old_backup
> create_new_backup
> }
>
> main
>
> Greets
> mo
>
>

Re: Recommendation: Backup system

[mo]

Am 01.10.2016 um 14:40 schrieb Mark Fletcher:

On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:

Hi Debian users :)

Information:
Distributor ID: Debian
Description:Debian GNU/Linux 8.6 (jessie)
Release:8.6
Codename:   jessie

As the title say i'm in search for a backup application/system.
Currently i manage my backups with a little script that i wrote... but it
does not really serve my needs anymore.
I want to be able to make backups on my main PC and also on my server, the
backups i would then store on my NAS.

Make a long story short:
Have you guys a recommendation for me?
Is there a specific application you use for your backups guys?


I know Gene is a fan of Amanda, I have it on my list to try it out
myself based on positive remarks he has made about it in the past.


Sounds good to me :) - I'll set that on the todo list myself to (Just 
checked the webpage of Amanda and it does sound pretty good!).


Gene, can you elaborate on Amanda? :) (If Gene does read this mail)


Mark



Greets

mo

Re: Recommendation: Backup system

[Mark Fletcher]

On Sat, Oct 01, 2016 at 11:37:31AM +0200, mo wrote:
> Hi Debian users :)
> 
> Information:
> Distributor ID:   Debian
> Description:  Debian GNU/Linux 8.6 (jessie)
> Release:  8.6
> Codename: jessie
> 
> As the title say i'm in search for a backup application/system.
> Currently i manage my backups with a little script that i wrote... but it
> does not really serve my needs anymore.
> I want to be able to make backups on my main PC and also on my server, the
> backups i would then store on my NAS.
> 
> Make a long story short:
> Have you guys a recommendation for me?
> Is there a specific application you use for your backups guys?

I know Gene is a fan of Amanda, I have it on my list to try it out 
myself based on positive remarks he has made about it in the past.

Mark

Recommendation: Backup system

[mo]

Hi Debian users :)

Information:
Distributor ID: Debian
Description:Debian GNU/Linux 8.6 (jessie)
Release:8.6
Codename:   jessie

As the title say i'm in search for a backup application/system.
Currently i manage my backups with a little script that i wrote... but 
it does not really serve my needs anymore.
I want to be able to make backups on my main PC and also on my server, 
the backups i would then store on my NAS.


Make a long story short:
Have you guys a recommendation for me?
Is there a specific application you use for your backups guys?

Btw: I dont mind configuring or playing around with new applications, 
every recommendation is welcome ;)



Here is my current backup script (Which is run by cron daily):
#!/bin/bash

TO_BACKUP="/home /etc /var/log"
BACKUP_DIR="/var/backup"
BACKUP_ARCHIVE="backup-`date +%d_%m_%Y-%H:%M`.tar"
TAR_OPTIONS='-cpf'

delete_old_backup() {
if [ -f ${BACKUP_DIR}/backup*.tar ]; then
rm -rf $BACKUP_DIR/backup*
fi
}

create_new_backup() {
tar $TAR_OPTIONS ${BACKUP_DIR}/$BACKUP_ARCHIVE $TO_BACKUP
}

main() {
delete_old_backup
create_new_backup
}

main

Greets
mo