Re: slightly OT - backup strategy

[Efraim Flashner]

Coming in late:

I keep a bunch of that stuff in git-annex. I can then use git-annex to
sync the data around to my different machines, and I use a couple of S3
compatible places online to keep an offsite location for the files
synced with git-annex.

One collection for my Music files, one for family photos. Documents and
Download folders are synced using syncthing.

-- 
Efraim Flashner  אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: slightly OT - backup strategy

[Mark E. Fuller]

20 Jul 2022 13:15:13 Rabin Yasharzadehe :

> Using ZFS with sanoid[https://github.com/jimsalterjrs/sanoid]
> 
> ZFS will give you all the benefits of COW filesystem, compression, and 
> snapshots (and much more),
> combined with sanoid utility will allow you to automate the snapshots and 
> send them to a remote system,
> and because ZFS is block-level aware of the changes between snapshots, 
> send is much more efficient,
> because unlike Rsync which needs to stats and compare each file to determine 
> what to sync,
> ZFS only need to compile a list of block which have changed between 2 
> snapshots and send only them.
> which also works if the volume is encrypted, which allows you to have a 
> remote system, which is encrypted on rest,
> and keep pushing/sending snapshots to it without having to unlock it.
> 
> 
> --
> Rabin
> 
> 
> On Sun, 17 Jul 2022 at 16:50, Shlomo Solomon  wrote:
>> I recently lost some files because of a bad disk - hardware problem.
>> 
>> I do regular backups so I was not really worried, but I now see that I
>> have a problem with my backup strategy so I'd like to know how others
>> handle/prevent what happened to me.
>> 
>> I backup files using rsync and I basically have 2 types of backups.
>> 
>> My most important files are backed up every night. I do incremental
>> backups using:    rsync -aqrlvtogS --ignore-errors  --backup
>> I keep about 4 months of backups. So if a file is damaged,
>> missing or accidentally deleted, I can find a good file - even if, for
>> example I screwed up the file and only discovered the problem a few
>> days later.
>> 
>> BUT, all the rest of my files - music, videos, pictures, etc are backed
>> up daily and weekly on 2 different physical drives using:
>> rsync -qrlvtogS --delete --ignore-errors
>> I use --delete to prevent accumulating garbage on my backup disks.
>> 
>> So here's the problem: Because of a hardware problem, several files on
>> one of my disks were lost. As a result, the daily backup script
>> "thought" that those files should be deleted from the daily backup.
>> Unfortunately, I did not notice the problem. A few days later, those
>> same files were also deleted from the weekly backup. So they are lost.
>> 
>> So on one hand, I need --delete to avoid keeping backups of old
>> garbage, but on the other hand, the --delete option does not know if I
>> deleted the file or if it's gone because of a hardware problem.
>> 
>> 
>> 
>> -- 
>> Shlomo Solomon
>> http://the-solomons.net
>> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>> 
>> ___
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Coming to this late, I use "backintime" which is a Python wrapper and GUI for 
rsync.
One of the settings, "smart remove" is set to remove old snapshots, but keep 
some for exactly the kind of problem described here, e.g.
2/day for the last week, then 1/week for a month, then 1/month for the previous 
year, etc. (you set all these yourself)

I made a similar deletion error once which propagated through my Dropbox that I 
only noticed after I was past the 30 day backup there, but I easily pulled the 
files off my backup HDD from an old image
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: slightly OT - backup strategy

[Rabin Yasharzadehe]

Using ZFS with sanoid 

ZFS will give you all the benefits of COW filesystem, compression, and
snapshots (and much more),
combined with sanoid utility will allow you to automate the snapshots and
send them to a remote system,
and because ZFS is block-level aware of the changes between snapshots,
send is much more efficient,
because unlike Rsync which needs to stats and compare each file to
determine what to sync,
ZFS only need to compile a list of block which have changed between 2
snapshots and send only them.
which also works if the volume is encrypted, which allows you to have a
remote system, which is encrypted on rest,
and keep pushing/sending snapshots to it without having to unlock it.


--
Rabin


On Sun, 17 Jul 2022 at 16:50, Shlomo Solomon 
wrote:

> I recently lost some files because of a bad disk - hardware problem.
>
> I do regular backups so I was not really worried, but I now see that I
> have a problem with my backup strategy so I'd like to know how others
> handle/prevent what happened to me.
>
> I backup files using rsync and I basically have 2 types of backups.
>
> My most important files are backed up every night. I do incremental
> backups using:rsync -aqrlvtogS --ignore-errors  --backup
> I keep about 4 months of backups. So if a file is damaged,
> missing or accidentally deleted, I can find a good file - even if, for
> example I screwed up the file and only discovered the problem a few
> days later.
>
> BUT, all the rest of my files - music, videos, pictures, etc are backed
> up daily and weekly on 2 different physical drives using:
> rsync -qrlvtogS --delete --ignore-errors
> I use --delete to prevent accumulating garbage on my backup disks.
>
> So here's the problem: Because of a hardware problem, several files on
> one of my disks were lost. As a result, the daily backup script
> "thought" that those files should be deleted from the daily backup.
> Unfortunately, I did not notice the problem. A few days later, those
> same files were also deleted from the weekly backup. So they are lost.
>
> So on one hand, I need --delete to avoid keeping backups of old
> garbage, but on the other hand, the --delete option does not know if I
> deleted the file or if it's gone because of a hardware problem.
>
>
>
> --
> Shlomo Solomon
> http://the-solomons.net
> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: slightly OT - backup strategy

[Dan Yasny]

On Sun, Jul 17, 2022 at 10:15 AM Dotan Shavit  wrote:

> Consider raid + zfs + daily snapshoots
>
>
RAID/ZFS/replication are NOT backups. This is a common problem that needs
explaining every time :)

https://www.raidisnotabackup.com/



> בתאריך יום א׳, 17 ביולי 2022, 16:50, מאת Shlomo Solomon ‏<
> shlomo.solo...@gmail.com>:
>
>> I recently lost some files because of a bad disk - hardware problem.
>>
>> I do regular backups so I was not really worried, but I now see that I
>> have a problem with my backup strategy so I'd like to know how others
>> handle/prevent what happened to me.
>>
>> I backup files using rsync and I basically have 2 types of backups.
>>
>> My most important files are backed up every night. I do incremental
>> backups using:rsync -aqrlvtogS --ignore-errors  --backup
>> I keep about 4 months of backups. So if a file is damaged,
>> missing or accidentally deleted, I can find a good file - even if, for
>> example I screwed up the file and only discovered the problem a few
>> days later.
>>
>> BUT, all the rest of my files - music, videos, pictures, etc are backed
>> up daily and weekly on 2 different physical drives using:
>> rsync -qrlvtogS --delete --ignore-errors
>> I use --delete to prevent accumulating garbage on my backup disks.
>>
>> So here's the problem: Because of a hardware problem, several files on
>> one of my disks were lost. As a result, the daily backup script
>> "thought" that those files should be deleted from the daily backup.
>> Unfortunately, I did not notice the problem. A few days later, those
>> same files were also deleted from the weekly backup. So they are lost.
>>
>> So on one hand, I need --delete to avoid keeping backups of old
>> garbage, but on the other hand, the --delete option does not know if I
>> deleted the file or if it's gone because of a hardware problem.
>>
>>
>>
>> --
>> Shlomo Solomon
>> http://the-solomons.net
>> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>>
>> ___
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: slightly OT - backup strategy

[Dan Yasny]

Inline

On Sun, Jul 17, 2022 at 9:50 AM Shlomo Solomon 
wrote:

> I recently lost some files because of a bad disk - hardware problem.
>
> I do regular backups so I was not really worried, but I now see that I
> have a problem with my backup strategy so I'd like to know how others
> handle/prevent what happened to me.
>
> I backup files using rsync and I basically have 2 types of backups.
>
> My most important files are backed up every night. I do incremental
> backups using:rsync -aqrlvtogS --ignore-errors  --backup
> I keep about 4 months of backups. So if a file is damaged,
> missing or accidentally deleted, I can find a good file - even if, for
> example I screwed up the file and only discovered the problem a few
> days later.
>
> BUT, all the rest of my files - music, videos, pictures, etc are backed
> up daily and weekly on 2 different physical drives using:
> rsync -qrlvtogS --delete --ignore-errors
> I use --delete to prevent accumulating garbage on my backup disks.
>
> So here's the problem: Because of a hardware problem, several files on
> one of my disks were lost. As a result, the daily backup script
> "thought" that those files should be deleted from the daily backup.
> Unfortunately, I did not notice the problem. A few days later, those
> same files were also deleted from the weekly backup. So they are lost.
>

I would consider an external backup that's cheap and not really limited in
terms of space - glacier or something similar. Drop a monthly archive there
without --delete and even if some garbage accumulates, it's not a big deal.

It all really depends on the amount of data in question, if it's not
insanely huge, you can simply buy some cheap drives and do the same thing.
Tiered backups aren't anything new, it is best practice to have a set of
fresh hot backups, like your dailies, a weekly or biweekly warm archive in
case you need to pull out something older, and a cold archive that's
essentially a big dump of ancient stuff on very cheap and very slow medium,
used for emergencies.


>
> So on one hand, I need --delete to avoid keeping backups of old
> garbage, but on the other hand, the --delete option does not know if I
> deleted the file or if it's gone because of a hardware problem.
>

For this, I would run a seperate infrequent backup without --delete, and
dump it on the cold storage, just in case.

Another option, if you want to save space, would be to drop rsync and
switch to something like backy2 or it's alternatives.


>
>
>
> --
> Shlomo Solomon
> http://the-solomons.net
> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: slightly OT - backup strategy

[Dotan Shavit]

Consider raid + zfs + daily snapshoots

בתאריך יום א׳, 17 ביולי 2022, 16:50, מאת Shlomo Solomon ‏<
shlomo.solo...@gmail.com>:

> I recently lost some files because of a bad disk - hardware problem.
>
> I do regular backups so I was not really worried, but I now see that I
> have a problem with my backup strategy so I'd like to know how others
> handle/prevent what happened to me.
>
> I backup files using rsync and I basically have 2 types of backups.
>
> My most important files are backed up every night. I do incremental
> backups using:rsync -aqrlvtogS --ignore-errors  --backup
> I keep about 4 months of backups. So if a file is damaged,
> missing or accidentally deleted, I can find a good file - even if, for
> example I screwed up the file and only discovered the problem a few
> days later.
>
> BUT, all the rest of my files - music, videos, pictures, etc are backed
> up daily and weekly on 2 different physical drives using:
> rsync -qrlvtogS --delete --ignore-errors
> I use --delete to prevent accumulating garbage on my backup disks.
>
> So here's the problem: Because of a hardware problem, several files on
> one of my disks were lost. As a result, the daily backup script
> "thought" that those files should be deleted from the daily backup.
> Unfortunately, I did not notice the problem. A few days later, those
> same files were also deleted from the weekly backup. So they are lost.
>
> So on one hand, I need --delete to avoid keeping backups of old
> garbage, but on the other hand, the --delete option does not know if I
> deleted the file or if it's gone because of a hardware problem.
>
>
>
> --
> Shlomo Solomon
> http://the-solomons.net
> Claws Mail 3.17.5 - KDE Plasma 5.18.5 - Kubuntu 20.04
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il