Re: Tarsnap GUI shows 0 data archived or backed up

[Garance AE Drosehn]

> On Sep 7, 2017, at 2:54 PM, Scott Robison <sc...@casaderobison.com> wrote:
> 
> On Thu, Sep 7, 2017 at 11:55 AM, Garance AE Drosehn wrote:
>> 
>> De-duplication works great in some other situations because the other 
>> service is de-duplicating files across (say) 20 different Windows machines.  
>> So the first machine may use up 10-gig of space, but all of the rest of the 
>> machines might have less than a gig of additional unique files per machine.  
>> Tarsnap cannot do that kind of optimization across multiple machines.
> 
> Depending on just how paranoid you are, and how disciplined you are,
> you can have multiple machines deduplicate to a common store, as long
> as you only have one running at a time.
> 
> What you wrote does not preclude that from happening, I just wanted to
> point it out in case others might be interested in something like
> that.

Hmm. Yes, that's an interesting point.  I can see how that could be setup and 
work okay.  I hadn't thought about doing things that way, but I see how it 
could work as long as it was setup with care.

But I'm not going to do it for my own machines!  :)

-- 
Garance Alistair Drosehn=  gadc...@earthlink.net
Senior Systems Programmer   or   g...@freebsd.org

Re: Question about mitigating large storage costs in case of ransomware

[Garance AE Drosehn]

Here's a strategy I use, although I wasn't thinking of ransomware at the time.

I have a script which first does a 'tarsnap --dry-run', and parses the summary 
output from that.  It checks the 'Total Size' and 'Compressed Size' of the new 
data, and will skip making a real-backup if that size seems unreasonable.  
Think about this for a second:  If you are hit by ransomware, then you probably 
don't want to upload *any* of that encrypted data to tarsnap's servers.

If the update-size seems reasonable, then I go ahead and create a real update.  
Note that I also break up my backup into multiple separate tarsnap archives.  
So, for instance, it might be reasonable for my "Documents" backup to have a 
lot of new data in it, but it would be extremely suspicious if the backup of 
all my svn repositories had a huge amount of new data.

What I have is even more convoluted than what I've described, and I'm pretty 
sure that my scripts wouldn't be useful for others.  But that's the direction I 
would consider if I were you.

-- 
  Garance Alistair Drosehn   =  gadc...@earthlink.net
  Senior Systems Programmer  or   g...@freebsd.org


> On Jul 12, 2017, at 9:36 AM, Rob Hoelz  wrote:
> 
> Hello fellow tarsnap users!
> 
> I just started using tarsnap, and I was wondering if there exists an
> option (or the potential interest in developing an option) to put a cap
> on an archive size.  The reason I ask is on the off chance another
> tarsnap user or I gets bitten by ransomware and it encrypts their /home
> directory, I would expect the archive size for an automated backup to
> really balloon in size (seeing as it's kind of hard to deduplicate and
> compress random data!), and it would be nice to avoid burning through
> one's account balance to add insult to injury.  Not that I intend on
> getting bitten by ransomware, but you never know!
> 
> The closest thing I could find is the --maxbw option; is there a
> corresponding option for archive size that I'm not seeing on the
> manpage?
> 
> Thanks,
> Rob

Re: a script to just back up files in a folder

[Garance AE Drosehn]

I assume you are writing a script, and that you want that script to back up 
only the files in your home directory (but no subdirectories, and no files 
inside those subdirectories).

One way to do that might be:

rm -f /tmp/home-files.txt
find ~ -maxdepth 1 -type f -print >> /tmp/home-files.txt
tarsnap --dry-run -v -cf "homefiles_$(date +%Y-%m%d)" \
--files-from /tmp/home-files.txt

Those three commands will show you what files tarsnap would save, and how much 
space would be used when backing up those files.  If you're happy with how that 
looks, you'd need to drop the '--dry-run' so that tarsnap would actually save 
the files.  And at some point you would probably want to drop the '-v' too, so 
that tarsnap won't print out the full list of the files it is saving each time 
you run the script.

-- garance alistair drosehn
-- senior systems programmer


> On Jun 8, 2016, at 7:53 PM, Sarah Alawami  wrote:
> 
> Hello to all. I set up my scripts for my home folder to back up only what I 
> want. Is there a way though to back up only the files at my root ~ directory? 
> Can I create a script that just backs up the files that should go in the root 
> or what do I need to do in this case? Not everything in my ~ directory is in 
> a folder so wanted to know what would be the best way of going about this.
> 
> Thanks all and be blessed.

Re: how to name an archive

[Garance AE Drosehn]

Note that you do want the backward-quotes for the 'date' part,
but not for the full string.  But it would also be easier to read
if you used the more modern alternative for the backward-quotes.

So try:
   ...  -f "pictures-and-music $(date +%D-%M-%Y_%H-%M)"

Also, my guess is that you don't really want that first two
date-values to be "%D-%M".  That gives you the full date followed
by the minute.  I expect you want "%d-%m', which would give you
day-of-month followed by the month-number.

Personally, I prefer to order time values by "bigness", so I'd
use the order of year, month, day, hour, minute.  To get that,
you'd go with:

   ...  -f "pictures-and-music $(date +%Y-%m%d_%H%M)"

Note that you need the string-delimiters to be double-quotes,
not single-quotes.  If you used single-quotes, then the shell
will not expand the $(date...) part.

I hope this helps.

-- garance alistair drosehn


> On Wed, 1 Jun 2016 13:44:54 -0700
> Sarah Alawami  wrote:
>> 
>> Here is the script.
>> 
>> #!/bin/sh
>> 
>> /usr/local/bin/tarsnap -c \
>> -v -f `pictures and music `date +%D-%M-%Y_%H-%M`` \
>> /users/sa/music /users/sa/movies /users/sa/pictures
>> # End of the script...
>> 
>> Do I have the right idea here? It looks like it half way works but it
>> is not naming the archive what I want to name it. Once I get this set
>> up I'll never need to worry again, I hope lol!
>> 
>> Everyone be blessed and have a happy wednesday.

Re: tarsnap warning on reading extended attributes

[Garance AE Drosehn]

> On Jan 20, 2016, Colin Percival  wrote:
> 
> Hi Daniel (& list),
> 
> On 01/18/16 19:05, Daniel LaFlamme wrote:
> 
>> I am on FreeBSD 9.3-RELEASE-p28 using the tarsnap-1.0.36 binary package.
>> 
>> I thought there was something wrong with my jail setup, but apparently
>> the inability to read extended attributes in the system name space is
>> well-known.
>> See http://lists.freebsd.org/pipermail/freebsd-jail/2012-May/001877.html.
> 
> I'm a bit torn here -- from one perspective, it makes sense to figure out
> which (system) extended attributes we won't be able to read and skip them;
> but from another, the user asked to archive these files and we shouldn't
> silently drop extended attributes...

Can tarsnap tell when it is running in a jail?  If so, maybe have it
print out one warning message when it starts up, instead of printing
one for each file where it can't read the extended attributes?  Or
maybe have a command-line option where the user can specify that they
(a) are running in a jail, or (b) don't care about saving extended
attributes?

-- garance alistair drosehn

Re: Feasability of using tarsnap on crap internet connections

[Garance AE Drosehn]

On Nov 5, 2015, at 5:16 PM, Quinn Comendant  wrote:
> 
> Another question has come up while evaluating tarsnap. I'll be uploading from 
> a MacBook in rural Colombia, with a flapping internet connection. It is a 
> 3Mbps radio uplink, which is sufficiently fast to upload my 400GB 
> (pre-de-duplication) data in a month or two (initial upload), BUT the 
> connection is intermittent: the connection usually drops several times per 
> hour, and at worst every few minutes. 
> 
> Does anybody suggest that tarsnap might be the wrong tool to use under such 
> circumstances?

One thing you might do is break up the data you want to save into different 
archives.  You can pick specific directories to archive, for instance, and that 
will take much less time to archive than doing the entire disk.

So you could have one set of tarsnap archives for the Documents folder under 
your userid.  And then another set for whatever folder has the next most 
important group of data for you (perhaps Pictures, for instance).  And then 
make a third archive which is your entire home directory *except* for Documents 
and Pictures.

This way even if it takes a long time to backup your entire disk, you can at 
least make sure the most-important collections of data are completely backed up 
before you spend time backing up other files which wouldn't mean as much to you.

I would suggest that you don't try to get *too* clever with this.  It's still 
nice to back up all the data, and to be reasonably sure that you really do have 
all the data.  For instance you're going to be really sad if you didn't happen 
to backup the ~/Library folder in your home directory, and that might be easy 
to overlook because Apple hides that folder so you won't see it when using the 
finder.

But if it's going to take you *weeks* to do a full backup, then start out by 
saving the files which would hurt the most if your hard disk dies tomorrow 
morning.

-- 
Garance Alistair Drosehn=  gadc...@earthlink.net
Senior Systems Programmer   or   g...@freebsd.org

Re: How to find a deleted file amongst many archives

[Garance AE Drosehn]

On Nov 5, 2015, at 5:21 PM, Quinn Comendant  wrote:

> On Thu, 5 Nov 2015 13:36:05 -0800, Colin Percival wrote:
>> There's no built-in way to do this.  Due to tarsnap's encryption, there's no
>> way to "index" the archives, so any built-in command would just do what you
>> can do manually anyway; there didn't seem much point.
> 
> That makes sense. 
> 
> It would be pretty easy to create a local index during the backup process and 
> save all file paths, upload timestamp, and archive names to a text file or 
> sqlite db. This would make individual file restorations much quicker. Perhaps 
> there exists a tarsnap wrapper that already has this feature?

I have one such wrapper partially written, but in it I'm trying to setup some 
other features which probably don't matter to anyone else.  And of course, it's 
the "other features" which got implemented first, not the index.

-- garance

Re: Ability to tell what has changed

[Garance AE Drosehn]

On Aug 29, 2015, at 3:36 PM, Tim Bishop tim-li...@bishnet.net wrote:

 On Sat, Aug 29, 2015 at 02:36:19PM -0400, Garance AE Drosehn wrote:
 
 How about some way that tarsnap would tell me all the filenames
 that it will include in the archive?  This time I don't mean the
 ones which have New data, but all of the filenames which would
 be included in This Archive?
 
 Does -v not do what you want?

Indeed, it does.  I don't know why I thought that it didn't. (I even
tried it back on the first day I was trying tarsnap, and somehow got
the idea that it wasn't doing what I wanted when used with --dry-run).

Thanks!

-- 
Garance Alistair Drosehn=  gadc...@earthlink.net
Senior Systems Programmer   or   g...@freebsd.org

Ability to tell what has changed

[Garance AE Drosehn]

I'm just getting started up with tarsnap, so this request might
be something that is already available but I haven't noticed how
to do it.

A few days ago I created an archive for a folder called cvsdepot.
Today I try a dry-run for creating another archive on that folder,
and tarsnap tells me:
   Total size  Compressed size
This archive20_468320 5_974899
New data   109878 7560

The thing is that I'm pretty sure I haven't changed anything in that
folder (although it is possible I did).  I do see that one subdirectory
named lily-rtools has been modified in the last two days. but none
of the files in all of cvsdepot have been modified.  And all (three)
of the files in lily-rtools add up to just 91272 bytes.  So I'm curious
what files tarsnap has determined to contain some new data.  In this
case this question is nothing more than idle curiosity, just because
I'm still in the process of learning tarsnap.  I'm confident there is
nothing wrong.   (fwiw, I also installed the new version 1.0.36 of
tarsnap since my earlier run)

But one of the reasons I like the idea of doing dry-runs is to see
if the amount of new data to backup seems reasonable.  I've been
known to download or generate pretty huge files temporarily, only
to forget about those files for years.  And there was one time that
a new version of the Tivoli Storage Manager caused my config file
to be handled differently, and I nearly backed up almost 1 TB of
data which I really did *not* want to be backed up to TSM.

Is there any way that tarsnap would tell me which files have new
data, at least when I'm doing a dry-run?  It would probably be
nice to have two options:  (1) a count of files which have new
data, (2) a list of the specific filenames.

And in a related question:  Is there way to do a dry-run and find
out that there is *no* new data to back up?  For some of the
folders that I back up, I do not want to create another backup if
nothing has changed.  Right now I check and assume that if the
new-data size is under 5000, then there's no point in creating
the backup.  Obviously a script could do this if tarsnap provides
a way to get the count of files with new data, but if it isn't
easy to get an exact count then it'd be nice to at least know
the difference between something somewhere has changed and
nothing has changed.

-- 
Garance Alistair Drosehn=  gadc...@earthlink.net
Senior Systems Programmer   or   g...@freebsd.org