subject:"backup strategy"

Excellent discussion Jon!  Thank you so very much for your thoughts!!

I was aware of Amanda's philosophy of "I want one full dump every dumpcycle
which is n days" and then to let Amanda find the optimal combination of
full/incrementals, and that's one of the reasons that Amanda was so
attractive, other than it uses native compression tools (BIG plus for me).

Heck, I even learned about the Tower of Hanoi scheme!  But one piece of
advice stuck with me, and that is "when restoring data, you don't have time
to try to remember/decipher which files are on which tapes". In other
words, backup / on a regular schedule, even if it'll cost you some disk
space/timeit'll be worth it when it comes time to recover.  I suppose I
can make things more complex when I get really bored. ;)

As I'm working out the backup plan, it's starting to seem that I'll want to
have a dumpcycle of 7 days, and then keep those weekly full dumps
elsewhere.  At this time, we don't have a massive quantity of critical
data, and we do have enough backup storage to keep 52 full dumps/year.  One
the reasons that I'm going to keep so many is because of the way one of our
database has been (poorly) designed. There have been questions asked such
as:

"On which day did this record change?"

Unfortunately, even with the binlog enabled, some of the queries don't get
recorded (long story), so at least for the database config, it seems like
we'll be keeping many full dumps.

Another part of my backup plan is going to have two configs.  One for the
database instance ('hypercritical data') that will be backed up insanely
often, and another config for merely really important data (svn repos,
wikis, etc) which will have a less taxing schedule.

Now I suppose that I could combine these two schedules into one config, but
it seems that it would make more sense from an administrative view to have
two (more?) configs.

Do you/anyone else have any thoughts on that?

--
Andrius

On Sat, Jan 11, 2014 at 9:10 PM, Jon LaBadie  wrote:

> On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> > Hmm...Jon's question got me to a'thnkin' about something that I haven't
> yet
> > read about.
> >
> > So let's say the dump cycle is 30 days and I want to keep a copy of the
> > previous month's dump before the tape1 gets overwritten so that I can
> keep
> > a record of the state of the database from two months ago.
> >
> > If I understand the terminology, this would be an archive?
>
> Andrius,
>
> You might want to read further about the way Amanda differs from most
> backup systems.  Typically they act as you are describing, a full dump
> of everything once in a while and incrementals in-between.  So you get
> a monster dump followed by a bunch of small dumps.
>
> Amanda tries to level out the size of the dumps.  You define objects to
> be backed up, called DiskList Entries (DLEs) in Amanda, and how you want
> them backed up (they need not all be the same).  Amanda then spreads the
> full back ups of the DLEs across the whole dump cycle.  At the beginning
> of each dump run Amanda spends time determining how big a dump it expects
> from each DLE as several levels, at least two, level 0 full and level 1
> incremental.  Then it determines how best to achieve balanced dumping
> while still keeping to your defined parameters.  For any specific DLE it
> may continue to do a level 1 incremental, switch it to a smaller level 2
> incremental, or promote it to an early level 0 full dump.
> >
> > So a strategy might be: on the 30th day, do a full dump, and copy it to
> > 'somewhere safe', whether on S3, or a DVD.That seems like a pretty
> > straight-forward concept, but might this idea be improved upon somehow?
>
> A strong recommendation is to never have less than two full dumpcycles.
> I suspect most Amanda installations maintain well over 2 dumpcycles.  If
> you are reusing your storage media, and you only have one level 0,
> should it be overwritten by mistake or something bad happen to it, you
> only have less useful incrementals.
>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (609) 477-8330 (C)
>

Re: Backup Strategy

2014-01-11 Thread Jon LaBadie

On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> Hmm...Jon's question got me to a'thnkin' about something that I haven't yet
> read about.
> 
> So let's say the dump cycle is 30 days and I want to keep a copy of the
> previous month's dump before the tape1 gets overwritten so that I can keep
> a record of the state of the database from two months ago.
> 
> If I understand the terminology, this would be an archive?

Andrius,

You might want to read further about the way Amanda differs from most
backup systems.  Typically they act as you are describing, a full dump
of everything once in a while and incrementals in-between.  So you get
a monster dump followed by a bunch of small dumps.

Amanda tries to level out the size of the dumps.  You define objects to
be backed up, called DiskList Entries (DLEs) in Amanda, and how you want
them backed up (they need not all be the same).  Amanda then spreads the
full back ups of the DLEs across the whole dump cycle.  At the beginning
of each dump run Amanda spends time determining how big a dump it expects
from each DLE as several levels, at least two, level 0 full and level 1
incremental.  Then it determines how best to achieve balanced dumping
while still keeping to your defined parameters.  For any specific DLE it
may continue to do a level 1 incremental, switch it to a smaller level 2
incremental, or promote it to an early level 0 full dump.
> 
> So a strategy might be: on the 30th day, do a full dump, and copy it to
> 'somewhere safe', whether on S3, or a DVD.That seems like a pretty
> straight-forward concept, but might this idea be improved upon somehow?

A strong recommendation is to never have less than two full dumpcycles.
I suspect most Amanda installations maintain well over 2 dumpcycles.  If
you are reusing your storage media, and you only have one level 0,
should it be overwritten by mistake or something bad happen to it, you
only have less useful incrementals.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

2014-01-11 Thread Charles Curley

On Sat, 11 Jan 2014 12:05:58 -0800
"Andrius D. Ilgunas"  wrote:

> I would expect them to be network-type speeds on a T1 even as the
> buckets are mounted via FUSE.  I don't have numbers, but I can say
> that a copy of one of my databases to /mnt/s3/bucket1 of around 250MB
> will timeout approximately 30% of the time.  But I'm sure that
> copying over incremental changes, either via amanda or rsync would be
> successful.

I'm not so sure about using FUSE. Check my thinking here, folks.

If you rsync between two computers, then rsync runs on both machines.
The two instances calculate hashes for chunks of the data, and send
the hashes back and forth. Based on the hashes, the originating instance
then sends only the changed chunks to the other machine. So you don't
have to carry the entire data set across the network in order to back
up the entire data set.

If you mount via FUSE, Samba, NFS or whatever, both instances of rsync
runs locally. So the "remote" rsync reads the entire archive over the
network in order to calculate its hashes. That's a huge overhead.

I have 69 virtual tapes, and typically use one, rarely two, tapes a
day. If I archive daily, I only have to move a few percent of my "tape
library" to the archive. Inverting that, if I archived via a network
mount, I would increase my overhead by hundreds of percent.

Can you have a remote virtual machine and rsync to it? Preferably over
SSH.

Of course, your data set may be small enough that you don't care.
That's fine, too.

-- 

The right of the people to be secure in their persons, houses, papers,
and effects, against unreasonable searches and seizures, shall not be
violated, and no Warrants shall issue, but upon probable cause,
supported by Oath or affirmation, and particularly describing the
place to be searched, and the persons or things to be seized.
-- U.S. Const. Amendment IV

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: Backup Strategy

Hmm...Jon's question got me to a'thnkin' about something that I haven't yet
read about.

So let's say the dump cycle is 30 days and I want to keep a copy of the
previous month's dump before the tape1 gets overwritten so that I can keep
a record of the state of the database from two months ago.

If I understand the terminology, this would be an archive?

So a strategy might be: on the 30th day, do a full dump, and copy it to
'somewhere safe', whether on S3, or a DVD.That seems like a pretty
straight-forward concept, but might this idea be improved upon somehow?

I just finished reading O'Reilly's Backup and Restore (a very disappointing
book) that only brushed over some strategies, and Google seems to show me
only the same few pages on how to setup a config with only a topical
discussion about scheduling.

Anyway, thanks for your responses and ideas!


--
Andrius


On Sat, Jan 11, 2014 at 12:05 PM, Andrius D. Ilgunas wrote:

> Hey Jon,
>
> I would expect them to be network-type speeds on a T1 even as the buckets
> are mounted via FUSE.  I don't have numbers, but I can say that a copy of
> one of my databases to /mnt/s3/bucket1 of around 250MB will timeout
> approximately 30% of the time.  But I'm sure that copying over incremental
> changes, either via amanda or rsync would be successful.
>
> At this time, my holding disk (two 2TB drives - zfs mirrored) is
> sufficient to contain two or three full dumps of / of both clients that I'm
> interested in backing up, and probably will be for another year or so - the
> data growth is expected to be around 7% annually.
>
> And now that I brought that up, I haven't yet considered any kind of
> long-term archiving. One step at a time though.
>
>
>
> --
> Andrius
>
>
> On Sat, Jan 11, 2014 at 11:38 AM, Jon LaBadie  wrote:
>
>> On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
>> > Thanks a brazillion Charles!!
>> >
>> > I'll probably setup my systems likewise, but I wonder if anyone has any
>> > other opinions on this.
>> >
>> > --
>> > Andrius
>>
>> What kind of write speed to S3 can you expect?
>> Can your backups to holding disk exceed that by a sufficient
>> amount that holding disk capacity might be a problem?
>>
>> Jon
>> >
>> >
>> > On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
>> > charlescur...@charlescurley.com> wrote:
>> >
>> > > On Sat, 11 Jan 2014 09:44:07 -0800
>> > > "Andrius D. Ilgunas"  wrote:
>> > >
>> > > > We're setting up amanda on our servers, and the primary backup is
>> > > > going to be on a dedicated disk/virtual tapes.  One of the offsite
>> > > > locations is going to be a bucket on Amazon's S3.
>> > > >
>> > > > Now I see that amanda has the capability of writing to multiple
>> > > > volumes in parallel, but I'm wondering if it would be more efficient
>> > > > to have amada write the backup to the localdisk first and then use
>> > > > rsync to mirror that backup onto the S3 bucket.
>> > > >
>> > > > Or would it be a better idea to have amanda manage both
>> > > > "tapechangers"?
>> > > >
>> > > > Any discussion would be most welcome!
>> > >
>> > > What I do is similar to your first proposal. I back up to virtual
>> > > tapes, then rsync to external hard drive for off site backup.
>> > >
>> > >
>> > >
>> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
>> > >
>> > > You probably want to save the metadata so that restores from offsite
>> > > will be easier.
>> > >
>> > >
>> > >
>> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
>> > >
>> > > --
>> > >
>> > > The right of the people to be secure in their persons, houses, papers,
>> > > and effects, against unreasonable searches and seizures, shall not be
>> > > violated, and no Warrants shall issue, but upon probable cause,
>> > > supported by Oath or affirmation, and particularly describing the
>> > > place to be searched, and the persons or things to be seized.
>> > > -- U.S. Const. Amendment IV
>> > >
>> > > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
>> > >
>> >>> End of included message <<<
>>
>> --
>> Jon H. LaBadie j...@jgcomp.com
>>  11226 South Shore Rd.  (703) 787-0688 (H)
>>  Reston, VA  20190  (609) 477-8330 (C)
>>
>
>

Re: Backup Strategy

Hey Jon,

I would expect them to be network-type speeds on a T1 even as the buckets
are mounted via FUSE.  I don't have numbers, but I can say that a copy of
one of my databases to /mnt/s3/bucket1 of around 250MB will timeout
approximately 30% of the time.  But I'm sure that copying over incremental
changes, either via amanda or rsync would be successful.

At this time, my holding disk (two 2TB drives - zfs mirrored) is sufficient
to contain two or three full dumps of / of both clients that I'm interested
in backing up, and probably will be for another year or so - the data
growth is expected to be around 7% annually.

And now that I brought that up, I haven't yet considered any kind of
long-term archiving. One step at a time though.



--
Andrius


On Sat, Jan 11, 2014 at 11:38 AM, Jon LaBadie  wrote:

> On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
> > Thanks a brazillion Charles!!
> >
> > I'll probably setup my systems likewise, but I wonder if anyone has any
> > other opinions on this.
> >
> > --
> > Andrius
>
> What kind of write speed to S3 can you expect?
> Can your backups to holding disk exceed that by a sufficient
> amount that holding disk capacity might be a problem?
>
> Jon
> >
> >
> > On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
> > charlescur...@charlescurley.com> wrote:
> >
> > > On Sat, 11 Jan 2014 09:44:07 -0800
> > > "Andrius D. Ilgunas"  wrote:
> > >
> > > > We're setting up amanda on our servers, and the primary backup is
> > > > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > > > locations is going to be a bucket on Amazon's S3.
> > > >
> > > > Now I see that amanda has the capability of writing to multiple
> > > > volumes in parallel, but I'm wondering if it would be more efficient
> > > > to have amada write the backup to the localdisk first and then use
> > > > rsync to mirror that backup onto the S3 bucket.
> > > >
> > > > Or would it be a better idea to have amanda manage both
> > > > "tapechangers"?
> > > >
> > > > Any discussion would be most welcome!
> > >
> > > What I do is similar to your first proposal. I back up to virtual
> > > tapes, then rsync to external hard drive for off site backup.
> > >
> > >
> > >
> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
> > >
> > > You probably want to save the metadata so that restores from offsite
> > > will be easier.
> > >
> > >
> > >
> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
> > >
> > > --
> > >
> > > The right of the people to be secure in their persons, houses, papers,
> > > and effects, against unreasonable searches and seizures, shall not be
> > > violated, and no Warrants shall issue, but upon probable cause,
> > > supported by Oath or affirmation, and particularly describing the
> > > place to be searched, and the persons or things to be seized.
> > > -- U.S. Const. Amendment IV
> > >
> > > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
> > >
> >>> End of included message <<<
>
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (609) 477-8330 (C)
>

Re: Backup Strategy

2014-01-11 Thread Jon LaBadie

On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
> Thanks a brazillion Charles!!
> 
> I'll probably setup my systems likewise, but I wonder if anyone has any
> other opinions on this.
> 
> --
> Andrius

What kind of write speed to S3 can you expect?
Can your backups to holding disk exceed that by a sufficient
amount that holding disk capacity might be a problem?

Jon
> 
> 
> On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
> charlescur...@charlescurley.com> wrote:
> 
> > On Sat, 11 Jan 2014 09:44:07 -0800
> > "Andrius D. Ilgunas"  wrote:
> >
> > > We're setting up amanda on our servers, and the primary backup is
> > > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > > locations is going to be a bucket on Amazon's S3.
> > >
> > > Now I see that amanda has the capability of writing to multiple
> > > volumes in parallel, but I'm wondering if it would be more efficient
> > > to have amada write the backup to the localdisk first and then use
> > > rsync to mirror that backup onto the S3 bucket.
> > >
> > > Or would it be a better idea to have amanda manage both
> > > "tapechangers"?
> > >
> > > Any discussion would be most welcome!
> >
> > What I do is similar to your first proposal. I back up to virtual
> > tapes, then rsync to external hard drive for off site backup.
> >
> >
> > http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
> >
> > You probably want to save the metadata so that restores from offsite
> > will be easier.
> >
> >
> > http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
> >
> > --
> >
> > The right of the people to be secure in their persons, houses, papers,
> > and effects, against unreasonable searches and seizures, shall not be
> > violated, and no Warrants shall issue, but upon probable cause,
> > supported by Oath or affirmation, and particularly describing the
> > place to be searched, and the persons or things to be seized.
> > -- U.S. Const. Amendment IV
> >
> > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
> >
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

Thanks a brazillion Charles!!

I'll probably setup my systems likewise, but I wonder if anyone has any
other opinions on this.

--
Andrius


On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
charlescur...@charlescurley.com> wrote:

> On Sat, 11 Jan 2014 09:44:07 -0800
> "Andrius D. Ilgunas"  wrote:
>
> > We're setting up amanda on our servers, and the primary backup is
> > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > locations is going to be a bucket on Amazon's S3.
> >
> > Now I see that amanda has the capability of writing to multiple
> > volumes in parallel, but I'm wondering if it would be more efficient
> > to have amada write the backup to the localdisk first and then use
> > rsync to mirror that backup onto the S3 bucket.
> >
> > Or would it be a better idea to have amanda manage both
> > "tapechangers"?
> >
> > Any discussion would be most welcome!
>
> What I do is similar to your first proposal. I back up to virtual
> tapes, then rsync to external hard drive for off site backup.
>
>
> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
>
> You probably want to save the metadata so that restores from offsite
> will be easier.
>
>
> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
>
> --
>
> The right of the people to be secure in their persons, houses, papers,
> and effects, against unreasonable searches and seizures, shall not be
> violated, and no Warrants shall issue, but upon probable cause,
> supported by Oath or affirmation, and particularly describing the
> place to be searched, and the persons or things to be seized.
> -- U.S. Const. Amendment IV
>
> Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
>

Re: Backup Strategy

2014-01-11 Thread Charles Curley

On Sat, 11 Jan 2014 09:44:07 -0800
"Andrius D. Ilgunas"  wrote:

> We're setting up amanda on our servers, and the primary backup is
> going to be on a dedicated disk/virtual tapes.  One of the offsite
> locations is going to be a bucket on Amazon's S3.
> 
> Now I see that amanda has the capability of writing to multiple
> volumes in parallel, but I'm wondering if it would be more efficient
> to have amada write the backup to the localdisk first and then use
> rsync to mirror that backup onto the S3 bucket.
> 
> Or would it be a better idea to have amanda manage both
> "tapechangers"?
> 
> Any discussion would be most welcome!

What I do is similar to your first proposal. I back up to virtual
tapes, then rsync to external hard drive for off site backup.

http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html

You probably want to save the metadata so that restores from offsite
will be easier.

http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html

-- 

The right of the people to be secure in their persons, houses, papers,
and effects, against unreasonable searches and seizures, shall not be
violated, and no Warrants shall issue, but upon probable cause,
supported by Oath or affirmation, and particularly describing the
place to be searched, and the persons or things to be seized.
-- U.S. Const. Amendment IV

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Backup Strategy

Hey All!

We're setting up amanda on our servers, and the primary backup is going to
be on a dedicated disk/virtual tapes.  One of the offsite locations is
going to be a bucket on Amazon's S3.

Now I see that amanda has the capability of writing to multiple volumes in
parallel, but I'm wondering if it would be more efficient to have amada
write the backup to the localdisk first and then use rsync to mirror that
backup onto the S3 bucket.

Or would it be a better idea to have amanda manage both "tapechangers"?

Any discussion would be most welcome!


--
Andrius

Re: backup strategy

2010-04-16 Thread Chris Hoogendyk




Stefan Zörner wrote:

Hi,

we are about to build up a backup system onto a network drive. We do 
not need long term full backup, we just want to be able to access data 
maximal a month old.
We thought of doing one full backup and then just doing cumulative 
backups. We are planning on using it over a longer period and since 
after a year the data can differ a lot from the original full backup 
each cumulative backup can get quite large. A possibility of course is 
to do a full backup now and then.


My question is, that instead of working with cumulative backup I want 
to use incremental ones. Now normally at the end of a cycle the oldest 
incremental backup is overwritten and the whole backup becomes 
worthless. Is there a method were the oldest increments is included 
into the full backup itself after a cycle has been reached?

Example:
- one full backup
- 10 increments
before starting increment 11. Renew full backup by including oldest 
incremental backup.


Thanks in advance for any help or pointers. 


You should take a look at this:

  
http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F


before launching off into backup strategy ideas based on prior knowledge 
from other kinds of backup software. After you have read through that a 
couple of times, you might have different questions.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Rory Campbell-Lange


On 17/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> Cyrille Bollu wrote:
> >Chris Hoogendyk wrote:
> >> I wouldn't put the holding disks in raid.

> http://blogs.zdnet.com/Ou/?p=484.
> 
> While this guy is looking at things like database servers and
> exchange, we ought to be able to interpret this for Amanda. A couple
> of points to note for Amanda: Amanda will use all the holding disk
> drives while doing parallel backups and storing output on the
> holding disks. When it is writing to tape, it is constrained by the
> sequential nature of the tape, and will only be doing one DLE at a
> time from those that it has completed on the holding disks. Also,
> Amanda's access is heavily sequential, although it may have multiple
> parallel processes hitting the drives.

A slow RAID1 off two 7200 RPM SATA disks on a BBU-backed LSI hardware
raid controller can do about 62031 KiB/s write and 86399 KiB/s read.
Those sorts of numbers improve steadily the number of spindles you add
to a RAID collection and the higher the RAID number and (in the case of
writing) if cacheing is enabled.

On 16/08/09, Rory Campbell-Lange (r...@campbell-lange.net) wrote:
> On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
> > Chris Hoogendyk wrote:
> With reference to Chris Hoogendyk's email "clarification on
> parallelism", I am very curious to learn if Amanda "...still require[s]
> a DLE to be completed to holding disk before it will send any of it to
> tape..." In our case this is a particularly important question as,
> although we can add in more AoE storage for a DLE, this will only run at
> the speeds above. Do we need a 1TB SAS disk array too?

>From the discussion here it seems preferable to have a DLE on two major
counts. One is that compression can happen prior to writing to tape,
which could result in shoe-shining, and another is that Amanda will be
clearer about the amount of data it will be trying to write to a tape,
in other words it will do a better fit of data to tape.

The most important question I now have to ask is:

How fast can a SAS-based LTO4 drive write to tape?

Regards
Rory


-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Chris Hoogendyk




Cyrille Bollu wrote:

Chris Hoogendyk wrote:
> I wouldn't put the holding disks in raid.

Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it 
isn't fast... I always wondered if I should use seperated (non-RAID) 
drives...


Here is an extremely interesting article that everyone should take a 
look at. Rather than just giving theoretical comparisons that you see in 
places like the Wikipedia raid article (which is nevertheless an 
excellent reference), this article is a case study analysis with both 
real and test environments throwing data at raid and instrumenting it -- 
http://blogs.zdnet.com/Ou/?p=484.


While this guy is looking at things like database servers and exchange, 
we ought to be able to interpret this for Amanda. A couple of points to 
note for Amanda: Amanda will use all the holding disk drives while doing 
parallel backups and storing output on the holding disks. When it is 
writing to tape, it is constrained by the sequential nature of the tape, 
and will only be doing one DLE at a time from those that it has 
completed on the holding disks. Also, Amanda's access is heavily 
sequential, although it may have multiple parallel processes hitting the 
drives.



--
---

Chris Hoogendyk

-
O__  Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Alan Hodgson

> On Monday 17 August 2009, Cyrille Bollu wrote:
> >> I wouldn't put the holding disks in raid.
> >
> >Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it
> > isn't fast... I always wondered if I should use seperated (non-RAID)
> > drives...

To drive an LTO-4 your holding disk needs to read somewhat over 100MB/sec 
sequentially, which requires at least 2 drives striped, but should be easy 
enough with any modern raid controller or software raid. 

Complicating this, it may also need to write at similar speed, 
simultaneously, which introduces a random access element and ups the demand 
considerably. I would think you would probably want at least 4 big SATA 
drives striped together to reliably feed an LTO-4 drive at full speed. 
Conveniently, this could also give you over 5TB of very cheap holding disk 
space.

Also, unless you're backing up exclusively large files over a fast SAN link 
(faster than Gig-E), I doubt you could get anywhere close to full tape 
performance without a holding disk.

-- 
"... a serious depression seems improbable; [we expect] recovery of business
next spring, with further improvement in the fall." Harvard Economic
Society, November 10, 1929

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Gene Heskett

On Monday 17 August 2009, Cyrille Bollu wrote:
>> I wouldn't put the holding disks in raid.
>
>Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it isn't
>fast... I always wondered if I should use seperated (non-RAID) drives...

Its been my observation that software raids are slower. Not tremendously so 
though.  When Jim put together a raid-5 with 4 drives several years ago, the 
drives were about 70meg/sec drives, and the overall was just a hair over 
50meg/sec.  I know he has rebuilt it with bigger & faster drives 2 or 3 times 
since, along with more iron in the cpu, so I don't know its current speed.  
Being retired means being out of the loop. :(

>Cyrille Bollu
>Responsable systèmes
>Fedasil - ICT
>tel: +32.2.213.43.49
>gsm: +32.478.23.08.15

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.

A prig is a fellow who is always making you a present of his opinions.
-- George Eliot

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Cyrille Bollu

> 
> I wouldn't put the holding disks in raid.

Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it isn't 
fast... I always wondered if I should use seperated (non-RAID) drives... 

Cyrille Bollu
Responsable systèmes
Fedasil - ICT
tel: +32.2.213.43.49
gsm: +32.478.23.08.15

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-17 Thread Chris Hoogendyk




Rory Campbell-Lange wrote:

On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
  

Chris Hoogendyk wrote:

Amanda will do the compression for you. You define it in the dumptype in 
amanda.conf. If you have a holding disk, then it will compress the data 
as it goes onto the holding disk. If you don't have a holding disk, then 
you might have issues with being able to stream a backup to tape, 
compressing it on the fly. Even with a really fast cpu, I don't know if 
you can maintain the throughput to drive LTO4 at a good speed.
  

You might want to consider configuring for client compression.  Not
only will that give you more CPU for feeding your tape, it also
minimizes network bandwidth. As usual, YMMV, it all depends on where
the bottlenecks are in your environment.



In our case the server _is_ the only client, with up to 30TB of direct
attached storage, with the storage running at between 80MB/s and 120MB/s
access speeds (Bytes rather than bytes).

I don't know if this is fast enough to deal with a SAS connected LTO4
drive, particularly if it is doing software compression along the way.

With reference to Chris Hoogendyk's email "clarification on
parallelism", I am very curious to learn if Amanda "...still require[s]
a DLE to be completed to holding disk before it will send any of it to
tape..." In our case this is a particularly important question as,
although we can add in more AoE storage for a DLE, this will only run at
the speeds above. Do we need a 1TB SAS disk array too?


You will get the best performance if you can do that. If the disk that 
is being copied to tape can give the speed the tape needs, that's going 
to do a better job of keeping things moving.


You have a couple of options.

You can go without a holding disk, and then each DLE will be streamed 
sequentially to tape. This will stretch out your backups. It will also 
mean that any compression you do in software will be done in line with 
that sequential stream. Your system may not be able to keep that all 
flying fast enough for the tape, and you may end up with shoe shining 
and very low speeds. You can certainly try it and see what happens. If 
(when?) that fails, you could try using hardware compression on the tape 
drive. The backups will still be sequential, one DLE streaming to tape 
at a time, and if your drives can't keep up, it will be slower than you 
might like. But, at least you are not dealing with network backups.


The option I would try, budget allowing, would be to add a couple of SAS 
drives to be used as holding disks. Then break up your DLEs so that each 
DLE is substantially smaller than the holding disks. Then Amanda can run 
them in parallel, compress them on the holding disk, and then stream 
completed, compressed DLEs from the holding disk to the tape.


I wouldn't put the holding disks in raid.


--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-16 Thread Rory Campbell-Lange

On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
> Chris Hoogendyk wrote:
> > Amanda will do the compression for you. You define it in the dumptype in 
> > amanda.conf. If you have a holding disk, then it will compress the data 
> > as it goes onto the holding disk. If you don't have a holding disk, then 
> > you might have issues with being able to stream a backup to tape, 
> > compressing it on the fly. Even with a really fast cpu, I don't know if 
> > you can maintain the throughput to drive LTO4 at a good speed.
> 
> You might want to consider configuring for client compression.  Not
> only will that give you more CPU for feeding your tape, it also
> minimizes network bandwidth. As usual, YMMV, it all depends on where
> the bottlenecks are in your environment.

In our case the server _is_ the only client, with up to 30TB of direct
attached storage, with the storage running at between 80MB/s and 120MB/s
access speeds (Bytes rather than bytes).

I don't know if this is fast enough to deal with a SAS connected LTO4
drive, particularly if it is doing software compression along the way.

With reference to Chris Hoogendyk's email "clarification on
parallelism", I am very curious to learn if Amanda "...still require[s]
a DLE to be completed to holding disk before it will send any of it to
tape..." In our case this is a particularly important question as,
although we can add in more AoE storage for a DLE, this will only run at
the speeds above. Do we need a 1TB SAS disk array too?

Rory

-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-14 Thread Frank Smith

Chris Hoogendyk wrote:
> 
> Rory Campbell-Lange wrote:
>> Hi Chris
>>
>> On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
>>   
>> 
>>> Typically, we set up Amanda with holding disk space.
>>> 
>> 
>>
>> If all the storage is locally attached (actually, AoE drives storage
>> units connected over Ethernet), I am hoping to avoid the disk space if I
>> can write to tape fast enough. I'd like to avoid paying for up to 15TB
>> of fast holding disk space if I can avoid it.
> 
> So, one way would be to logically divide the storage into smaller DLE's. 
> A DLE (Disk List Entry -- http://wiki.zmanda.com/man/disklist.5.html) 
> for Amanda can be a mount point or directory. Obviously, I don't know 
> how your storage is organized; but, if you can define your DLE's as 
> separate directories on the storage device, each one of which is much 
> smaller, then you could use a smaller holding disk and still benefit 
> from Amanda's parallelism. In one of the other departments here, the 
> sysadmin has successfully divided a large array this way and is driving 
> LTO4 near top speed.
> 
>>> Compression can be done either on the client, on the server, or on
>>> the tape drive. Obviously, if you use software compression, you want
>>> to turn off the tape drive compression. I use server side
>>> compression, because I have a dedicated Amanda server that can
>>> handle it. By not using the tape drive compression, Amanda has more
>>> complete information on data size and tape usage for its planning.
>>> If your server is more constrained than your clients, you could use
>>> client compression. This is specified in your dumptypes in your
>>> amanda.conf.
>>> 
>> I don't have any clients, so this is an interesting observation. I'll be
>> trying to do sofware compression then I think. The Unix backup book
>> (google for "amanda software compression") suggests that compression can
>> be used on a "per-image basis"; presumably I can pass the backup data
>> stream through gzip or bzip2 on the way to a tape?
> 
> Amanda will do the compression for you. You define it in the dumptype in 
> amanda.conf. If you have a holding disk, then it will compress the data 
> as it goes onto the holding disk. If you don't have a holding disk, then 
> you might have issues with being able to stream a backup to tape, 
> compressing it on the fly. Even with a really fast cpu, I don't know if 
> you can maintain the throughput to drive LTO4 at a good speed.

You might want to consider configuring for client compression.  Not
only will that give you more CPU for feeding your tape, it also
minimizes network bandwidth. As usual, YMMV, it all depends on where
the bottlenecks are in your environment.

Frank


-- 
Frank Smith  fsm...@hoovers.com
Sr. Systems Administrator   Voice: 512-374-4673
Hoover's Online   Fax: 512-374-4501

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-14 Thread Chris Hoogendyk




Rory Campbell-Lange wrote:

Hi Chris

On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
  


Typically, we set up Amanda with holding disk space.




If all the storage is locally attached (actually, AoE drives storage
units connected over Ethernet), I am hoping to avoid the disk space if I
can write to tape fast enough. I'd like to avoid paying for up to 15TB
of fast holding disk space if I can avoid it.


So, one way would be to logically divide the storage into smaller DLE's. 
A DLE (Disk List Entry -- http://wiki.zmanda.com/man/disklist.5.html) 
for Amanda can be a mount point or directory. Obviously, I don't know 
how your storage is organized; but, if you can define your DLE's as 
separate directories on the storage device, each one of which is much 
smaller, then you could use a smaller holding disk and still benefit 
from Amanda's parallelism. In one of the other departments here, the 
sysadmin has successfully divided a large array this way and is driving 
LTO4 near top speed.



Compression can be done either on the client, on the server, or on
the tape drive. Obviously, if you use software compression, you want
to turn off the tape drive compression. I use server side
compression, because I have a dedicated Amanda server that can
handle it. By not using the tape drive compression, Amanda has more
complete information on data size and tape usage for its planning.
If your server is more constrained than your clients, you could use
client compression. This is specified in your dumptypes in your
amanda.conf.



I don't have any clients, so this is an interesting observation. I'll be
trying to do sofware compression then I think. The Unix backup book
(google for "amanda software compression") suggests that compression can
be used on a "per-image basis"; presumably I can pass the backup data
stream through gzip or bzip2 on the way to a tape?


Amanda will do the compression for you. You define it in the dumptype in 
amanda.conf. If you have a holding disk, then it will compress the data 
as it goes onto the holding disk. If you don't have a holding disk, then 
you might have issues with being able to stream a backup to tape, 
compressing it on the fly. Even with a really fast cpu, I don't know if 
you can maintain the throughput to drive LTO4 at a good speed.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-14 Thread Chris Hoogendyk




Cyrille Bollu wrote:

Here's my (very) small personnal experience:

A few years ago, when I tried it, I couldn't enable server-side 
software compression while bypassing the holding disk with my IBM 
ULTIUM LTO-3 drive: Tape speed was sinking to about 5MB/s.


My backup server was a Dell PowerEdge 2850 with 4 Intel Xeon 3GHz and 
8MB RAM using RHEL-4.0 and amanda-2.4.4p3-1.


Maybe did I do something wrong at that time (I just had 1 try). Beware 
though. 


That actually makes perfect sense.

By not using a holding disk, you are disabling Amanda's ability to run 
multiple things in parallel. The tape device now controls everything. 
That is to say, you cannot do a backup without streaming it to the tape. 
So, you cannot do more than one at a time. Furthermore, as that one 
backup gets done, the compression has to be done as it is being streamed 
to the tape. So all the processes from reading a remote disk, to 
transferring it over the network, to compressing it, to writing it to 
the tape are all tied together in a single pipe. Any slowdown along that 
pipe will affect everything else. When the tape doesn't get what it 
needs to keep going, it will stop and then have to start up again and 
reposition, and then you get shoe shining.


When you use a holding disk, Amanda can stream multiple backups to the 
holding disk simultaneously. It can compress them there when it has them 
and do it in parallel with other processes. Once it has something ready 
to go to tape, it can dd it straight from the disk to the tape as an 
independent process in parallel with the other things that are going on. 
That final step out to the tape is no longer constrained by any of the 
other steps along the way. Now all you have to worry about is tuning 
various pieces of hardware and software to get the throughput you want.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-14 Thread Cyrille Bollu

Hi,

Here's my (very) small personnal experience:

A few years ago, when I tried it, I couldn't enable server-side software 
compression while bypassing the holding disk with my IBM ULTIUM LTO-3 
drive: Tape speed was sinking to about 5MB/s.

My backup server was a Dell PowerEdge 2850 with 4 Intel Xeon 3GHz and 8MB 
RAM using RHEL-4.0 and amanda-2.4.4p3-1.

Maybe did I do something wrong at that time (I just had 1 try). Beware 
though.

Cyrille

owner-amanda-us...@amanda.org wrote on 14/08/2009 15:57:45:

> Hi Chris
> 
> On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> > ... the solution is akin to the Japanese monks caring for Bonzai
> 
> I liked this idea about tape archives -- constant pruning and
> maintenance. Difficult to sell though.
> 
> > As for your specific questions:
> > 
> > You should be able to do LVM snapshots. I use fssnap on Solaris 9 and
> > 10, and scanning through, here are just a couple of references I find
> > to people using LVM snapshots with Amanda:
> 
> > With the latest releases of Amanda, there is a new API that could make
> > it even easier to implement.
> 
> Great; thanks for the pointers.
> 
> > Typically, we set up Amanda with holding disk space.
> 
> 
> If all the storage is locally attached (actually, AoE drives storage
> units connected over Ethernet), I am hoping to avoid the disk space if I
> can write to tape fast enough. I'd like to avoid paying for up to 15TB
> of fast holding disk space if I can avoid it.
> 
> > Compression can be done either on the client, on the server, or on
> > the tape drive. Obviously, if you use software compression, you want
> > to turn off the tape drive compression. I use server side
> > compression, because I have a dedicated Amanda server that can
> > handle it. By not using the tape drive compression, Amanda has more
> > complete information on data size and tape usage for its planning.
> > If your server is more constrained than your clients, you could use
> > client compression. This is specified in your dumptypes in your
> > amanda.conf.
> 
> I don't have any clients, so this is an interesting observation. I'll be
> trying to do sofware compression then I think. The Unix backup book
> (google for "amanda software compression") suggests that compression can
> be used on a "per-image basis"; presumably I can pass the backup data
> stream through gzip or bzip2 on the way to a tape?
> 
> > Deduplication is not available with Amanda. However, some people
> > stage different kinds of tools and use Amanda for the final staging
> > and management of tapes and archives. So, in some situations,
> > BackupPC could be used to do deduplication from, say, desktop
> > clients to a server archive which is then backed up by Amanda. That
> > could start complicating your 12 year recovery scenario and what
> > happens when software is not available or doesn't run.
> 
> Great -- thanks for the details.
> 
> > Amanda uses the term "index" rather than "catalog" -- see
> > http://wiki.zmanda.com/index.php/Amanda_Index.
> > 
> > Note that if you are putting tapes into a long term archive with no
> > intent of recycling them in subsequent backups, you can use amadmin
> > to mark them as no-reuse. I periodically (typically at the end of
> > semesters) do a force full, mark the tapes as no-reuse, and then
> > pull them out of my tapecycle and put them in storage.
> 
> Very useful again, thanks.
> 
> Regards
> Rory
> -- 
> Rory Campbell-Lange
> Director
> r...@campbell-lange.net
> 
> Campbell-Lange Workshop
> www.campbell-lange.net
> 0207 6311 555
> 3 Tottenham Street London W1T 2AF
> Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-14 Thread Rory Campbell-Lange

Hi Chris

On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> ... the solution is akin to the Japanese monks caring for Bonzai

I liked this idea about tape archives -- constant pruning and
maintenance. Difficult to sell though.

> As for your specific questions:
> 
> You should be able to do LVM snapshots. I use fssnap on Solaris 9 and
> 10, and scanning through, here are just a couple of references I find
> to people using LVM snapshots with Amanda:

> With the latest releases of Amanda, there is a new API that could make
> it even easier to implement.

Great; thanks for the pointers.

> Typically, we set up Amanda with holding disk space.

If all the storage is locally attached (actually, AoE drives storage
units connected over Ethernet), I am hoping to avoid the disk space if I
can write to tape fast enough. I'd like to avoid paying for up to 15TB
of fast holding disk space if I can avoid it.

> Compression can be done either on the client, on the server, or on
> the tape drive. Obviously, if you use software compression, you want
> to turn off the tape drive compression. I use server side
> compression, because I have a dedicated Amanda server that can
> handle it. By not using the tape drive compression, Amanda has more
> complete information on data size and tape usage for its planning.
> If your server is more constrained than your clients, you could use
> client compression. This is specified in your dumptypes in your
> amanda.conf.

I don't have any clients, so this is an interesting observation. I'll be
trying to do sofware compression then I think. The Unix backup book
(google for "amanda software compression") suggests that compression can
be used on a "per-image basis"; presumably I can pass the backup data
stream through gzip or bzip2 on the way to a tape?

> Deduplication is not available with Amanda. However, some people
> stage different kinds of tools and use Amanda for the final staging
> and management of tapes and archives. So, in some situations,
> BackupPC could be used to do deduplication from, say, desktop
> clients to a server archive which is then backed up by Amanda. That
> could start complicating your 12 year recovery scenario and what
> happens when software is not available or doesn't run.

Great -- thanks for the details.

> Amanda uses the term "index" rather than "catalog" -- see
> http://wiki.zmanda.com/index.php/Amanda_Index.
> 
> Note that if you are putting tapes into a long term archive with no
> intent of recycling them in subsequent backups, you can use amadmin
> to mark them as no-reuse. I periodically (typically at the end of
> semesters) do a force full, mark the tapes as no-reuse, and then
> pull them out of my tapecycle and put them in storage.

Very useful again, thanks.

Regards
Rory
-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-13 Thread Chris Hoogendyk

Rory Campbell-Lange wrote:

On 13/08/09, Charles Curley (charlescur...@charlescurley.com) wrote:

On Thu, 13 Aug 2009 01:08:03 -0400
Jon LaBadie wrote:

On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:

So maybe you should provide a complete OS distribution, including the

backup software. Like a customized version of one of the "live CD"
releases of Linux. But wait, will that distribution's included
device drivers work on the devices that will exist in 12 years? Will
that era's computers still have CD drives. Will they be bootable?

Oh, folks 12 years hence ought to be able to dig out 12 year old
computers to run their 12 year old distributions on.

Many thanks for this note, Charles, and to the other notes Chris,
Charles and Jon about their commentary about using Amanda to provide a
long-term archive format. The points about being able to use standard
Unix tools to retrieve information is well made, as is the point that
the current machines and architectures (and CDs!) may not be around in
12 years' time. Thanks very much for those observations.

I'd like to return the other part of my question if I may:

The backup tape format is to be LT04 and we have a second-hand Dell
PowerVault 124T 16 tape autoloader to work with currently. Backup from
a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
snapshot ensuring that the source data does not change during the
backup process. We have the possibility of pre-preparing backup or
compressed images if this is advisable.

I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:

- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?
- after scanning through the wiki I can't see any references to what
I think of as a backup job "catalogue". How does one know what
files were part of a particular backup job?

Thanks for any further advice.

One further comment on the nature of long term archives (and then on to
your specific questions):

I used to work in the Systems Office of the University Library. I
handled backups there, and had close contact with a group of librarians
who were into digital content, archives and special collections. Among
other things, we kicked around ideas about how to archive digital
collections, the life expectancies and failure rates of various types of
CDs (generally terrible in reality), etc. When librarians talk about
archives, they don't just talk decades. They expect things to last a
hundred years and more. In that light, they have concluded that the
solution is akin to the Japanese monks caring for Bonzai. There are
records of Bonzai trees that have been cared for for hundreds of years.
So, think of sysadmins as monks caring for data. The archive librarians
solution is raid6 with hot spares and mirrored to another location. The
sysadmins maintain and update hardware and software and transfer data
when necessary. Although I haven't kept up with that area, they were
developing a cooperative distributed archive software as an open source.
The idea was that different libraries join the cooperative, run the
software, and they end up with multiple copies of their digital
collections distributed geographically among other libraries. If your
library burns down, you rebuild, set up the software, and bring your
collection back. Sort of a cloud library, if you will.

So, as technology changes, you need to be frequently reviewing the state
of your archives, keeping an eye on compatibility bottlenecks and
transferring data to newer media when it becomes necessary. I have a
faculty member who ran his own backups on AIT2 for years. His drives are
fairly old now. I periodically urge him to read them back in to a disk
archive and allow me to put them on AIT5. He's too busy. Ah, well. It's
his data.

As for your specific questions:

You should be able to do LVM snapshots. I use fssnap on Solaris 9 and
10, and scanning through, here are just a couple of references I find to
people using LVM snapshots with Amanda:

http://wiki.zmanda.com/index.php/FAQ:Which_backup_program_for_filesystems_is_better%3F
http://archives.zmanda.com/amanda-archives/viewtopic.php?t=2711&sid=f1535cf0b0782bf2b99aebc033e91c9c
http://archives.zmanda.com/amanda-archives/viewtopic.php?p=9823&sid=8e54f6a0b4ab2cd58bd02e048c299844

In the past that sort of thing had always been done with a wrapper
script (described toward the end of
http://wiki.zmanda.com/index.php/Backup_client). Paul Bijens refers to a
script that he uses in one of the above links. With the latest releases
of Amanda, there is a new API that could make it even easier to implement.

Typically, we set up Amanda with holding disk space.
See the section of the sample amanda.conf partway down regarding holding
disks -- http:

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-13 Thread Rory Campbell-Lange

On 13/08/09, Charles Curley (charlescur...@charlescurley.com) wrote:
> On Thu, 13 Aug 2009 01:08:03 -0400
> Jon LaBadie  wrote:
> > On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:

> > So maybe you should provide a complete OS distribution, including the
> > backup software.  Like a customized version of one of the "live CD"
> > releases of Linux.  But wait, will that distribution's included
> > device drivers work on the devices that will exist in 12 years?  Will
> > that era's computers still have CD drives.  Will they be bootable?
> 
> Oh, folks 12 years hence ought to be able to dig out 12 year old
> computers to run their 12 year old distributions on. 

Many thanks for this note, Charles, and to the other notes Chris,
Charles and Jon about their commentary about using Amanda to provide a
long-term archive format. The points about being able to use standard
Unix tools to retrieve information is well made, as is the point that
the current machines and architectures (and CDs!) may not be around in
12 years' time. Thanks very much for those observations. 

I'd like to return the other part of my question if I may:

> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from
> a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
> snapshot ensuring that the source data does not change during the
> backup process. We have the possibility of pre-preparing backup or
> compressed images if this is advisable.

I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:

- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?   
- after scanning through the wiki I can't see any references to what
  I think of as a backup job "catalogue". How does one know what
  files were part of a particular backup job?   

Thanks for any further advice.

Rory

-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-13 Thread Charles Curley

On Thu, 13 Aug 2009 01:08:03 -0400
Jon LaBadie  wrote:

> On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:
> > 

> So maybe you should provide a complete OS distribution, including the
> backup software.  Like a customized version of one of the "live CD"
> releases of Linux.  But wait, will that distribution's included
> device drivers work on the devices that will exist in 12 years?  Will
> that era's computers still have CD drives.  Will they be bootable?

Oh, folks 12 years hence ought to be able to dig out 12 year old
computers to run their 12 year old distributions on. I suspect the
bottle neck will be finding drives to read LT04 tapes; maybe keep one or
two spares around?

Or put two or three complete machines aside and keep them for the
purpose.

-- 

Charles Curley  /"\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: Post subject: [Amanda-users] Advice needed on Linux backup strategy to LTO

2009-08-13 Thread Rory Campbell-Lange

Apologies that this email doesn't have the correct threading ID. I
posted through Backup Central originally and I can't get hold of the
original mails on the Amanda users list since subscribing properly to it
-- the advertised ftp archives don't seem to exist any more.

Many thanks to Chris, Charles and Jon about their commentary about using
Amanda to provide a long-term archive format. The points about being
able to use standard Unix tools to retrieve information is well made, as
is the point that the current machines and architectures may not be
around in 12 years' time. Thanks very much for those points.

I'd like to return the other part of my question if I may:

> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from
> a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
> snapshot ensuring that the source data does not change during the
> backup process. We have the possibility of pre-preparing backup or
> compressed images if this is advisable. 

I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:

- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?
- after scanning through the wiki I can't see any references to what
  I think of as a backup job "catalogue". How does one know what
  files were part of a particular backup job?

Thanks for any further advice.

Rory


-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-12 Thread Jon LaBadie

On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:
> 
> I'm going to cross-post this text on the Amanda and Bacula lists.
> Apologies in advance if you see this twice.
> 
> Our company is about to provide centralised backups for several pools of
> backup data of between 1 and 15TB in size. Each pool changes daily but
> backups to tape will only occur once a month for each pool.
> 
> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from a
> pool may be taken off a Linux LVM (or hopefully soon a BTRFS) snapshot
> ensuring that the source data does not change during the backup process.
> We have the possibility of pre-preparing backup or compressed images if
> this is advisable.
> 
> An important aspect of the system is that the tapes should be readable
> for 12 years, by other parties if necessary. From this point of view we
> like the idea of providing a CD with each tape set of the software
> needed to extract the contents, together with a listing of the enclosed
> files in a UTF8 text file. We will be required to audit each backup set
> by successfully extracting files from tape.

Others have mentioned that even without amanda software, amanda backups
are recoverable with standard unix/linux tools.

I question whether the concept of providing the software is reasonable.
Programs are compiled for a particular environment, versions of libraries,
devices, operating system, etc.  Amanda software, compiled for today's
systems, is unlikely to be able to run on systems a dozen years from now.
What systems were around in 1997?  Many instances of them still running?  

So maybe you should provide a complete OS distribution, including the backup
software.  Like a customized version of one of the "live CD" releases of
Linux.  But wait, will that distribution's included device drivers work on
the devices that will exist in 12 years?  Will that era's computers still
have CD drives.  Will they be bootable?

This requirement may take some additional thought.

jl
-- 
Jon H. LaBadie  j...@jgcomp.com
 JG Computing
 12027 Creekbend Drive  (703) 787-0884
 Reston, VA  20194  (703) 787-0922 (fax)

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-12 Thread Charles Curley

On Wed, 12 Aug 2009 18:17:17 -0400
rorycl  wrote:

> An important aspect of the system is that the tapes should be readable
> for 12 years, by other parties if necessary. From this point of view
> we like the idea of providing a CD with each tape set of the software
> needed to extract the contents, together with a listing of the
> enclosed files in a UTF8 text file. We will be required to audit each
> backup set by successfully extracting files from tape.

I assume you already have verified that your tapes will last that long
before print-through makes them unreadable.

Another thought is to provide a CD/DVD of a suitable distribution of
Linux with Amanda, mt, etc. That way you don't have compatibility
problems with the current version of Amanda and some future
distribution of Linux. Install and go, and at least then you should be
able to read the tapes.

-- 

Charles Curley  /"\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-12 Thread Chris Hoogendyk




rorycl wrote:

An important aspect of the system is that the tapes should be readable
for 12 years, by other parties if necessary. From this point of view we
like the idea of providing a CD with each tape set of the software
needed to extract the contents, together with a listing of the enclosed
files in a UTF8 text file. We will be required to audit each backup set
by successfully extracting files from tape.


Just taking up that one point for the moment -- Amanda is not just open 
source and open format, but the tape format is based on standard 
UNIX/Linux tools. If you pull off the first file of the tape, it 
actually tells you how to read the tape. You don't need Amanda or any 
Amanda tools to read it. Just standard UNIX/Linux tools that come with 
every distribution, such as dd, gnutar, and gzip.


That said, it is easier to read and recover using the Amanda tools, 
because they will give you an index, allow you to specify what it is you 
want to recover, tell you which tapes you need, and get it for you. But, 
in the event that the tape lands in the hands of a UNIX/Linux admin who 
has never heard of Amanda, but who needs to recover the data, it can be 
done. And those tools are more likely to be available in stable or 
compatible forms in 12 years. It just happens that 12 years is about the 
lifecycle of a particular version of Solaris. That is, from the first 
introduction of Solaris X to its final EOL and drop of all support is 
about 12 years. I think Linux turns over faster than that, but the basic 
tools are typically compatible between versions.


If you want, you can use amreport to generate a report on the contents 
of a backup. Since you won't need a CD of software (and won't need to 
worry about whether it will run, whether the right libraries will be 
available, etc.), you might decide that a printout provided with each 
tape might be easier. Sysadmin looks at printout and immediately sees 
what's on the tape and, Oh, gee, it's that easy to read the tape. That 
avoids the difficulty of a CD not being stable or readable. The tapes 
are typically going to outlive a CD.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

[Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

2009-08-12 Thread rorycl


I'm going to cross-post this text on the Amanda and Bacula lists.
Apologies in advance if you see this twice.

Our company is about to provide centralised backups for several pools of
backup data of between 1 and 15TB in size. Each pool changes daily but
backups to tape will only occur once a month for each pool.

The backup tape format is to be LT04 and we have a second-hand Dell
PowerVault 124T 16 tape autoloader to work with currently. Backup from a
pool may be taken off a Linux LVM (or hopefully soon a BTRFS) snapshot
ensuring that the source data does not change during the backup process.
We have the possibility of pre-preparing backup or compressed images if
this is advisable.

An important aspect of the system is that the tapes should be readable
for 12 years, by other parties if necessary. From this point of view we
like the idea of providing a CD with each tape set of the software
needed to extract the contents, together with a listing of the enclosed
files in a UTF8 text file. We will be required to audit each backup set
by successfully extracting files from tape.

We are very familiar with working on the command-line in Linux,
Postgresql and Python.

As we have not run backup to tape on Linux before I would be very
grateful to receive advice on what approach members of this list would
take to meeting the above requirements.

Many thanks,
Rory

+--
|This was sent by r...@campbell-lange.net via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--

Re: How to configure Amanda for our backup strategy?

2007-10-01 Thread Jon LaBadie

On Mon, Oct 01, 2007 at 08:50:59AM -0500, [EMAIL PROTECTED] wrote:
> I'm setting up Amanda for the first time and would like to configure
> it for our existing backup strategy.  We want to do complete backups
> nightly during weekdays, i.e. Monday-Friday night.  Once a week, let's
> say Wednesday, we want to take one tape to an off-site vault and bring
> another tape back, meaning we'd be taking a tape out of the normal
> rotation and bringing another one back into the rotation.  We'd want
> to have a minimum of 4 tapes in rotation (3 here and 1 off-site),
> possibly a few more, but not a huge number of tapes.  All will be
> complete backups, no incrementals.
> 
> What parameters would work best in Amanda for doing this with a
> minimal need for manual "adjustments" during the week?  Is it possible
> to just take a tape out of the rotation for a week and give Amanda a
> different tape than it expects, or does a person have to tell Amanda
> they've done this each time they do it?
> 
> Thanks for any help or tips anyone might have!

Set your dumpcycle and runspercycle to 1.  That will do full dumps
each run.  Set your tapecycle to 1 or 2.  If set to 1, amanda will
use any available tape.  If set to 2 it will not reuse the most
recently used tape.  But all the others will be usable.  If the
tape it would prefer to use is off-site, it will use one of the
other available tapes.  You may get some messages about not being
able to use the tape it wants.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

How to configure Amanda for our backup strategy?

2007-10-01 Thread DavidG

I'm setting up Amanda for the first time and would like to configure
it for our existing backup strategy.  We want to do complete backups
nightly during weekdays, i.e. Monday-Friday night.  Once a week, let's
say Wednesday, we want to take one tape to an off-site vault and bring
another tape back, meaning we'd be taking a tape out of the normal
rotation and bringing another one back into the rotation.  We'd want
to have a minimum of 4 tapes in rotation (3 here and 1 off-site),
possibly a few more, but not a huge number of tapes.  All will be
complete backups, no incrementals.

What parameters would work best in Amanda for doing this with a
minimal need for manual "adjustments" during the week?  Is it possible
to just take a tape out of the rotation for a week and give Amanda a
different tape than it expects, or does a person have to tell Amanda
they've done this each time they do it?

Thanks for any help or tips anyone might have!

David

Advice for setup and backup strategy needed

2006-12-01 Thread Markus Koppenberger

Hi,

I'm quite new to amanda but as far as I have read about it and after
some first tries, it seems that it could do the job for our needs, but I
would need some advice for a backup strategy and for how to setup up amanda

I will describe you our current situation and our needs as well as my
thoughts of how to do it.

Every help is very apreciated!


We are planing to change from hard-disk backup (mirroring) to use amanda
to backup our RAID storage(s) with a total of 9 TB available on a 8x
LTO-2 (200GB) tape changer.

Currently about 4.8 TB of total storage space on three partitions is used:
  - 3.0T on a 3.3T RAID,
  - 1.4T on a 1.9T RAID and
  - 0.4T on another 1.9TB RAID.

Most of the stored data will not change over time, but the total amount
of data will (slowly) grow.

So what we would need is an initial full backup followed by regular
(maybe daily) incremental backups (which will be quite small)

Of course it is possible to split up the backup into smaller chunks
(single directories, not 3.3 T at once) but most of these smaller chunks
still would be greater then the 200G tapes we have and therefor I would
use tape-spanning.

Most of these directories have the following structure:

- directory1
   +- bigdata (> 200 GB, but does not change [often])
   +- otherdata (smaller [<< 200 GB], but changes more often)
- directory2
   +- bigdata
   +- otherdata
- ...


This could maybe used to define different backup-set:
- set(s): bigdata:
contains the 'bigdata' subdirectory of one (ore more,
depending on total size) directories be run maybe manually when
new files are added.
- set(s): otherdata:
contains the 'otherdata' sub-directories of various directories and
which will be run on daily basis with maybe monthly or two-weekly
full backup

Is this possible and does it make sense to split up the directories on
sub-directory basis into two (well actually more) backup sets ('big' and
'other')?

And if this would be the right way to do it, how would I configure the
different sets?

Maybe you could give me a general hint on how to use tape-spanning,
especially how to configure the 'runtapes' parameter.
Am I right that I have to configure it that 'runtapes' * 'size-of-tape'
= 'total_size_to_backup'? So, that when I have to backup let's say 1TB I
would need a minimum of runtapes=5 that the whole (full) backup fits on
the tapes ( 5 x 200GB)?

Thanks in advance,
koppi

-- 
Grupo de Tecnologia Musical (MTG)
Universidad Pompeu Fabra (UPF), Barcelona, Espana
http://www.iua.upf.es/mtg/



signature.asc
Description: OpenPGP digital signature

Re: HD backup strategy ?

2005-08-17 Thread Mike Delaney

On Wed, Aug 17, 2005 at 03:03:59PM +0300, Toomas Aas wrote:
> Jon LaBadie wrote:
> 
> >Haven't seen anyone on the list mention using it, but Iomega
> >introduced some interesting hardware last year.  I think they
> >call it "Rev", basically a small, removalble hard drive
> >cartridge.  Think high capacity, tiny zip drive as it has
> >35GB native capacity and a builtin compressor.  Along with
> >their single slot device, they also introduce an autoloader
> >that can hold 10 cartridges.
> 
> Funny, I just stumbled upon this on another mailing list yesterday. 
> Looks like IOMega continues to live up to its reputation:
> 
> http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

Or someone didn't look hard enough.  While most systems device drivers
do detect REV drives as CD-ROMs, that doesn't prevent one from writing
to it.  And while the filesystem format isn't ISO-9660, it's not some
Iomega proprietary thing either: it's UDF.

Basically, from the system's perspective REV drives look and act like
high capacity DVD-RAM units.  I've been using one with SuSE Linux for
several months now.

Re: HD backup strategy ?

2005-08-17 Thread Jon LaBadie

On Wed, Aug 17, 2005 at 03:03:59PM +0300, Toomas Aas wrote:
> Jon LaBadie wrote:
> 
> >Haven't seen anyone on the list mention using it, but Iomega
> >introduced some interesting hardware last year.  I think they
> >call it "Rev", basically a small, removalble hard drive
> >cartridge.  Think high capacity, tiny zip drive as it has
> >35GB native capacity and a builtin compressor.  Along with
> >their single slot device, they also introduce an autoloader
> >that can hold 10 cartridges.
> 
> Funny, I just stumbled upon this on another mailing list yesterday. 
> Looks like IOMega continues to live up to its reputation:
> 
> http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

A peek at the IOMygosh website confirms the articles comment that
it is windows only!  Time for me to stop even thinking of it as a
potential amanda backup device.

jl
-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

2005-08-17 Thread Toomas Aas


Jon LaBadie wrote:


Haven't seen anyone on the list mention using it, but Iomega
introduced some interesting hardware last year.  I think they
call it "Rev", basically a small, removalble hard drive
cartridge.  Think high capacity, tiny zip drive as it has
35GB native capacity and a builtin compressor.  Along with
their single slot device, they also introduce an autoloader
that can hold 10 cartridges.


Funny, I just stumbled upon this on another mailing list yesterday. 
Looks like IOMega continues to live up to its reputation:


http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

--
Toomas Aas 
|arvutivõrgu peaspetsialist | head specialist on computer networks|
|Tartu Linnakantselei   | Tartu City Office   |
- +372 736 1274

Re: HD backup strategy ?

2005-08-16 Thread Jon LaBadie

On Tue, Aug 16, 2005 at 01:02:33PM -0400, Joshua Baker-LePain wrote:
> On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote
> 
> > On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > > Hello everybody,
> > > 
> > > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > > thing it's time to switch for HD backups.
> > > 
> > > Does some of you using HD backup ? if so what is you stategy ?
> > > How you take backup off-site ?
> > > 
> > > What kind of hardware are you using ? SCSI or SATA RAID ?
> > 
> > Haven't seen anyone on the list mention using it, but Iomega
> > introduced some interesting hardware last year.  I think they
> > call it "Rev", basically a small, removalble hard drive
> > cartridge.  Think high capacity, tiny zip drive as it has
> > 35GB native capacity and a builtin compressor.  Along with
> > their single slot device, they also introduce an autoloader
> > that can hold 10 cartridges.
> > 
> > If the capacity is reasonable for your needs you might have
> > your HD backup and off-site capability combined.
> 
> Why does the thought of trusting my backups to anything made by Iomega 
> scare the living [EMAIL PROTECTED] out of me?
> 

That is why I made sure to mention the manufacturer.
But it is a neat concept.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

2005-08-16 Thread Tobias Bluhm

Joshua Baker-LePain <[EMAIL PROTECTED]> 08/16/2005 01:02:33 PM:

> On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote
> 
> > On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > > Hello everybody,
> > > 
> > > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > > thing it's time to switch for HD backups.
> > > 
> > > Does some of you using HD backup ? if so what is you stategy ?
> > > How you take backup off-site ?
> > > 
> > > What kind of hardware are you using ? SCSI or SATA RAID ?
> > 
> > Haven't seen anyone on the list mention using it, but Iomega
> > introduced some interesting hardware last year.  I think they
> > call it "Rev", basically a small, removalble hard drive
> > cartridge.  Think high capacity, tiny zip drive as it has
> > 35GB native capacity and a builtin compressor.  Along with
> > their single slot device, they also introduce an autoloader
> > that can hold 10 cartridges.
> > 
> > If the capacity is reasonable for your needs you might have
> > your HD backup and off-site capability combined.
> 
> Why does the thought of trusting my backups to anything made by Iomega 
> scare the living [EMAIL PROTECTED] out of me?


Hehe - you're not thinking of the i'll-eat-your-disk-then-die zip-100, are 
you?


-
toby bluhm
philips medical systems, cleveland ohio
[EMAIL PROTECTED]
440-483-5323

Re: HD backup strategy ?

2005-08-16 Thread Joshua Baker-LePain

On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote

> On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > Hello everybody,
> > 
> > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > thing it's time to switch for HD backups.
> > 
> > Does some of you using HD backup ? if so what is you stategy ?
> > How you take backup off-site ?
> > 
> > What kind of hardware are you using ? SCSI or SATA RAID ?
> 
> Haven't seen anyone on the list mention using it, but Iomega
> introduced some interesting hardware last year.  I think they
> call it "Rev", basically a small, removalble hard drive
> cartridge.  Think high capacity, tiny zip drive as it has
> 35GB native capacity and a builtin compressor.  Along with
> their single slot device, they also introduce an autoloader
> that can hold 10 cartridges.
> 
> If the capacity is reasonable for your needs you might have
> your HD backup and off-site capability combined.

Why does the thought of trusting my backups to anything made by Iomega 
scare the living [EMAIL PROTECTED] out of me?

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: HD backup strategy ?

2005-08-16 Thread Jon LaBadie

On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> Hello everybody,
> 
> My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> thing it's time to switch for HD backups.
> 
> Does some of you using HD backup ? if so what is you stategy ?
> How you take backup off-site ?
> 
> What kind of hardware are you using ? SCSI or SATA RAID ?

Haven't seen anyone on the list mention using it, but Iomega
introduced some interesting hardware last year.  I think they
call it "Rev", basically a small, removalble hard drive
cartridge.  Think high capacity, tiny zip drive as it has
35GB native capacity and a builtin compressor.  Along with
their single slot device, they also introduce an autoloader
that can hold 10 cartridges.

If the capacity is reasonable for your needs you might have
your HD backup and off-site capability combined.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

2005-08-16 Thread Bernhard Ott

FM wrote:
> Hello everybody,
> 
> My ibm 35810 died and I will cost use 2 CA$ to replace it. So I
> thing it's time to switch for HD backups.
> 
> Does some of you using HD backup ? if so what is you stategy ?
> How you take backup off-site ?
> 
> What kind of hardware are you using ? SCSI or SATA RAID ?
> 
> Thanks !
> 
> 
Have a look at backuppc: http://backuppc.sourceforge.net/
The archiving can't be compared to amanda, but still it's quite
powerfull ...
I'm using amanda for off-site backups and backuppc (SATA Raid) to give
users permanent access to their data .

Regards, Bernhard

Re: HD backup strategy ?

2005-08-12 Thread Gene Heskett

On Friday 12 August 2005 09:40, FM wrote:
>Hello everybody,
>
>My ibm 35810 died and I will cost use 2 CA$ to replace it. So I
>thing it's time to switch for HD backups.
>
>Does some of you using HD backup ? if so what is you stategy ?
>How you take backup off-site ?
>
>What kind of hardware are you using ? SCSI or SATA RAID ?
>
>Thanks !

Here, a home site, I'm, using a 200GB atapi drive, with 180GB of it
partitioned for amanda.  Its worked flawlessly for everythng for
about a year. It is not removeable other than by a powerdown & ata133
cable disconnection.  I'd not have an offsite storage place available
anyway.  I'm backing up the working parts of 2 machines & may add a
third if it ever gets to stay on 24/7 like the first 2 do.

Candid opinion:  HD's are far more friendly than tapes when it comes
to recovery operations, and hundreds of times faster for recovery.  I
did not find that tapes (but then I was using DDS2 (spit) tapes too)
were in fact all that dependable, with something getting a tummy ache
at weekly intervals.  I've had no such tummy aches with the HD setup.

BUT:  Quality tapes would have the option of removing for permanent
storage, unlike using a non-removable HD.  If I really thought my
stuff was *that* precious, I'd burn a stack of dvd's once  month. 
But, I'd have to justify the cost of the dvd+(or-)r's as an ongoing
expense, something my SS would have to support as I'm just your
typical old fart, half retired, tv chief engineer whose played with
computers since 1977 & refuses to act my age of 70. 

Trying to stay in shape, and needing to get it done, I've moved about
4 yards of dirt with a shovel and wheelborrow in the last 3 days
with the temps in the high 80's.   And thats burning about 2 lbs a day
off me that I can well afford to lose, and keeping the sugar in check
too.  But when I'm done, it'll be mowable. Nothings flat in West
Virginia unless you "make it so"...

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: HD backup strategy ?

2005-08-12 Thread Toomas Aas


FM wrote:


Does some of you using HD backup ? if so what is you stategy ?
How you take backup off-site ?


At one site, I'm using two external FireWire HDDs (Maxtor 5000DV), 5 
'virtual tapes' on each. One of the HDDs is off-site, every Monday it's 
brought in and the HDD that was used during previus week is then 
disconnected and taken off-site.





---
... How can you tell when you've run out of invisible ink?

HD backup strategy ?

2005-08-12 Thread FM


Hello everybody,

My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
thing it's time to switch for HD backups.


Does some of you using HD backup ? if so what is you stategy ?
How you take backup off-site ?

What kind of hardware are you using ? SCSI or SATA RAID ?

Thanks !

Re: What tapes can I send off-site ? How do I implement this backup strategy ?

2005-05-10 Thread L Sydow

How can I manage to do full backups of every server each FRIDAY ? I
know I should create a new config specifying always full and an
infinite tapecylce and run it on fridays, and run the "regular"
schedule on the other days.
Instead of having cron run amdump directly, you could have it run a 
script like:

set -- `date`
case $1 in
Fri)
su backup /local/sbin/amdump Full
;;
*)
su backup /local/sbin/amdump Daily
;;
esac

Re: What tapes can I send off-site ? How do I implement this backup strategy ?

2005-05-10 Thread Matt Hyclak

On Tue, May 10, 2005 at 03:45:57PM -0400, Guy Dallaire enlightened us:
> We use a nice GUI program to do that, but the product is outdated and
> buying a new licence is too costly. Amanda seems good, and we use more
> and more open source software here so we decided to try it out.
> 
> Our current method of backup is the traditional incrementals Mon, Tue,
> Wed, Thu, and Friday we do a Full backup. No backups on week end.
> 
> The tapes are stored in a locked cabinet outside the server room.
> After 3 "business days" on site, the tapes are sent to a remote
> location.  This permits to have the tape handy if we have to restore a
> file that was modified and deleted in the last 2 days.
> 
> Monthly tapes (Full backup) are kept forever for archiving/financial
> regulation  pupose.
> 
> I'm having a hard time figuring out how to do something similar with amanda.
> 

You'll need to keep "dumpcycle" worth of tapes on site. If you set dumpcycle
to 1 week, and runspercycle to 5 (M-F), then you'll need to keep the latest
5 tapes to be able to do those quick restores. What a lot of people will do
with enough tapes is have 4 "sets" of tapes. In your case you might have
something like the following:

* Tape1, Tape2, Tape3, Tape4, Tape5 - "current" set that backups are performed
  on

* Tape6, Tape7, Tape8, Tape9, Tape10 - "most recent" set from last week,
  stored in your locked cabinet

* Tape11, Tape12, Tape13, Tape14, Tape15 - set stored offsite

* Tape16, Tape17, Tape18, Tape19, Tape20 - in transit back onsite to become
  next weeks "current"

Depending on how accessible your offsite location is, you can probably get
away with 3 "sets"

> I'm not familiar with backup levels. It's a concept I always had a
> difficulty to grasp.  If I send the tapes older than 3 days to the
> remote site, will the tapes of the last three days ALWAYS contain the
> file modified in the last 3 days ?
>

Backup levels work like this:

Level 0 contains every file
Level 1 contains every file that has changed since the last Level 0
Level 2 contains every file that has changed since the last Level 1
Level n contains every file that has changed since the last Level n-1

For a simple example, let's consider some directory "foo". "foo" contains the
files "a", "b", "c", and "d".

* Monday - Level 0 - contains a,b,c and d
- User edits file b
* Tuesday - Level 1 - contains b
- User edits file c
* Wednesday - Level 1 - contains b and c
- User edits file a
* Thursday - Level 2 - contains a

So the idea is that if you need to restore an entire directory, you use the
latest-dated of each level, in this case Monday, Wednesday and Thursday's
tapes.

Amanda uses the bump* variables to decide when to increase to the next
level. Above it seems that the level was bumped arbitrarily from 1 to 2, but
what happened for example was that b and c were both extremely large files,
and a was very small. It wouldn't make much sense to use tape for those
large files when the only change is the small "a" file, so it bumps the
level up.

> How can I manage to do full backups of every server each FRIDAY ? I
> know I should create a new config specifying always full and an
> infinite tapecylce and run it on fridays, and run the "regular"
> schedule on the other days.
>

It is generally not recommended to do so. Why do you feel it necessary to
only do full dumps on Friday? Let amanda even out the schedule and you will
find good usage accross tapes, and fairly consistent backup runtimes.

> One last question: Is there an easy way with amanda to do an "ad hoc"
> backup ? I mean, say I have to backup a group of directories on a
> server, do I have to create a new config, a new disklist, etc... ?
> There must be an easier way ?
> 

The host would have to be in the disklist, but you can do individual
backups. amdump takes optional host and disk parameters, so if a machine was
offline during the backup, the next morning you can bring it up and run
something like:

amdump CONFIGNAME missinghost /backupdir

> I'm reading the documentation, but there is a lot of doc, and not
> enough samples and use-cases IMHO.
> 

I hope my examples helped.

Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263


pgpuEy3ZZ3Uuh.pgp
Description: PGP signature

What tapes can I send off-site ? How do I implement this backup strategy ?

2005-05-10 Thread Guy Dallaire

We use a nice GUI program to do that, but the product is outdated and
buying a new licence is too costly. Amanda seems good, and we use more
and more open source software here so we decided to try it out.

Our current method of backup is the traditional incrementals Mon, Tue,
Wed, Thu, and Friday we do a Full backup. No backups on week end.

The tapes are stored in a locked cabinet outside the server room.
After 3 "business days" on site, the tapes are sent to a remote
location.  This permits to have the tape handy if we have to restore a
file that was modified and deleted in the last 2 days.

Monthly tapes (Full backup) are kept forever for archiving/financial
regulation  pupose.

I'm having a hard time figuring out how to do something similar with amanda.

I'm not familiar with backup levels. It's a concept I always had a
difficulty to grasp.  If I send the tapes older than 3 days to the
remote site, will the tapes of the last three days ALWAYS contain the
file modified in the last 3 days ?

How can I manage to do full backups of every server each FRIDAY ? I
know I should create a new config specifying always full and an
infinite tapecylce and run it on fridays, and run the "regular"
schedule on the other days.

One last question: Is there an easy way with amanda to do an "ad hoc"
backup ? I mean, say I have to backup a group of directories on a
server, do I have to create a new config, a new disklist, etc... ?
There must be an easier way ?

I'm reading the documentation, but there is a lot of doc, and not
enough samples and use-cases IMHO.

Thanks

Re: Determining appropriate backup strategy

2005-04-01 Thread Joshua Baker-LePain

On Thu, 31 Mar 2005 at 5:24pm, Vicki Stanfield wrote

> I think what I am really talking about is the dumptype. I have the ones 
> I listed defined. I think I might be confusing them with priority. I 
> suppose the dumptype could only mean level of compression. But then I 
> need more information about how Amanda decides what to back up when. 
> I'll look back to the amanda.org site.

Every DLE is backed up every night.  The level depends not on the dumptype 
(unless you've specified always-full or something like that) but on 
dumpcycle and runspercycle.

dumpcycle=max amount of time before full (level 0) backups
runspercycle=number of 'amdump' runs per dumpcycle

You'll also need to coordinate these with tapecycle, which is the number 
of tapes you'll let amanda use.  Generally you'll want 
tapecycle>=1.5*runspercycle for error protection.  Note also that all of 
the above assumes runtapes=1.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Determining appropriate backup strategy

2005-03-31 Thread Jon LaBadie

On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> I am using amanda to back up my servers. I have the following situation:
> 
> There is about 42G of data to be backed up on one particular machine. 
> This data is mostly static data in a directory structure which creates a 
> new directory whenever 1000 files are in the current directory and 
> numbers the new directory incrementally.  It looks like this:
> 
> /project/newarchive/0 contains files 0-999
> /project/newarchive/1 contains files 1000-1999
> /project/newarchive/1 contains files 2000-2999
> etc.
> 
> The previous directories are necessary for the program to run 
> successfully but will remain static once the next directory is created. 
> I need a backup solution that is appropriate for this. What is 
> appropriate for this situation?

Wouldn't the standard solution be appropriate here?

I presume you do want an occassional full backup of even the static data.
For the incrementals, only things which change (I guess files added) are
backed up.  As long as the files haven't changed since the last higher
level dump (lower numerically) then those files will not appear in the
incrementals.

You may want to play with bumpdays and bumpmultiplier to ensure you get
several dump levels between level 0 dumps.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Determining appropriate backup strategy

2005-03-31 Thread Jon LaBadie

On Thu, Mar 31, 2005 at 05:24:43PM -0500, Vicki Stanfield wrote:
> Jon LaBadie wrote:
> 
> >On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> > 
> >
> >>P.S. I think I asked before but didn't get an answer. Is there a source 
> >>for an explanation of the different priorities of backups. I have the 
> >>following defined from an inherited amanda.conf file:
> >>
> >>always-full (obvious)
> >>comp-low-tar
> >>comp-med-tar
> >>comp-high-tar
> >>low-tar
> >>med-tar
> >>high-tar
> >>comp-low-dump
> >>comp-med-dump
> >>comp-high-dump
> >>low-dump
> >>med-dump
> >>high-dump
> >>
> >>The explanations in the amanda.conf are insufficient. I need to know 
> >>what decisions amanda makes when determining how often to back up each 
> >>dumptype.
> >>   
> >>
> >
> >A good additional place to read is the man pages.
> >Under priority this appears:
> >
> >priority "string"
> > Default:  medium.  When there is no tape to  write  to,
> > Amanda will do incremental backups in priority order to
> > the  holding  disk.   The  priority  may  be   high(2),
> > medium(1), low(0) or a number of your choice.
> >
> >So the difference appears important only when saving ONLY to holding disk.
> >Then you can define which come earlier or later incase the holding disk
> >fills.
> >
> > 
> >
> I think what I am really talking about is the dumptype. I have the ones 
> I listed defined. I think I might be confusing them with priority. I 
> suppose the dumptype could only mean level of compression. But then I 
> need more information about how Amanda decides what to back up when. 
> I'll look back to the amanda.org site.
> 

Set one of your DLE's to the dumptype.  Maybe even a dummy DLE.
Then run amadmin ... disklist ... command to see the relevant
parameters.  Change the dumptype and rerun amadmin to compare.
A sample run:

  # amadmin DS1 disklist butch /var
  line 10:
host butch:
interface default
disk /var:
program "GNUTAR"
exclude list "/usr/local/etc/amanda/exclude.gtar"
priority 0
dumpcycle 7
maxdumps 8
maxpromoteday 1
strategy STANDARD
compress CLIENT FAST
comprate 0.50 0.50
auth BSD
kencrypt NO
holdingdisk YES
record YES
index YES
skip-incr NO
skip-full NO

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Determining appropriate backup strategy

2005-03-31 Thread Vicki Stanfield

Jon LaBadie wrote:
On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
 

P.S. I think I asked before but didn't get an answer. Is there a source 
for an explanation of the different priorities of backups. I have the 
following defined from an inherited amanda.conf file:

always-full (obvious)
comp-low-tar
comp-med-tar
comp-high-tar
low-tar
med-tar
high-tar
comp-low-dump
comp-med-dump
comp-high-dump
low-dump
med-dump
high-dump
The explanations in the amanda.conf are insufficient. I need to know 
what decisions amanda makes when determining how often to back up each 
dumptype.
   

A good additional place to read is the man pages.
Under priority this appears:
priority "string"
 Default:  medium.  When there is no tape to  write  to,
 Amanda will do incremental backups in priority order to
 the  holding  disk.   The  priority  may  be   high(2),
 medium(1), low(0) or a number of your choice.
So the difference appears important only when saving ONLY to holding disk.
Then you can define which come earlier or later incase the holding disk
fills.
 

I think what I am really talking about is the dumptype. I have the ones 
I listed defined. I think I might be confusing them with priority. I 
suppose the dumptype could only mean level of compression. But then I 
need more information about how Amanda decides what to back up when. 
I'll look back to the amanda.org site.

Vicki

Re: Determining appropriate backup strategy

2005-03-31 Thread Jon LaBadie

On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> 
> P.S. I think I asked before but didn't get an answer. Is there a source 
> for an explanation of the different priorities of backups. I have the 
> following defined from an inherited amanda.conf file:
> 
> always-full (obvious)
> comp-low-tar
> comp-med-tar
> comp-high-tar
> low-tar
> med-tar
> high-tar
> comp-low-dump
> comp-med-dump
> comp-high-dump
> low-dump
> med-dump
> high-dump
> 
> The explanations in the amanda.conf are insufficient. I need to know 
> what decisions amanda makes when determining how often to back up each 
> dumptype.

A good additional place to read is the man pages.
Under priority this appears:

 priority "string"
  Default:  medium.  When there is no tape to  write  to,
  Amanda will do incremental backups in priority order to
  the  holding  disk.   The  priority  may  be   high(2),
  medium(1), low(0) or a number of your choice.

So the difference appears important only when saving ONLY to holding disk.
Then you can define which come earlier or later incase the holding disk
fills.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Determining appropriate backup strategy

2005-03-31 Thread Vicki Stanfield

I am using amanda to back up my servers. I have the following situation:
There is about 42G of data to be backed up on one particular machine. 
This data is mostly static data in a directory structure which creates a 
new directory whenever 1000 files are in the current directory and 
numbers the new directory incrementally.  It looks like this:

/project/newarchive/0 contains files 0-999
/project/newarchive/1 contains files 1000-1999
/project/newarchive/1 contains files 2000-2999
etc.
The previous directories are necessary for the program to run 
successfully but will remain static once the next directory is created. 
I need a backup solution that is appropriate for this. What is 
appropriate for this situation?

Vicki
P.S. I think I asked before but didn't get an answer. Is there a source 
for an explanation of the different priorities of backups. I have the 
following defined from an inherited amanda.conf file:

always-full (obvious)
comp-low-tar
comp-med-tar
comp-high-tar
low-tar
med-tar
high-tar
comp-low-dump
comp-med-dump
comp-high-dump
low-dump
med-dump
high-dump
The explanations in the amanda.conf are insufficient. I need to know 
what decisions amanda makes when determining how often to back up each 
dumptype.

Re: Laptop Backup Strategy

2004-10-27 Thread Jonathan Dill

I'm beginning to investigate using external USB 2.0 / Firewire drives to 
do backups of some systems.  The idea is that we could keep 2-3 spare 
PCs around, then if your computer is toast, we just ship it out for 
repairs, plug your external drive into one of the spare PCs, and rebuild 
the system from the external drive.

--jonathan

Re: Laptop Backup Strategy

2004-10-27 Thread Geert Uytterhoeven

On Wed, 27 Oct 2004, Richard Karnesky wrote:
> 3. Force users to backup to a server

If you have sufficient spare diskspace, let the user rsync from time to time to
this space. Then let Amanda backup this space, together with the other machines
that are always available.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Laptop Backup Strategy

2004-10-27 Thread Richard Karnesky

I'd like to add several laptops to my current configuration.  I was 
wondering what solutions people used for "transient" hosts.  Ideally, the 
main backup configuration (which handles workstations) would run in the 
evening.  The downside of this is that laptops are VERY rarely (if ever) on 
the network at this time.  So some of the possible solutions:

1. Force users to occasionally leave their laptops on the network over 
night.  Probably not going to happen.

2. Setup a second config, to be executed for only transient hosts during 
the work day.
Is there a way to keep this on a holding disk & trick amanda into dumping 
it with the first config, so that I only use a single set of tapes?  We 
have tapes that are somewhat large & somewhat expensive & it would be 
overly wastefull to use 2 tapes/day when 1 could be sufficient.  It would 
also be a shame to maintain two indexes & figure out which to restore from.

3. Force users to backup to a server
What do Windows & OS X users use for this?  BackupPC?  A tar or some 
proprietary backup format saved to a samba share?

I was wondering which of these solutions (or others that I didn't think of 
and couldn't find in the list archives) people use & if there is any 
additional advice that could be useful.

Thanks,
Richard Karnesky

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

2004-04-06 Thread pll+amanda


In a message dated: Tue, 06 Apr 2004 12:12:05 BST
"William Hargrove" said:

>I was thinking I could do the incremental backups to a hard disk area on the tape
>server each night and then run Amanda once a week to archive the full backups plus
>the HDD incremental ones.

I did this once, way back when all I had was a single DLT drive and 
no autoloader.  I would just leave the drive empty all week, and on 
Friday mornings I would pop that week's tape in then amflush 
everything to tape.  When that was done, I would place the weekend 
tape in the drive and the full dumps on Saturday night would go 
direct to tape, or, if I forgot, end up on the holding disk for 
amflushing Monday morning.

This worked quite well IMO for the small environment I had at the 
time.

HTH,
-- 
Seeya,
Paul

GPG Key fingerprint = 1660 FECC 5D21 D286 F853  E808 BB07 9239 53F1 28EE

 If you're not having fun, you're not doing it right!

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

2004-04-06 Thread Jonathan Dill

Here's a strategy that I implemented about a month ago that is working 
pretty well so far:

1. run amdump every night to large RAID w/o tape, mix of full and incr
2. run script to separate images to amanda-incr and amanda-full
3. when amanda-full exceeds size of tape, run amflush
4. when RAID approaches capacity, flush some incrs to tape
Since it's a RAID, there is less chance it will go to crap than a 
regular disk partition, but it is still possible.  And if it does, I 
still have the full dumps on tape to fall back on.

To give you some gauge of capacities, I'm backing up about 300 GB of 
disks, have ~600 GB RAID5 (on a Snap 4500) and am using 100/200 GB LTO 
(Ultrium-1) tapes.  So far, it's mostly web servers, but I haven't moved 
everything to the new backup cycle yet.  I have flushed 2 tapes worth of 
full dumps, and accumulated about 167 GB of incrementals that are still 
on the RAID.

The goal and idea is that > 90% of the time, I will only need to load 
one tape to do a restore, and the rest will come out of holding files.  
Even better, there is a good chance that the tape that I need will be 
the last one that I flushed, so I won't have to change the tape.

/snap/amanda-hd is the directory that amanda.conf looks for, which is 
actually a symbolic link to /snap/amanda-full.  Therefore, I just move 
the incremental dumps from amanda-full to amanda-incr, both actually on 
the same filesystem (a simple directory rename in the filesystem inode 
table).  If I need to flush incrementals, I just change the link 
amanda-hd -> amanda-incr and run the flush (this could be done more 
elegantly with a patch, but this is so easy, so why bother?)  In the 
cron job, I make sure amanda-hd -> amanda-full before amdump runs.

I'm using NFSv3 to XFS filesystem through a dedicated 100baseTX FD 
connection, so I'm using a chunksize of ~600 GB, and that makes the 
rename script very simple since I don't have to worry about any backup 
getting split into chunks.  AFAIK NFSv3 and XFS both support 64-bit 
addressing, so I shouldn't run into any filesize limits.

--jonathan

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

2004-04-06 Thread Jon LaBadie

On Tue, Apr 06, 2004 at 12:12:05PM +0100, William Hargrove wrote:
> Hello All,
> 
> I would like to ask a general Amanda backup 'strategy' question here, given
> by backup 'goals' below. I have a configured and working Amanda set-up but I
> want to try and fit a backup policy that I believe is best suited to our
> site around Amanda.
> 
> I have an 8-tape (DLT 35/70gb each) auto changer library linked to Amanda
> via mtx, this works well.
> 
> I want to do a full backup of various file systems on a remote host once a
> week and then, ideally, incremental backups once a day to that same tape. At
> the end of a week's backup, the tape should switch. One week's backup is far
> less than a single tape's capacity.
> 
> I would like to be able to hold 8 weeks worth of backups on tape, that is
> one tape a week and then reuse the oldest tape.
> 
> I'm not sure this is possible with Amanda.. as I believe I can't append the
> daily incremental backups onto the same tape as the full backups. Is it the
> case that if I want to run Amanda every day that I would also have to use a
> tape every day, thus meaning the tapes in the library are all used after a
> week. I'm fixed in the number of tapes I can use and can't add/replace them,
> so they would get reused after a week and wouldn't provide much of a 'backup
> horizon'.
> 
> If this isn't possible does anyone have some suggestions? I was thinking I
> could do the incremental backups to a hard disk area on the tape server each
> night and then run Amanda once a week to archive the full backups plus the
> HDD incremental ones.
> 


Where is that "Top Ten Questions" document when you need it?

Amanda will not reuse a tape for a new run of amdump, fulls or
incrementals, until it cycles through "tapecycle" number of
tapes.  So you are correct, amanda will NOT append a second
amdump to a tape.  Amanda only overwrites tapes!

However, an amflush can overwrite a tape with multiple dumps
from the holding disk.  Thus collect as many amdumps as you
want with the tape offline.  Then amflush when you want to
tape them.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Amanda backup strategy thoughts, full and incremental backups to tape once a week.

2004-04-06 Thread William Hargrove

Hello All,

I would like to ask a general Amanda backup 'strategy' question here, given
by backup 'goals' below. I have a configured and working Amanda set-up but I
want to try and fit a backup policy that I believe is best suited to our
site around Amanda.

I have an 8-tape (DLT 35/70gb each) auto changer library linked to Amanda
via mtx, this works well.

I want to do a full backup of various file systems on a remote host once a
week and then, ideally, incremental backups once a day to that same tape. At
the end of a week's backup, the tape should switch. One week's backup is far
less than a single tape's capacity.

I would like to be able to hold 8 weeks worth of backups on tape, that is
one tape a week and then reuse the oldest tape.

I'm not sure this is possible with Amanda.. as I believe I can't append the
daily incremental backups onto the same tape as the full backups. Is it the
case that if I want to run Amanda every day that I would also have to use a
tape every day, thus meaning the tapes in the library are all used after a
week. I'm fixed in the number of tapes I can use and can't add/replace them,
so they would get reused after a week and wouldn't provide much of a 'backup
horizon'.

If this isn't possible does anyone have some suggestions? I was thinking I
could do the incremental backups to a hard disk area on the tape server each
night and then run Amanda once a week to archive the full backups plus the
HDD incremental ones.

Many thanks in advance for everyone's thoughts.

William.

Re: Amanda Backup Strategy

2003-02-28 Thread Jon LaBadie

On Wed, Feb 26, 2003 at 11:05:44AM -0500, Ralph Bearpark wrote:
> > Remember, you can always split up your 
> > filesystems into multiple disklist entries using tar.  Then you could 
> > reduce your tapelength to have *just enough* to fit everything once.
> 
> Hmm, dividing my clients into multiple DLEs that I hope will never exceed 
> 600MB ... that's an ugly concept.

I realize that 600MB is 18GB/30 days.  But you needed not set the tapelength
to 600MB.  'tapelength' is not a figure amanda tries to fill.  It is how
much it has to work with.  My tapelength is 12GB.  I average 5-6GB nightly.
But some nights it goes as low as 3GB and the occasional 10GB is seen.

Set your tapelength to some reasonable value, big enough for your large DLE's
plus some for incrementals.

As to those DLE's getting level 0's every other day;  I only recall similar
behavior under two conditions.  If the DLE is very active and level 0 or
level 1 would make little difference, or if the DLE is small and is one
that is promoted quite often to aid "future balancing" of large DLE's.
Because it is a small DLE, the impact is small also.  Is the behavior
of your frequently backed up DLE different?

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Amanda Backup Strategy

2003-02-26 Thread Gene Heskett

On Wed February 26 2003 17:22, Seth, Wayne (Contractor) wrote:
>Someone recently recommended "Unix Backup & Recovery" by W. Curtis
> Preston to me.  I am just now wading through the amanda section
> so I cannot tell you if it's any good.  But, there certainly is a
> lot of material there!

Yes, and your quoting is broken, making it look as if I wrote both 
you message above, and my message below.  Please adjust your email 
agent.

>-Original Message-
From: Gene Heskett [mailto:[EMAIL PROTECTED]
>Sent: Wednesday, February 26, 2003 12:29 PM
>To: Ralph Bearpark; [EMAIL PROTECTED]
>Cc: [EMAIL PROTECTED]
>Subject: Re: Amanda Backup Strategy
>
>On Wed February 26 2003 11:16, Ralph Bearpark wrote:
>>> Yep.  Amanda gets estimates for all the DLEs at multiple
>>> incremental levels, and figures out the "best" thing to do
>>> within the tapelength it has.
>>
>>Even to the extent of not doing any sort of backup at all on some
>> or many of the DLEs?  If so, how on earth does it decide which?
>
>Amanda shouldn't have to deal with that, so the normal "startup"
>procedure is to only give amanda enough acive DLE's to fill a tape
>each night until all have been done, thereby giving amanda some
>help on the initial "balance" settings.  But, if the tapetype is
>properly set, I don't *think* amanda will take a chance on hitting
>EOT as it will delay enough to assure it won't hit EOT.  I've seen
>that in my email from amanda a time or 2.
>
>>(If this is all documented somewhere, please tell me where and
>> I'll stop asking my stupid questions! :-))
>
>The only really stupid question is the one not asked. ;-)

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.23% setiathome rank, not too shabby for a WV hillbilly

RE: Amanda Backup Strategy

2003-02-26 Thread Seth, Wayne (Contractor)

Someone recently recommended "Unix Backup & Recovery" by W. Curtis Preston
to me.  I am just now wading through the amanda section so I cannot tell you
if it's any good.  But, there certainly is a lot of material there!

-Original Message-
From: Gene Heskett [mailto:[EMAIL PROTECTED]
Sent: Wednesday, February 26, 2003 12:29 PM
To: Ralph Bearpark; [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Amanda Backup Strategy

On Wed February 26 2003 11:16, Ralph Bearpark wrote:
>> Yep.  Amanda gets estimates for all the DLEs at multiple
>> incremental levels, and figures out the "best" thing to do
>> within the tapelength it has.
>
>Even to the extent of not doing any sort of backup at all on some
> or many of the DLEs?  If so, how on earth does it decide which?

Amanda shouldn't have to deal with that, so the normal "startup" 
procedure is to only give amanda enough acive DLE's to fill a tape 
each night until all have been done, thereby giving amanda some 
help on the initial "balance" settings.  But, if the tapetype is 
properly set, I don't *think* amanda will take a chance on hitting 
EOT as it will delay enough to assure it won't hit EOT.  I've seen 
that in my email from amanda a time or 2.

>(If this is all documented somewhere, please tell me where and
> I'll stop asking my stupid questions! :-))

The only really stupid question is the one not asked. ;-)

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.23% setiathome rank, not too shabby for a WV hillbilly

Re: Amanda Backup Strategy

2003-02-26 Thread Gene Heskett

On Wed February 26 2003 11:16, Ralph Bearpark wrote:
>> Yep.  Amanda gets estimates for all the DLEs at multiple
>> incremental levels, and figures out the "best" thing to do
>> within the tapelength it has.
>
>Even to the extent of not doing any sort of backup at all on some
> or many of the DLEs?  If so, how on earth does it decide which?

Amanda shouldn't have to deal with that, so the normal "startup" 
procedure is to only give amanda enough acive DLE's to fill a tape 
each night until all have been done, thereby giving amanda some 
help on the initial "balance" settings.  But, if the tapetype is 
properly set, I don't *think* amanda will take a chance on hitting 
EOT as it will delay enough to assure it won't hit EOT.  I've seen 
that in my email from amanda a time or 2.

>(If this is all documented somewhere, please tell me where and
> I'll stop asking my stupid questions! :-))

The only really stupid question is the one not asked. ;-)

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.23% setiathome rank, not too shabby for a WV hillbilly

RE: Amanda Backup Strategy

> Yep.  Amanda gets estimates for all the DLEs at multiple incremental 
> levels, and figures out the "best" thing to do within the 
> tapelength it has.

Even to the extent of not doing any sort of backup at all on some or many
of the DLEs?  If so, how on earth does it decide which?

(If this is all documented somewhere, please tell me where and I'll stop
asking my stupid questions! :-))

Regards, Ralph.

Re: Amanda Backup Strategy

> Remember, you can always split up your 
> filesystems into multiple disklist entries using tar.  Then you could 
> reduce your tapelength to have *just enough* to fit everything once.

Hmm, dividing my clients into multiple DLEs that I hope will never exceed 
600MB ... that's an ugly concept.

I assume that Amanda is clever enough to not try to put, say, two 500MB 
DLE backups on a single 600MB tape?

Regards, Ralph.

RE: Amanda Backup Strategy

On Wed, 26 Feb 2003 at 11:16am, Ralph Bearpark wrote

> > Yep.  Amanda gets estimates for all the DLEs at multiple incremental 
> > levels, and figures out the "best" thing to do within the 
> > tapelength it has.
> 
> Even to the extent of not doing any sort of backup at all on some or many
> of the DLEs?  If so, how on earth does it decide which?

Yep.  But, when it does that, you'll get big fat warnings such as:

FAILURE AND STRANGE DUMP SUMMARY:
  $CLIENT $DISK lev 1 FAILED [dumps way too big, must skip incremental 
dumps]

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Amanda Backup Strategy

On Wed, 26 Feb 2003 at 11:05am, Ralph Bearpark wrote

> > Remember, you can always split up your 
> > filesystems into multiple disklist entries using tar.  Then you could 
> > reduce your tapelength to have *just enough* to fit everything once.
> 
> Hmm, dividing my clients into multiple DLEs that I hope will never exceed 
> 600MB ... that's an ugly concept.
> 
> I assume that Amanda is clever enough to not try to put, say, two 500MB 
> DLE backups on a single 600MB tape?

Yep.  Amanda gets estimates for all the DLEs at multiple incremental 
levels, and figures out the "best" thing to do within the tapelength it 
has.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Amanda Backup Strategy

On Wed, 26 Feb 2003 at 10:33am, Ralph Bearpark wrote

> > Not that I know of, but in the 2.4.4 ReleaseNotes it says 
> > that it tries to 
> > reduce the number of level 0s per host each night.
> 
> I guess that *might* help.  I could probably get it down to 
> a full backup every week.  Would I just need 2.4.4 on the 
> server?  Or the clients too?

Just on the server -- that's the job of planner.

> > If you want amanda to use less disk space, crank down your 
> > 'tapelength'.  
> > Since you're using diskless backup, you can define whatever 
> > size "tapes" 
> > you want.
> 
> True, but I have 18GB over 30 tapes.  So, to be completely 
> safe from Amanda filling the disk with unnecessary full 
> backups, I'd have to set the tapelength to 600MB, which, 
> unfortunately, wouldn't be enough for any single client full 
> backup.

Single *client* or single *disk*?  Remember, you can always split up your 
filesystems into multiple disklist entries using tar.  Then you could 
reduce your tapelength to have *just enough* to fit everything once.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Amanda Backup Strategy

Linebreaks, man, use some line breaks...

On Wed, 26 Feb 2003 at 4:10am, Ralph Bearpark wrote

> However, when I look an amoverview I see that several disks are getting 
> a full backup EVERY OTHER DAY!  What's the idea of this?  I'll grant 
> it's within the letter of the manual, but it's hardly space-efficient is 
> it?

The goal of amanda is to equalize the amount of data backed up each night 
while staying within the bounds set by dumpcycle, runspercycle, and 
tapecycle.  If this means it promotes dumps, it does it.

> Is there any way I can force amanda to only do the full backup at most 
> n times during the dumpcycle?

Not that I know of, but in the 2.4.4 ReleaseNotes it says that it tries to 
reduce the number of level 0s per host each night.

> Also, I see that other disks are getting repeated level 1 incrementals.  
> I would be happier if it rather did level 1 then 2 then 3 etc. and no 
> backup if no files had changed.  Can I specify this behaviour somehow?

You can play with bumpsize and bumpmult, but the extra hassle restoring 
really isn't worth it, IMO.

> I know it will make a full restore a little messier but I simply don't 
> have unlimited disk space for amanda to use.
> 
If you want amanda to use less disk space, crank down your 'tapelength'.  
Since you're using diskless backup, you can define whatever size "tapes" 
you want.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Amanda Backup Strategy

> Linebreaks, man, use some line breaks...

Sorry, dude, sorry...

> Not that I know of, but in the 2.4.4 ReleaseNotes it says 
> that it tries to 
> reduce the number of level 0s per host each night.

I guess that *might* help.  I could probably get it down to 
a full backup every week.  Would I just need 2.4.4 on the 
server?  Or the clients too?

> If you want amanda to use less disk space, crank down your 
> 'tapelength'.  
> Since you're using diskless backup, you can define whatever 
> size "tapes" 
> you want.

True, but I have 18GB over 30 tapes.  So, to be completely 
safe from Amanda filling the disk with unnecessary full 
backups, I'd have to set the tapelength to 600MB, which, 
unfortunately, wouldn't be enough for any single client full 
backup.

Is there a wishlist for 2.4.5 anywhere?

Regards, Ralph.

Amanda Backup Strategy