On Sunday 12 January 2014 03:27:48 Andrius D. Ilgunas did opine:
> Excellent discussion Jon! Thank you so very much for your thoughts!!
>
> I was aware of Amanda's philosophy of "I want one full dump every
> dumpcycle which is n days" and then to let Amanda find the optimal
> combination of full/incrementals, and that's one of the reasons that
> Amanda was so attractive, other than it uses native compression tools
> (BIG plus for me).
>
> Heck, I even learned about the Tower of Hanoi scheme! But one piece of
> advice stuck with me, and that is "when restoring data, you don't have
> time to try to remember/decipher which files are on which tapes". In
> other words, backup / on a regular schedule, even if it'll cost you
> some disk space/time....it'll be worth it when it comes time to
> recover. I suppose I can make things more complex when I get really
> bored. ;)
>
> As I'm working out the backup plan, it's starting to seem that I'll want
> to have a dumpcycle of 7 days, and then keep those weekly full dumps
> elsewhere.
Unless you track amanda's logs in excruciating detail, you won't know when
in your week that the full was done, so you are creating a lot more
housekeeping outside of amanda's own record keeping that is not really
required. When you do a recovery or restore (2 different operations
although I am not sure what the exact difference is after using amanda for
well over a decade) you give the recover a filename and a date, or perhaps
a directory and a date, and amrecover then backs up to the last full before
that date, restores that, then restores all the incrementals between that
last full and the date you gave it.
Referring back to amanda's planner, if you have a fixed tape size that is
maybe a bit small like my situation 10 years ago when I was trying to make
do on DDS2 tapes, which amanda can put, using external compression, nearly
4Gb of data on one a tape. By knowing the tapes capacity, its possible to
train amanda to come within 100 megs of filling up a tape every night,
without ever running out of tape.
But I found that tape format to be very lacking in dependability, and as a
home hobbiest, there was no way I could afford the LTO drives or its tapes
so when 500Gig and up hard drives became available, I converted to vtapes
on a hard drive, which has turned out to be, dollar for dollar, about 100x
more dependable, and am currently using 30 vtapes on a 1 terrabyte drive,
which is currently about 65% utilized as I am also backing up the home dirs
on a pair of computers out in the shop that run my cnc mill and cnc lathe.
Linux has excellent drive health drive monitoring utilities, so the 3 drive
failures I have had since converting about 8 or 9 years ago, has always
given me sufficient warning that no data has been lost, not a byte.
Then about 12 years back I realized that a bare metal recovery to a newly
formatted operating drive would always be a 1 days loss because the amanda
data would also be lost, what was on the tapes was a day old, I wrote some
wrapper scripts that make a tarball each of the instant configuration
directory, and a tarball of the data directory amanda keeps and appends
them both to the tape just used, so that when doing the full recovery, it
started from the last known good state. That adds a few percent to the
size of each backup run, so in the instance where I was still using DDS2
tapes, I had the tape size set at 3650 megabytes to assure that amanda left
me enough room on the tape to append the data that included the just
completed backup.
With vtapes, there is no EOT so that potential overflow no longer exists.
For offsite storage, you could remove that drive, and prepare a new one
monthly, but I wouldn't take the removed drive offsite until one full
dumpcycle is on the new drive, else you would just have to go get it back
should you need to recover in that initial dumpcycle on the fresh drive.
Properly carved up bash scripts make the new drive prep a matter of
swapping the drive, starting the script and go fix your cold & empty coffee
cup. If 2 or more configs, use separate scripts because you'll be naming
the tapes according to the config they go with.
> At this time, we don't have a massive quantity of critical
> data, and we do have enough backup storage to keep 52 full dumps/year.
> One the reasons that I'm going to keep so many is because of the way
> one of our database has been (poorly) designed. There have been
> questions asked such as:
>
> "On which day did this record change?"
>
> Unfortunately, even with the binlog enabled, some of the queries don't
> get recorded (long story),
And one that needs attended to in case forensics are required.
> so at least for the database config, it
> seems like we'll be keeping many full dumps.
>
>
> Another part of my backup plan is going to have two configs. One for
> the database instance ('hypercritical data') that will be backed up
> insanely often, and another config for merely really important data
> (svn repos, wikis, etc) which will have a less taxing schedule.
Separate configs need separate storage, you cannot mix tapes if you want to
keep your sanity. Let each amanda config keep its own database and its own
named tapes. Its really quite good at it.
> Now I suppose that I could combine these two schedules into one config,
> but it seems that it would make more sense from an administrative view
> to have two (more?) configs.
>
> Do you/anyone else have any thoughts on that?
>
Thats my $0.02, but adjust for inflation since 1934. Yeah, I'm an older
user. ;-)
>
> --
> Andrius
>
> On Sat, Jan 11, 2014 at 9:10 PM, Jon LaBadie <[email protected]> wrote:
> > On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> > > Hmm...Jon's question got me to a'thnkin' about something that I
> > > haven't
> >
> > yet
> >
> > > read about.
> > >
> > > So let's say the dump cycle is 30 days and I want to keep a copy of
> > > the previous month's dump before the tape1 gets overwritten so that
> > > I can
> >
> > keep
> >
> > > a record of the state of the database from two months ago.
> > >
> > > If I understand the terminology, this would be an archive?
> >
> > Andrius,
> >
> > You might want to read further about the way Amanda differs from most
> > backup systems. Typically they act as you are describing, a full dump
> > of everything once in a while and incrementals in-between. So you get
> > a monster dump followed by a bunch of small dumps.
> >
> > Amanda tries to level out the size of the dumps. You define objects
> > to be backed up, called DiskList Entries (DLEs) in Amanda, and how
> > you want them backed up (they need not all be the same). Amanda then
> > spreads the full back ups of the DLEs across the whole dump cycle.
> > At the beginning of each dump run Amanda spends time determining how
> > big a dump it expects from each DLE as several levels, at least two,
> > level 0 full and level 1 incremental. Then it determines how best to
> > achieve balanced dumping while still keeping to your defined
> > parameters. For any specific DLE it may continue to do a level 1
> > incremental, switch it to a smaller level 2 incremental, or promote
> > it to an early level 0 full dump.
> >
> > > So a strategy might be: on the 30th day, do a full dump, and copy it
> > > to 'somewhere safe', whether on S3, or a DVD. That seems like a
> > > pretty straight-forward concept, but might this idea be improved
> > > upon somehow?
> >
> > A strong recommendation is to never have less than two full
> > dumpcycles. I suspect most Amanda installations maintain well over 2
> > dumpcycles. If you are reusing your storage media, and you only have
> > one level 0, should it be overwritten by mistake or something bad
> > happen to it, you only have less useful incrementals.
> >
> > Jon
> > --
> > Jon H. LaBadie [email protected]
> >
> > 11226 South Shore Rd. (703) 787-0688 (H)
> > Reston, VA 20190 (609) 477-8330 (C)
Cheers, Gene
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
"I'd love to go out with you, but my favorite commercial is on TV."
A pen in the hand of this president is far more
dangerous than 200 million guns in the hands of
law-abiding citizens.
