Re: Databases

[Winston Sorfleet]

Databases are a bit of an odd animal when it comes to backups.  If a
single entry in a table changes, to the outside world, it appears that
the entire database (as a file) needs to be backed up.  That's why most
real databases come with their own backup mechanisms (often transaction
logs).

That being said, I use (for mysql) mysqldump from a daily cronjob,
putting it into a file that amanda picks up.  I.e. I am not doing it
like a pro, but my databases are quite small.

On 2018-01-05 03:22 PM, Chris Miller wrote:
> Hi Folks,
>
> I can't find any explicit guidance for databases, so I assume that I
> do any database preparation in the cron job before I invoke amdump,
> and then tell Amanda to prune the database. Am I right?  OR, is there
> an agent for the common databases, like MySQL, and PostgresSQL?
> -- 
> Chris.
>
> V:916.974.0424
> F:916.974.0428

Re: Databases

[Schlacta, Christopher]

There's a program called ZRM that does backups and restores of
MySQL/Maria, but not PgSQL or PlSQL...  It works brilliantly, and I've
only ever had one complaint about it, and that's that despite being
developed by the same folks as Amanda, it doesn't just work as a DLE
in Amanda like it should to be a proper integrated point-in-time
backup.

On Fri, Jan 5, 2018 at 1:01 PM, Debra S Baddorf  wrote:
> Amanda just runs OS provided backup programs, such as DUMP and TAR.
> Any other work your database needs must be done in advance, as you suggest 
> with your cron job suggestion.
> Amanda won’t “prune” databases … but she’ll back up the file that’s on the 
> disk.
>
> Others: correct me if I’m wrong, of course.
> Debra Baddorf
> Fermilab
>
>
>
>> On Jan 5, 2018, at 2:22 PM, Chris Miller  wrote:
>>
>> Hi Folks,
>>
>> I can't find any explicit guidance for databases, so I assume that I do any 
>> database preparation in the cron job before I invoke amdump, and then tell 
>> Amanda to prune the database. Am I right?  OR, is there an agent for the 
>> common databases, like MySQL, and PostgresSQL?
>> --
>> Chris.
>>
>> V:916.974.0424
>> F:916.974.0428
>
>

Re: Databases

[J Chapman Flack]

On Jan 5, 2018, at 2:22 PM, Chris Miller  wrote:
> I can't find any explicit guidance for databases, so I assume that
> I do any database preparation in the cron job before I invoke amdump,
> and then tell Amanda to prune the database. Am I right?  OR, is there
> an agent for the common databases, like MySQL, and PostgresSQL?

There's one for PostgreSQL, described here:

http://wiki.zmanda.com/man/3.5.1/ampgsql.8.html

It's one example of an Amanda 'application', which is essentially
a plug-in for Amanda to back up a certain special kind of stuff.

If you can write a little program (in Perl or another language)
that, roughly, knows how to read your special kind of stuff and
send it out a pipe as a stream of bytes, or later get that stream
of bytes piped back to it and write out the recovered stuff, you
can write an application specific to another database, or other
specialized application you want to back up. This extensibility
is an Amanda strength (if a bit underutilized; there is a pull
request pending right now to make it easier to do).

In the case of a DBMS like PostgreSQL, there is a lot of latitude
for deciding what you even mean by "taking a backup", "pruning",
"incremental backup", and so on. The existing ampgsql application
implements one particular way of thinking about that. If a different
way would fit your setup better, it could be implemented as another
application (maybe just ampgsql slightly changed, or perhaps very
different).

-Chap

Re: Databases

[Debra S Baddorf]

Amanda just runs OS provided backup programs, such as DUMP and TAR.
Any other work your database needs must be done in advance, as you suggest with 
your cron job suggestion.
Amanda won’t “prune” databases … but she’ll back up the file that’s on the disk.

Others: correct me if I’m wrong, of course.
Debra Baddorf
Fermilab



> On Jan 5, 2018, at 2:22 PM, Chris Miller  wrote:
> 
> Hi Folks,
> 
> I can't find any explicit guidance for databases, so I assume that I do any 
> database preparation in the cron job before I invoke amdump, and then tell 
> Amanda to prune the database. Am I right?  OR, is there an agent for the 
> common databases, like MySQL, and PostgresSQL?
> -- 
> Chris.
> 
> V:916.974.0424
> F:916.974.0428

Re: First run...

[Jon LaBadie]

On Fri, Jan 05, 2018 at 11:23:49AM -0800, Chris Miller wrote:
> Hi Jean-Louis, 
> 
> | runtapes 1  # number of tapes to be used in a single run of amdump
> 
> | Because you tell it to use one tape.
> As advertised, Amanda behaviors that I don't yet understand... 
> 
> So, I suspect that vtape behavior is a big piece of Amanda and from this low 
> point on the learning curve, it looks formidable. Is there some document that 
> describes it sufficiently that I don't have to make a lot of mistakes and ask 
> a lot of stupid questions? 
> 
> So, having been so helpful with that question, I have two more: 
> 
> 1. If I am backing up a client to a NAS appliance, then I don't see how 
> it makes sense to have a holding disk, so I think I want to configure that 
> out. Am I right? 

No you are not.

Amanda does not just store backups on random access devices.  Some
amanda sites still use real tape.  The core sections of amanda are
written to be storage device agnostic.  With real tape, only one
DLE can have its backup being written to the storage device at a
time.  If a backups are written directly to the storage device each
DLE will have to wait its sequential turn.

In my small environment, 6 hosts and 25 DLEs, that could take as long
as 10+ hours ('Dump Time' in the daily report).  Yet the longest the
entire run has taken has been 3.5 hours ('Run Time').  That is the
first benefit of a local random access device, called a holding disk,
to collect DLE dumps before sending them to the storage device.  The
holding disk can be collecting multiple simultaneous dumps, even while
sending a previously completed dump to the storage device.

I configure my setup to allow a maximum of 4 simultaneous dumps, not
more than 2 from the same host or disk spindle.

Perhaps, though I doubt it, you are fortunate enough to never have
your connection to the NAS down.  That is a second benefit of a
holding disk.  Where do you put backups when the storage device is
not available?  My holding disk is sufficiently large to save 2-5
days worth of normal backups.  For extended outages, when holding
disk space gets tight, amanda switches to doing only incrementals.

BTW the normal amanda recovery tools know if a backup is on the
holding disk or on the storage device and can do recoveries from
either.

So yes, you can run amanda without a holding disk.  But I wouldn't.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Databases

[Chris Miller]

Hi Folks, 

I can't find any explicit guidance for databases, so I assume that I do any 
database preparation in the cron job before I invoke amdump, and then tell 
Amanda to prune the database. Am I right? OR, is there an agent for the common 
databases, like MySQL, and PostgresSQL? 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428 

Re: First run...

[Winston Sorfleet]

You set runtapes to 1.  Therefore, Amanda is only allowed to use one
tape for a backup, and your backup doesn't fit on one tape.


On 2018-01-05 01:35 PM, Chris Miller wrote:
> Hi Folks,
>
> I have followed these instructions
> (http://docs.huihoo.com/zmanda/amanda-community-setup-15-minutes-ver13.pdf),
> interpolating where necessary, and solved all the "computer" problems
> along the way. I am now into "Amanda" problems, meaning things I don't
> completely understand. (-:
>
> I have completed "amcheck " successfully, and I have tried to
> request a backup, "amdump ". I was enormously optimistic...
> The email report arrived quickly and complained of what is probably a
> configuration problem, but I don't understand what I need to do. I
> could use some advice. I have included what I think is the relevant
> config as an attachment.
>
> The e-mail reports that the problem is:
> FAILURE DUMP SUMMARY:
>   planner: FATAL cannot fit anything on tape, bailing out
>   mail.tclc.org /var/log lev 0  FAILED [dump estimate ( 10051085 KB )
> is larger than available tape space ( 512 KB ),  but cannot
> incremental dump new disk]
>
>
> Now, I recognize that the simple answer is I have made my vtapes too
> small, but there is a larger confusion. Why didn't the backup use as
> many vtapes as necessary? Somewhere there must be a good discussion of
> how vtapes work and what Amanda does in response to various
> situations. Can anybody advise me?
>
> Thanks for the help,
> -- 
> Chris.
>
> V:916.974.0424
> F:916.974.0428
>

Re: First run...

[Winston Sorfleet]

No manual just a wiki and FAQ.  The best way to learn is read the section on 
basic configuration then the section "under the hood" which tells you what the 
config is doing.  

Many of your questions about how to do the things you want with Amanda seem to 
be under some misunderstanding on what Amanda is supposed to do.  Forgive me 
but it seems you are trying to dumb down Amanda to a single vtape on a NAS 
without a holding drive now.  You can do this but it removes all the value that 
comes with Amanda.  You might as well just tar to a file on a NAS NFS mount.  
It'll be a lot simpler and there's nothing wrong with that if you don't need 
volume management.

On January 5, 2018 2:23:49 PM EST, Chris Miller  wrote:
>Hi Jean-Louis, 
>
>| runtapes 1   # number of tapes to be used in a single run of amdump
>
>| Because you tell it to use one tape.
>As advertised, Amanda behaviors that I don't yet understand... 
>
>So, I suspect that vtape behavior is a big piece of Amanda and from
>this low point on the learning curve, it looks formidable. Is there
>some document that describes it sufficiently that I don't have to make
>a lot of mistakes and ask a lot of stupid questions? 
>
>So, having been so helpful with that question, I have two more: 
>
>1. If I am backing up a client to a NAS appliance, then I don't see how
>it makes sense to have a holding disk, so I think I want to configure
>that out. Am I right? 
>2. How to I specify that I am backing up to a NAS and not
>"file://var/amanda/vtapes/..."? 
>
>If anybody know were this stuff is discussed in a manual, then telling
>me where it is would be helpful. 
>
>Thanks for the help, 
>-- 
>Chris. 
>
>V:916.974.0424 
>F:916.974.0428 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: First run...

[Chris Miller]

Hi Jean-Louis, 

| runtapes 1# number of tapes to be used in a single run of amdump

| Because you tell it to use one tape.
As advertised, Amanda behaviors that I don't yet understand... 

So, I suspect that vtape behavior is a big piece of Amanda and from this low 
point on the learning curve, it looks formidable. Is there some document that 
describes it sufficiently that I don't have to make a lot of mistakes and ask a 
lot of stupid questions? 

So, having been so helpful with that question, I have two more: 

1. If I am backing up a client to a NAS appliance, then I don't see how it 
makes sense to have a holding disk, so I think I want to configure that out. Am 
I right? 
2. How to I specify that I am backing up to a NAS and not 
"file://var/amanda/vtapes/..."? 

If anybody know were this stuff is discussed in a manual, then telling me where 
it is would be helpful. 

Thanks for the help, 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428 

Re: Serious tape error

[Jon LaBadie]

On Fri, Jan 05, 2018 at 08:54:36AM -0500, Jean-Louis Martineau wrote:
> This is an issue in 3.4, 3.4.1 and 3.4.2
> 
> It is fixed in 3.4.3

Ahh yes, I do [now] recall that fix being reported.

Thanks, I will look for an available update.
> 
> Jean-Louis
> 

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: First run...

[Jean-Louis Martineau]

On 05/01/18 01:35 PM, Chris Miller wrote:
Now, I recognize that the simple answer is I have made my vtapes too 
small, but there is a larger confusion. Why didn't the backup use as 
many vtapes as necessary? Somewhere there must be a good discussion of 
how vtapes work and what Amanda does in response to various 
situations. Can anybody advise me?


runtapes 1  # number of tapes to be used in a single run of amdump


Because you tell it to use one tape.

Jean-Louis

Thanks for the help,
--
Chris.

V:916.974.0424
F:916.974.0428


This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

First run...

[Chris Miller]

Hi Folks, 

I have followed these instructions 
(http://docs.huihoo.com/zmanda/amanda-community-setup-15-minutes-ver13.pdf), 
interpolating where necessary, and solved all the "computer" problems along the 
way. I am now into "Amanda" problems, meaning things I don't completely 
understand. (-: 

I have completed "amcheck " successfully, and I have tried to request a 
backup, "amdump ". I was enormously optimistic... The email report 
arrived quickly and complained of what is probably a configuration problem, but 
I don't understand what I need to do. I could use some advice. I have included 
what I think is the relevant config as an attachment. 

The e-mail reports that the problem is: 
FAILURE DUMP SUMMARY: 
planner: FATAL cannot fit anything on tape, bailing out 
mail.tclc.org /var/log lev 0 FAILED [dump estimate ( 10051085 KB ) is larger 
than available tape space ( 512 KB ), but cannot incremental dump new disk] 


Now, I recognize that the simple answer is I have made my vtapes too small, but 
there is a larger confusion. Why didn't the backup use as many vtapes as 
necessary? Somewhere there must be a good discussion of how vtapes work and 
what Amanda does in response to various situations. Can anybody advise me? 

Thanks for the help, 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428 



amanda.conf
Description: Binary data

Re: amanda tape algorithm

[Jean-Louis Martineau]

On 04/01/18 12:59 PM, Ned Danieley wrote:
>
> I'm using
>
> taperalgo largestfit
>
> which I assume means that amanda will write to tape the largest DLE
> available. I have 'runtapes' set to 2, and occasionally I'll see amanda
> move on to the second tape when the first tape is reporting around 80% 
> full.
> I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
> that would fit in the remaining 20%. is there any way to find out (after
> the fact) what DLEs were in the queue? I've looked at the taper debug file
> but it just seems to have a list of each DLE as it is written.
Not easily, you can parse the amdump. file

Make a copy of amdump.
Remove all lines after it sent the command to the taper
Run amstatus on that modified file

What was the size of the dle that failed to be written to the first tape?
>
> I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
> about the ability to split DLEs, but I'm not ready to take that step.
You should try.

Jean-Louis
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: Serious tape error

[Jean-Louis Martineau]

This is an issue in 3.4, 3.4.1 and 3.4.2

It is fixed in 3.4.3

Jean-Louis

On 04/01/18 06:37 PM, Jon LaBadie wrote:
> On Thu, Jan 04, 2018 at 05:12:27PM -0500, Jon LaBadie wrote:
>> I use vtapes of 100GB and runtapes == 3.  Most days only
>> one tape is used.  About once every 2 or 3 weeks a second
>> tape is needed.  Yesterday was one of those days.  But
>> amanda chose the same vtape for the second tape and over
>> wrote the data it had just written to the "first" tape.
>>
>> >From the report:
>>
>>Tape Size (meg) 179621.1   171549.4 8071.8
>>
>> Need multiple tapes.
>>
>>USAGE BY TAPE:
>>  Label Time Size  %  DLEs Parts
>>  DS1-215   1:45  179621M  175.42558
>>
>> Only one tape listed, filled to 175%.
>>
>>Notes:
>>  taper: tape DS1-215 kb 104856364 fm 34 [OK]
>>  taper: tape DS1-215 kb 79075697 fm 24 [OK]
>>
>> Tape DS1-215 selected twice.  Only the 24 files from the
>> later run are present on the vtape, not the 58 total files.
>> The first run was overwritten.
>>
>>
>> JLM I've saved all the log files I could find from this run.
>> What might you need to analyze this further?
>>
>> I looked at /var/log/amanda/server/DS1/taper. and it
>> appears to only contain information about the 2nd usage of
>> the tape.
>>
> I was wrong, there is info from both uses of the same tape in
> the taper logfile.
>
> Jon
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail