Re: garbage collect: free orphaned chunks from disks?

[Jon LaBadie]

On Wed, Mar 08, 2023 at 12:20:00PM +0100, Heiko Schlittermann wrote:

Hi,

Jens Berg  (Mi 08 Mär 2023 10:06:34 CET):

She does in the moment a tape (= slot) is re-used.

…

Remember that amanda was initially designed to work with tapes and that


Olivier  (Mi 08 Mär 2023 11:46:37 CET):

While the tape/slot could free the level 1 dumps for a-disk, there could be
some DLE for other disks/hosts that are still valid on that tape/slot.



Yes, I know that amanda was designed for tapes initially (and I'm still
using it happily with tapes.).

The "disk garbage collection" was just a minor issue I found when
checking what space is available on one of our disk based backup
servers.

Thanks for the input.


Thinking outside of amanda provided tools:

1. Assuming tar is your backup program, the "garbage" files do still
   contain recoverable data.  I did make use of this "once" to recover
   a file deleted just before that oldest full dump.

2. The files on vtapes are named such that a find(1) command could be
   constructed to scan the backup disks looking for the "garbage" files
   (they would be "older" than the oldest full dump) and either list
   them or delete them.

jl

--
Jon H. LaBadie j...@labadie.us
 154 Milkweed Dr (540) 868-8052 (H)
 Lake Frederick, VA 22630(703) 935-6720 (M)

Re: (OT) What backup to use with Windows

[Jon LaBadie]

On Tue, May 31, 2022 at 03:02:15PM +0700, Olivier wrote:

Hi,

I know this is not about Amanda, butt this is about backup.

I am trying to find a suitable solution for Windows/Mac/Linux
desktops. Amanda has a major flaw: it is working on a pull schedule, the
client must be available when the server is starting the backup.

But clients tend to have unpredictable schedule, so the backup must be
initiated from the client side.



I've not used it could be off base here, others please correct.

Isn't "amdump_client" used to initiate dumps from the client side?

--
Jon H. LaBadie j...@labadie.us
 154 Milkweed Dr (540) 868-8052 (H)
 Lake Frederick, VA 22630(703) 935-6720 (M)

Re: Is there a write-up on using amvault with an archive config?

[Jon LaBadie]

Thanks Winston, this moves me a bit closer.

A couple of inline questions:

On Tue, Mar 22, 2022 at 12:34:16AM -0400, Winston Sorfleet wrote:

I use amvault to tertiarary media (LTO-2, I am just a casual home user)
while my main archive is VTL on a slow NAS.  I use cron, but obviously I
could just run as-needed from the command line.


My archive will be on its own disk.  I planned to keep it in the server
along with the regular backups, but moving it to a different computer
could further reduce some failure modes.


You're right, it is a bit hard to intuit, and I had to get some help
from the community here as it is using overrides.

The command line I use is as follows:

/usr/sbin/amvault -q --latest-fulls --dest-storage "tape_storage" vtl

Where vtl is the config.  The key part is the "tape_storage" which
refers to the appropriate vault-storage template in the amanda conf
file.  E.g.


So "vtl" is the config name of the archive, correct?
Would that config name be used in restore/recovery commands as well?

Just doing "latest-fulls" would not be appropriate for my use case.
For some pretty static DLEs I only do fulls about every 6 weeks.
An example is my almost never changing collection of online music.
No need for amvault to archive many copies of that.  I likely would
use a date specification ("--src-timestamps ...) and "--fulls-only".

I see that specific DLE can be specified at the end of the amvault
command line.  From the manpage it shows:

   [hostname [ disk [ date [ level ...

However I could have multiple amanda configs with the same hostname and 
disk combination, say a DailySet and a WeeklySet.  Where would you

specify the amanda config you wish to archive (or "vault" if you wish).

The manpage says "latest" can be used as an alternative to a date
specification.  The wording is "then the most recent amdump or amflush
run will be used."  Do you know if that is literally accurate?  If I
use both "--fulls-only" and "latest" plus list a specific DLE, will
nothing be archived if the level 0 was in the 2nd most recent dump?
Or might it locate the latest level 0 of that DLE?




storage "vtl"
vault-storage "tape_storage"

define storage "tape_storage" {
    erase-on-failure yes
    policy "HP_Robot"
    runtapes 1
    set-no-reuse no
    tapedev "LTO-2"
    tapetype "LTO2"
    tapepool "$r"
    tpchanger "LTO-2"
    labelstr "Vault-[1-7]"
    autolabel "Vault-%" any
}


For this requirement, I love the idea of autolabeling.
Will be another first for me.


define changer LTO-2 {
    tpchanger "chg-single:/dev/nst0"
    device-property "LEOM" "TRUE"
}

define tapetype LTO2 {
   comment "HP Ultrium 448, hardware compression off"
   length 193024 mbytes
   filemark 0 kbytes
   speed 20355 kps
}

Obviously for you it will be simpler since you don't have to engage the
SCSI subsystem and define actual tapetype parameters or fiddle with
blocksizes.  And you're not limited to a single "tape".


I think I'll be looking to "spin down" the archive disk.
It would be used so seldom.

Thanks again.
Jon



On 2022-03-21 15:46, Jon LaBadie wrote:

*** Apologies if a near duplicate has been posted ***
*** I initially submitted it with the wrong email ***


Amazing, I've used amanda for about 25 years and never set up
an archive config nor used amvault.  No time like the present
as I setup a new server with increased capacity.

I don't want an archive config that does periodic massive
dumps.  Instead I'd prefer that on-demand I could copy a
level 0 DLE to the archive in such a way that amrecover/
amrestore could use the archive config.  Both the source
and the archive destination would be vtapes but on
different drives in different housings.

I "think" that amvault would be the appropriate tool.
If not, correct my error please.

Has anyone done a write-up on setting up and using such
a scheme?

Thanks,
Jon


End of included message <<<


--
Jon H. LaBadie j...@labadie.us
 154 Milkweed Dr (540) 868-8052 (H)
 Lake Frederick, VA 22630(703) 935-6720 (M)

Is there a write-up on using amvault with an archive config?

[Jon LaBadie]

*** Apologies if a near duplicate has been posted ***
*** I initially submitted it with the wrong email ***


Amazing, I've used amanda for about 25 years and never set up
an archive config nor used amvault.  No time like the present
as I setup a new server with increased capacity.

I don't want an archive config that does periodic massive
dumps.  Instead I'd prefer that on-demand I could copy a
level 0 DLE to the archive in such a way that amrecover/
amrestore could use the archive config.  Both the source
and the archive destination would be vtapes but on
different drives in different housings.

I "think" that amvault would be the appropriate tool.
If not, correct my error please.

Has anyone done a write-up on setting up and using such
a scheme?

Thanks,
Jon

--
Jon H. LaBadie j...@jgcomp.com
 154 Milkweed Dr (540) 868-8052 (H)
 Lake Frederick, VA 22630(703) 935-6720 (M)

Re: LVM for vtapes

[Jon LaBadie]

On Tue, Sep 21, 2021 at 10:01:04AM +0200, Diego Zuccato wrote:

A different approach here.
I configured a single MD-RAID6 of 16 4TB disks and the vtapes are 
simple folders.
I did it mostly because I didn't know the exact space requirements our 
backups would have. Today I'd go with a smaller RAID (probably just 10 
disks, so to have 32TB and a sane stripe size of 4k).


I didn't like the idea of having "hard-sized" vtapes because of the 
wasted space.


Doesn't your "tapetype" definition specify a size?  Perhaps huge?

Defining a "hard size" does not allocate that amount of disk to each
vtape.  It sets a maximum size that is taken from the available pool.
In your case the pool is the entire raid, in mine it is each disks
filesystem.

My vtapes are "sized" to 100GB (0.1TB) and I create about 12 vtapes
per base10 TB.  Obviously I can't allocate 1.2TB of space from 1.0TB.
Experience in my environment shows this over allocations results in
about 90% filesystem usage as some tapes fill to 100%, others much less.

Jon


I don't like having to remove SAS/SATA HDDs because the connector is 
usually specced for about 500 mating cycles. If you need frequent 
swaps it's better to use a caddy with its own connectors (like 
ESATA/USB external enclosure, one per removable disk). With USB3 the 
performance shouldn't be an issue. Maybe the CPU load, but a recent 
system should be able to handle it quite well.


HIH.

Il 21/09/2021 05:17, Olivier ha scritto:

Jon,


Interesting discussion in other threads got me wondering
whether I should have made some other choices when setting
up my vtape environment.  Particularly whether I should
have used LVM (Logical Volume Management) to create one
large filesystem covering my multiple dedicated disks.

Its a topic I do not recall being discussed, pros & cons.


I am using 7 disks of 3 (or is that 4) and 6 TB (should upgrade them all to 6TB
soon) almost dedicated to vtapes (the last disk also has a copy of the
deleted accounts). I have them configured as individual disks. The size
of my vtapes is also about 100GB and I am using a small chunck size, so
my disks end up being 80% full at least.

When I designed my vtape architecture, I decided to keep each disk
individual so that it can be put offline after usage. My idea was to
have a system that could prompt an operator to "mount a disk" before the
backup and the disk could be manually unmounted and safe stored each
day. It is taking advantage of the automount service on FreeBSD.

Mounting could be USD disk, or hot-swap. I never went very far in the
implementation. I wrote all that many years ago when vtapes were new and
limited to a single directory, that is why I wrote my own tape changer.

I knew about the risk of loosing a disk and it being a good portion of
consecutive backups. But what I had in mind was:

- have the system as simple and as portable as possible, so I can shove
  a disk in another machine and extract contents manually (during the
  great flood of Bangkok in 2011, I moved all the servers and also took
  all my hard disks from Amanda backup, but I did not need to move the
  rack mounted server itself);

- a side advantage of my own tape changer is that I can keep the older
  disks (each disk has an individual label, like any vtape has a label)
  (I have updated them from 500GB to 1TB to 3TB and soon to 6TB) and the
  vtapes are still known into tapelist (they are marked noreuse). If the
  need arise, I can still remount that old disk.

So far (10+ years) the only disk I had failling was the disk having the
holding partition, I guess it was because of excessive usage.

I understand that vtapes have evolved since I started using them, but my
system works for me so I never took the time to search any further.

Best regards,

Olivier





--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

LVM for vtapes

[Jon LaBadie]

Interesting discussion in other threads got me wondering
whether I should have made some other choices when setting
up my vtape environment.  Particularly whether I should
have used LVM (Logical Volume Management) to create one
large filesystem covering my multiple dedicated disks.

Its a topic I do not recall being discussed, pros & cons.

Let me describe my current vtape environment as a starting
point.  I have 6 hard drives dedicated to vtapes, 3x3TB
and 3x4TB, 21TB total.  I have 240 vtapes of 100GB (0.1TB)
allocated.  That works out to 11-12 vtapes/TB.

When I first set things up, I did not like the idea of
one drive having all consecutive vtapes meaning it would
exclusively be used for about a month and not used again
for about 4 months.

Instead I allocated blocks of 8-12 vtapes on one drive
and the next block on the next drive ...  In part I felt
on drive failure, it would minimize my loss exposure of
recent backups to about a week rather than a month.

I wonder whether I might have instead just pooled the
six drives into one logical volume and not worry about
specific allocation ov vtapes to specific drives.

Hmmm, 21TB filesystem, hell of a beast to fsck ;)

Has anyone used LVM for their vtapes?  Positive or
negative results to report?

Jon

--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: DLE splitting

[Jon LaBadie]

is there a way to run a user supplied program just
before amanda's taper opens each tape file?  I.e.,
before it actually opens and creates a new file on
the tape.

Do I recall correctly that it opens a tmp file on
the vtape and when complete renames it?  Or am I
thinking of the chunks on the holding disk?

jon

--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: DLE splitting

[Jon LaBadie]

On Fri, Sep 17, 2021 at 11:55:09AM -0400, Jon LaBadie wrote:

I'm seeing instances of wasted VTape with amanda switching
to a new tape for the last DLE even though there is room
left on the current tape.

Here is the relevant section from yesterday.

USAGE BY TAPE:
 Label Time Size  %  DLEs Parts
 DS1-010   0:08   10884M   10.63131
 DS1-011   0:51   74205M   72.5 115

If combined on one tape it would have only filled 83% of the
tape.  Earlier this month the numberw were worse, 9% and 66%.
Even if a second tape were needed, the last DLE should have
at least started on the first tape.



A new thought.  When the amanda taper has been writing to a
vtape and attempts to open the next part file, is there some
timeout that would look like the vtape is full/unavailable?

Four of my six vtape drives, including the one in use above,
are in an external USB enclosure.  These drives go to sleep
if inactive for a period.  When a drive is accessed, it takes
20 to 30 seconds for the 4 drives to spin up and be ready.

The DLE that was not taped on the first vtape took over an
hour to dump.  Certainly the USB box went to idle state
before it was ready to tape.

When I do things from the command line that awakens the idle
USB box, the command just blocks until the device is active
(as I say, 20-30sec) and then completes normally.

But perhaps the calls (perl or C) that amanda uses to open
the next part return with a code that says EOF or error
rather than "currently unavailable, try again".

Jon

--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: allow-split configuration parameter

[Jon LaBadie]

On Fri, Sep 17, 2021 at 01:09:48PM -0400, Gene Heskett wrote:

On Friday 17 September 2021 12:09:55 Jon LaBadie wrote:


While investigating a tape issue (separate thread)
I investigated the "allow-split" parameter of
amanda.conf.  The man page says it defaults to "true"
and my configuration has never set it.

Thinking perhaps the default has changed I tried
setting allow-split several ways and places.  amcheck
always complains "configuration parameter expected".

Also, amgetconf says there is no configuration parameter
called "allow-split".

Does anyone have "allow-split" in their configuration?
If so, how/where?


I do acc grep:
gene@coyote:~$ sudo grep allow /usr/local/etc/amanda/Daily/amanda.conf
maxdumps 1  # allowing one per host makes sense to me
#   skip-full - skip the disk when a level 0 is due, to allow full
backups

define dumptype  global {
...
allow-split true
}


Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett


OF COURSE, it a dumptype parameter.   My stupidity.

I've added it, but I suspect it will have no affect
as the default is true.

Jon

--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

allow-split configuration parameter

[Jon LaBadie]

While investigating a tape issue (separate thread)
I investigated the "allow-split" parameter of
amanda.conf.  The man page says it defaults to "true"
and my configuration has never set it.

Thinking perhaps the default has changed I tried
setting allow-split several ways and places.  amcheck
always complains "configuration parameter expected".

Also, amgetconf says there is no configuration parameter
called "allow-split".

Does anyone have "allow-split" in their configuration?
If so, how/where?

--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

DLE splitting

[Jon LaBadie]

I'm seeing instances of wasted VTape with amanda switching
to a new tape for the last DLE even though there is room
left on the current tape.

Here is the relevant section from yesterday.

USAGE BY TAPE:
  Label Time Size  %  DLEs Parts
  DS1-010   0:08   10884M   10.63131
  DS1-011   0:51   74205M   72.5 115

If combined on one tape it would have only filled 83% of the
tape.  Earlier this month the numberw were worse, 9% and 66%.
Even if a second tape were needed, the last DLE should have
at least started on the first tape.

Ideas?

--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amflush surprize auth=ssh.

[Jon LaBadie]

On Tue, Aug 24, 2021 at 07:06:44PM +, Debra S Baddorf wrote:

Wasn’t there, at some point,  a parameter in addition to the chunksize param?
I’m using chunking,  but yes, the whole set of chunks is always re-taped.
But I’m using real tapes so that’s what I want.

Wasn’t there another parameter, besides chunking onto the holding disk,
which actually allowed the chunks to make it to your (virtual) tape?
To say splitting was allowed on ’tape’  (as well as on holding disk)?
That sounds like the parameter you need, to make your re-taping not
start at the beginning of the set.

I can’t find such a param in my setup files right now.
Of course, I’m not using it,   so maybe I didn’t add it to my files.

Deb Baddorf
Fermilab


My bad.  Terminology is "chunks" on holding disk, "parts" on tape.
I was refering to the 45 parts that would be on the tape(s).

Yes there is a parameter "allow-split" which defaults to "yes".
The parameter is absent from my config files so it is default.

There are also parameters for specifying the part size.

Jon




On Aug 24, 2021, at 12:10 AM, Jon LaBadie  wrote:

On Mon, Aug 23, 2021 at 06:30:24PM +0100, Jose M Calhariz wrote:

Hi,

What you describe is what I see when I do not use chunks.  The last
backup to the last vtape will be transfered again to vtape on next
amdump or amflush.

Kind regards
Jose M Calhariz


Yes, that is the historically desired behavior when DLE dumps could
not be split across physical or virtual tapes.  The last DLE dumped
when the tape "ran out" must be retaped in its entirety.  But with
chunks, no more than the last chunk should need to be retaped.
Amanda has record of which chunks were successfully taped.

Jon



On Sun, Aug 22, 2021 at 02:40:52PM -0400, Jon LaBadie wrote:

I have one large DLE that gets broken into about
45 chunks for taping.  My vtapes hold 20 of these
chunks and I allow up to 3 tapes to be used nightly.

Two evening back another large DLE was in the same
dump so only about 30 of the 45 chunks fit on the
three tapes.

Today I ran amflush thinking amanda would just tape
the 15 untaped chunks onto 1 vtape.  But no, it is doing
the entire 45 which will take 3 vtapes.

In a vtape environment, is this reasonable behavior?

Was there a way to flush only the untaped chunks?

Jon







End of included message <<<




--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amflush surprize auth=ssh.

[Jon LaBadie]

On Mon, Aug 23, 2021 at 06:30:24PM +0100, Jose M Calhariz wrote:

Hi,

What you describe is what I see when I do not use chunks.  The last
backup to the last vtape will be transfered again to vtape on next
amdump or amflush.

Kind regards
Jose M Calhariz


Yes, that is the historically desired behavior when DLE dumps could
not be split across physical or virtual tapes.  The last DLE dumped
when the tape "ran out" must be retaped in its entirety.  But with
chunks, no more than the last chunk should need to be retaped.
Amanda has record of which chunks were successfully taped.

Jon



On Sun, Aug 22, 2021 at 02:40:52PM -0400, Jon LaBadie wrote:

I have one large DLE that gets broken into about
45 chunks for taping.  My vtapes hold 20 of these
chunks and I allow up to 3 tapes to be used nightly.

Two evening back another large DLE was in the same
dump so only about 30 of the 45 chunks fit on the
three tapes.

Today I ran amflush thinking amanda would just tape
the 15 untaped chunks onto 1 vtape.  But no, it is doing
the entire 45 which will take 3 vtapes.

In a vtape environment, is this reasonable behavior?

Was there a way to flush only the untaped chunks?

Jon







End of included message <<<


--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

amflush surprize auth=ssh.

[Jon LaBadie]

I have one large DLE that gets broken into about
45 chunks for taping.  My vtapes hold 20 of these
chunks and I allow up to 3 tapes to be used nightly.

Two evening back another large DLE was in the same
dump so only about 30 of the 45 chunks fit on the
three tapes.

Today I ran amflush thinking amanda would just tape
the 15 untaped chunks onto 1 vtape.  But no, it is doing
the entire 45 which will take 3 vtapes.

In a vtape environment, is this reasonable behavior?

Was there a way to flush only the untaped chunks?

Jon

--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

network problems

[Jon LaBadie]

Some days my system logs are being flooded with hundreds
of thousands of network entries saying "STATE_INVALID_DROP".

Nearly all are caused by packets coming from my regular
desktop system to my amanda server.  And based on the
times and sizes (all full to MTU) they are amanda dump
packets.

When it occurs, a run of bad packets will last for about
one minute (300-400 bad packets per second) and after a
gap of about nine minutes will start again for one minute.
Here is a sample from last night.

  Bad Run of Packets
   startend
Feb 1303:28:4303:29:56
Feb 1303:38:4403:39:54
Feb 1303:48:4503:50:20
Feb 1303:58:4604:00:00
Feb 1304:08:4704:10:25
Feb 1304:18:4804:20:09
Feb 1304:28:4904:30:01

This continued until after 6AM.

Note, the amdump run successfully completed even
with this bad activity, but took much longer than
normal (like 2-3 times longer).

I am getting some of these bad packet runs each night.
However on most nights it is only a couple of bad runs
lasting a minute or even less.

Any suggestions on what/how to investigate this.

Thanks,
Jon
--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Amanda really blew up last night.

[Jon LaBadie]

On Mon, Feb 01, 2021 at 07:08:54PM +, Debra S Baddorf wrote:


I keep reading that there is less wasted space with tapes.   Just because you
allocate X space for each tape,  I believe they only use up the space that’s
actually used.


Certainly that is the case here.  I have 6 "over"subscribed disks
for my vtapes.  For example D1 is a 4 TB drive with 49 100 GB vtapes.
It currently is at 91% capacity.

--
Jon H. LaBadie j...@labadie.us
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

zmanda windows client

[Jon LaBadie]

I've been using the Zmanda Windows Client (ZWC) to
back up my two Windows Clients.  After several years
of working just fine, one has stopped working.
It is a Windows 7 laptop.  The other is a Windows 8
VirtualBox VM that continues to work fine.

The report error message is:

   planner: ERROR catwoman.jgcomp.com:  Server validation Failed.
   Please register server with client.

Registration on the client has had no effect.  Nor has reinstallation
of the dated ZWC software.

amcheck gives the same error message.

DNS is working on both client and amanda server.

What other settings might be out of whack?

Jon
--
Jon H. LaBadie j...@jgomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: How's amanda feeling these days?

[Jon LaBadie]

On Mon, Nov 16, 2020 at 07:25:41AM -0600, Dave Sherohman wrote:
> Hello, again!
> 
> You may recall my earlier question to the list, included below.  I've
> now talked with my other coworkers who work with servers and they've
> agreed to go with amanda for our new backup system.
> 
> Now I'd like to get some hardware recommendations.  I'm mostly unsure
> about what we'll need in terms of capacity, both for processing power
> and for storing the actual backups.  Less interested in specific model
> or part numbers, because it will need to come from one of our approved
> vendors, of course, and most likely by way of a formal tender process -
> but I can say that we almost always end up buying complete Dell
> rackmount systems.
> 
> The basic parameters I'm working with are:
> 
> - Backing up around 75 servers (mostly Debian, with a handful of other
>   linux distros and a handful of windows machines).
> 
> - Total amount of data to back up is currently in the 40 TB range.
> 
> - Everything is connected by fast (10- or 100-gigabit) networks.
> 
> - Backup will be to disk/vtapes.
> 
> - I've been asked to have backups available for the previous 6 months.
> 
> - I'm assuming that the best way to handle backup of windows clients
>   will be to mount the disk on a linux box and back it up from there,
>   although some of them are virtual machines, so doing a kvm snapshot
>   and backing that up instead would also be an option.
> 
> Given all that, how beefy of a box should I be looking at, and how much
> disk space can I expect to need?

I did reply to the original message, but looking back it was addressed
to you rather than the list.  In case it was overlooked, here were my
regarding space:

"Just some simple numbers.  Assuming a 7 day dumpcycle and daily runs.
 40TB / 7 day plus some promotion is about 7TB of level 0 (full) dumps
 per day.  Add a TB for incrementals means about 8TB of backup data / day.

 8TB / 1GB/sec is about 8000sec network traffic.  3-4 hrs, doable on your
 slower network.

 6 months retension is nominally 200 days X 8TB / day is 1600 TB of
 vtape capacity.  With 5TB disks thats 320 disks.  Compression will
 reduce that some, how much only experience will tell you."

> 
> Also, as a side note, I'm planning on using VDO (Virtual Data Optimizer)
> to provide on-the-fly data compression and deduplication on the backup
> server, which should reduce disk consumption at the cost of CPU
> overhead.  I'm thinking it would make the most sense to use VDO only for
> the filesystem holding the vtapes, and not for the staging area, but
> feel free to correct me on that.

Did not know what VDO was, so I read a Red Hat description.  It seems
to consist of 3 components each I question the value of for amanda backup.
Hopefully someone with VDO experience can share it.

Elimination of zero filled blocks:  Compression is likely to greatly
shrink the storage of a string of zeros.

Only one copy of duplicate blocks:  Were your files being backed up
individually, as I do in a separate backup of my Home directory using
rsync, this could provide a worthwhile savings.  But you will likely
be merging your files into a tarball or a dumpfile.  The original
disk block alignment will be lost and likely not even match in one
day's tarball to the next.

LZ4 compression on the fly:  I don't know the cpu load for the server
compressing 8TB of data daily.  One thing you would have to deal with
is amanda's view of what has been sent to the backup device and what
size the data actually consume on the device.

There are points where amanda calculates how much space is left on
the device based on it configuration-specified size and how much it
has already sent.  Of course there is actually more space available
because the compression occurs after amanda's involvement.  The
difference may cause amanda to make less than optimal decisions.

Amanda administrators who use tape drive compression face the same
problem.  I believe most over specify the size of the storage medium
to allow more complete tape utilization.


As to Windows backup, I hope someone suggests a good solution.  I
currently use the proprietary "Zmanda Windows Client".  Generally
works well but suffers from a lack of development and unexplained
failures to connect.  It is often corrected by restarting the ZWC
services on the Windows system and always corrected by rebooting.

In the distant past I backed up windows systems by mounting the drives
on UNIX host.  Most often used NFS.  Liked that approach except for
one thing.  Windows, at least then, does not like a file to be opened
by multiple processes.  So each backup included several files that
did not backup because the file was already opened by another Windows
process.  And a few system files were never backed up.

Regarding backing up a KVM snapshot, would that mean that to recover
one file you would have to take a new snapshot, restore the entire system
from the backed up snap, copy the file to somewhere 

amanda-excludes file - includes?

[Jon LaBadie]

Silly question, can an include be added to an
amanda-exclude file.  I.e., is there a syntax
to say "exclude this everything under this
directory except for FuBar".

The backstory:  My mailserver has a quarantine
directory (QD) for spam and virus containing mail.
This QD directory is listed in the amanda-excudes
file at the root of the DLE.  Who wants to backup
spam and viruses.

I have developed several scripts for dealing with
the collected spam and as they were of no use except
in the QD, I made a QD/.bin directory to hold them.

Stupidly I never considered the QD/.bin directory
would not be backed up (due to the amanda-excludes entry)
nor that I might moronically delete the QD/.bin contents.

As I redevelop the scripts, I'll put them in a relocated
bin directory.  But I just wonder if there was some way
to address this in an amanda-excludes file.  I doubt it.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

Glad you brought up this "feature" Nathan.  I had heard it before
but not using tape, promptly forgot it.

Jon


On Thu, May 14, 2020 at 05:30:53PM -0400, Nathan Stratton Treadway wrote:
> On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> > Interesting, how can a "dirty" drive trigger this behavior?
> > 
> > I'd expect failures all along and not after ~200 or 300 GB written.
> > 
> > I don't see any interrupted writing or so (until that End Of Tape).
> 
> 
> (We switched to disk-drive vtapes a long time ago so when I was last
> looking into the details of backup-tape-drive behavior it was probably
> for pre-LTO technology, but I would assume that for this discussion LTO
> is similar)
> 
> For "modern" error-correcting tape drives, when the computer sends data
> out to the tape drive to be written to tape, the drive actually then
> uses the read head to immedately read back in the data it just wrote. 
> If that read fails, the drive will automatically/transparently try the
> write again... repeating the process until it is able to achieve a
> successful confirmation read of that block of data.
> 
> Normally this just happens once in a while, when there's a bad spot on
> the tape or some fluke of writing makes the data unreadable, and one
> doesn't even notice it's happening.  
> 
> However, if the drive head is dirty or the tape media in general is
> wearing out, then what happens is that many many many of the data blocks
> either will be written badly or will fail to read back in [depending on
> what exactly is dirty or failing], and the drive will have to re-write
> data multiple times before a succesful write/read cycle.
> 
> When that happens, then lots of the linear space on the tape is used by
> all the repeated writes -- thus making the tape appear to have a lower
> capacity than you would expect -- and also all that re-writing means the
> data throughput from the server's point of view is much reduced.
> 
> (Note that in this scenario the drive just keeps retrying to write a
> block up data until it succeeds... or until it hits the end of the tape. 
> So that's why you don't get "interrupted writing" in the sense of having
> mid-tape write errors returned by the tape device the computer.  [But it
> is "interrupted" in the sense that a block takes much longer to write
> than it should so the computer has to wait a long time before it can
> sent the next block of data down to the drive.])
> 
> Hope that makes sense.
> 
>   Nathan
> 
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
> Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
>  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
>  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

On Thu, May 14, 2020 at 09:14:17AM +0200, Stefan G. Weichinger wrote:
> Am 14.05.20 um 07:59 schrieb Jens Berg:
> > There was a discussion back in 2014 with subject "Backups to tape
> > consistently under 60% tape capacity". I haven't read the whole lengthy
> > thread but one participant mentioned that in his case a bad cleaning
> > tape was found to be responsible for the capacity loss. Others reported
> > that they had trouble with the compression settings. My gut feeling
> > tells me that this is not the problem here but maybe your impression is
> > a different one or you get a new idea if you read through the postings.
> 
> Thanks for the pointer.
> 
> Interesting, how can a "dirty" drive trigger this behavior?
> 

Because somewhere along the line, some piece of the software does
not distinguish an error return as distinct from an EOF or EOM.
By the time it gets to the human, it is an EOF even if it was an
error.

For example, some writing functions return only success/failure.

Is that failure the end of tape or some other error?  Does the code
assume one and not investigate further?  If the code does investigate
further, is there a way to tell the upstream code success/EOF/error
or just two statuses?

> But it's worth a try, the customer for sure hasn't used the cleaning
> tape for years.

Sounds like a plan.  Maybe two runs?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

On Wed, May 13, 2020 at 06:17:02PM +0200, Stefan G. Weichinger wrote:
> Now look at this run of
> 
> "amtapetype -b 128k /dev/nst0"
> 
> with another tape, FUJI instead of HP:
> 
> define tapetype LTO3-fuji {
> comment "Created by amtapetype; compression disabled"
> length 284180096 kbytes
> filemark 20803 kbytes
> speed 38376 kps
> blocksize 128 kbytes
> }
> 
> ~290 GB ... faster, and large filemarks.
> 
> Maybe that drive is somehow failing .. ?

I wasn't sure what LEOM was.  I assume it is "Logical End of Media".

Anyway I came across two references that said need for cleaning
is one reason for getting early EOM.

I'm wondering also if this could be a case of Amanda tapes being
labelled with the mode set to LTO-2 capacity.  I know you check
the mode and it shows 44, but Amanda always reads the tape before
writing.  Could this be setting the mode back to 42 because the
tapes were initially labelled with the mode set incorrectly?

What if you forget about amtapetype and simply use dd to see how
much random data it will write to tape.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: smbclient

[Jon LaBadie]

On Tue, Apr 21, 2020 at 01:06:13PM +0100, Nuno Dias wrote:
>  Hi,
> 
>  I tried everything, I'm using the Administrator user in windows 10, I
> check and he has the rights to do everything, nevertheless in smbclient
> I have several ERRORS saying "NT_STATUS_ACCESS_DENIED listing" this is
> system files or system directories.
> 
>  The backup I want to do is the users directories, and I can read that
> directories and files, it seems amanda fails because of the previous
> ERRORS.
> 
>  Even if I put something like this //pcwindows/c$/Users/user/files and
> I'm positive that I can read these dir and files, I still have error
> in //pcwindows/c$/Progranas
> NT_STATUS_ACCESS_DENIED listing \Programas\*
> 
>  And this is not in the PATH of the backup :(

Putting on my "way back hat".  I recall problems backing up
Windows due to it refusal to allow multiple processes to
access the same file simultaneously.  The work-around that
M$ introduced was called, I think, Volume Shadow Copy (VSC?).

The Zmanda windows client (again I think) uses VSC to
overcome this deficiency in M$ file systems.

Might smbclient be bumping up against this M$ limitation?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: balance calculation [Experiment Results Report]

[Jon LaBadie]

In February I expressed concern that when scheduling and promoting,
amanda's planner does not properly consider DLEs with longer than
default dumpcycles.

In my case I had a default dumpcycle of 1 week.  But more than
half my DLEs had custom dumpcycles of 2, 3, or 4 weeks.  When
looked at as "what does the full dump schedule look like for
the next 28 days" using "amadmin DS1 balance --days 28", there
were days with NO level 0s scheduled and others with 5 and 7 
large DLEs schedules totalling up to 4 times the "balance".

Sure, the "no scheduled level 0" days will get some promoted
dumps, but the monster dump days never seemed to go away.

This is not a recent configuration/installation.  There have
been years to get the balance adjusted to a reasonable state.

As described in the Feb 25 message below, I lengthed my default
dumpcycle and runspercycle to 28 days.  How I did it is described
in the message but is not recommended except for an experiment
like this.  After the included February message I describe the
results of the experiment.


On Tue, Feb 25, 2020 at 05:32:28PM -0500, Jon LaBadie wrote:
> In a recent discussion on widely out of balance dumps and
> frequent promotions to level 0 dumps I noted this about
> my configuration.
> 
> On Thu, Feb 13, 2020 at 11:50:34PM -0500, Jon LaBadie wrote:
> ...
> > 
> > My default {dump/runsper}cycle is 7 days, set in amanda.conf.
> > But more than 1/2 of my DLEs have custom dumpcycles (14,21,and
> > 28 days) set in their dumptypes.  You can not set custom values
> > for runspercycle in dumptypes (bummer!!).
> > 
> 
> I decided to give amanda's planner a different scenario, long
> dumpcycles and daily runspercycle to see if that would have an
> effect on the balances and promotions.  It seems to be working.
> 
> Using "amadmin  balance", the report shows the
> expected high balance over the next 28 days only exceeds 57%
> one time.  On that one date, 5 DLEs are scheduled to dump 350%
> of the balanced amount.  But the 3 days before that are well
> under balance, so perhaps the planner will promote some of
> the 5 DLEs.
> 
> Here are the changes I made.
> 
> In "amanda.conf", dumpcycle and runspercycle were both changed
> from 7 to 28 days.
> 
> In "disklist" I had many entries that defaulted to 7 day cycles
> like:
> 
>cyber.jgcomp.com  Root/comp-root-tar
> 
> and many othere entries with longer dumpcycles like:
> 
>cyber.jgcomp.com  Vault-Monthly/opt/vault-monthly {
>comp-user-tar
>dumpcycle 28
>}
> 
> I made NO changes to disklist.  Thus it seems like Root above
> would get the default 28 day dumpcycle rather than the desired
> 7 day.  To fix this I edited the "dumptypes" file.  The dumptype
> "global" is incorporated into nearly every other dumptype so
> I added a dumpcycle line.
> 
>define dumptype global {
>comment "Global definitions"
> index yes
> record yes
> auth "bsdtcp"
> # added jhl 2/13/2020
> dumpcycle 7
>}
> 
> So now, DLE Root gets its dumpcyle from the global dumptype
> (7 days), Vault-Monthly overrides the global value to 28 days,
> and amanda's planner uses the default value from amanda.conf,
> also 28 days.
> 
> The primary downside is an administrative one.  If someone
> adds a new DLE they might expect it to get the default from
> amanda.conf when that is actually overridden by the global
> dumptype.
> 
> If I see any surprise effects from these changes I'll report
> back.  But so far (12 days) it has been fine.
> 
> Jon


Well, not a hick-cup.  Ran smoothly the entire 2 months since
I made the changes.  As you will see in the 28 day schedule
that follows, there are no days with zero full dumps scheduled
and the largest days are only 2x the balance size.

$ amadmin DS1 balance

 due-date  #fsorig MB out MB   balance
--
 4/14 Tue1 107003 106124+59.5%
 4/15 Wed3  76638  65826 -1.0%
 4/16 Thu3  80792  51735-22.2%
 4/17 Fri4 192853 133781   +101.1%
 4/18 Sat7 110353  88447+33.0%
 4/19 Sun2  79663  58196-12.5%
 4/20 Mon1 123755  97200+46.1%
 4/21 Tue1  63332  51495-22.6%
 4/22 Wed3  76638  65826 -1.0%
 4/23 Thu3  80792  51735-22.2%
 4/24 Fri3 120982  61484 -7.6%
 4/25 Sat6  38743  17041-74.4%
 4/26 Sun1  79663  58196-12.5%
 4/27 Mon1 123755  97200+46.1%
 4/2

Re: balance calculation -- was Re: separate spindle for dump use

[Jon LaBadie]

In a recent discussion on widely out of balance dumps and
frequent promotions to level 0 dumps I noted this about
my configuration.

On Thu, Feb 13, 2020 at 11:50:34PM -0500, Jon LaBadie wrote:
...
> 
> My default {dump/runsper}cycle is 7 days, set in amanda.conf.
> But more than 1/2 of my DLEs have custom dumpcycles (14,21,and
> 28 days) set in their dumptypes.  You can not set custom values
> for runspercycle in dumptypes (bummer!!).
> 

I decided to give amanda's planner a different scenario, long
dumpcycles and daily runspercycle to see if that would have an
effect on the balances and promotions.  It seems to be working.

Using "amadmin  balance", the report shows the
expected high balance over the next 28 days only exceeds 57%
one time.  On that one date, 5 DLEs are scheduled to dump 350%
of the balanced amount.  But the 3 days before that are well
under balance, so perhaps the planner will promote some of
the 5 DLEs.

Here are the changes I made.

In "amanda.conf", dumpcycle and runspercycle were both changed
from 7 to 28 days.

In "disklist" I had many entries that defaulted to 7 day cycles
like:

   cyber.jgcomp.com  Root/comp-root-tar

and many othere entries with longer dumpcycles like:

   cyber.jgcomp.com  Vault-Monthly/opt/vault-monthly {
   comp-user-tar
   dumpcycle 28
   }

I made NO changes to disklist.  Thus it seems like Root above
would get the default 28 day dumpcycle rather than the desired
7 day.  To fix this I edited the "dumptypes" file.  The dumptype
"global" is incorporated into nearly every other dumptype so
I added a dumpcycle line.

   define dumptype global {
   comment "Global definitions"
index yes
record yes
auth "bsdtcp"
# added jhl 2/13/2020
dumpcycle 7
   }

So now, DLE Root gets its dumpcyle from the global dumptype
(7 days), Vault-Monthly overrides the global value to 28 days,
and amanda's planner uses the default value from amanda.conf,
also 28 days.

The primary downside is an administrative one.  If someone
adds a new DLE they might expect it to get the default from
amanda.conf when that is actually overridden by the global
dumptype.

If I see any surprise effects from these changes I'll report
back.  But so far (12 days) it has been fine.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

xattr fail

[Jon LaBadie]

I just added a new client to my dumps.  That client
has some "snap" modules that are causing "strange"
result messages in amreport.  Specifically:

   ? /usr/bin/tar: :
 Warning: Cannot flistxattr:
 Operation not supported

I get the same message if I manually run getfattr on
the file(s) from the command line.

I don't see where the lack of extended attributes on a
source file would be a problem.  Any other thoughts?

Is this message a candidate for inclusion in the list
of ignored gtar messages?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

balance calculation -- was Re: separate spindle for dump use

[Jon LaBadie]

On Wed, Feb 12, 2020 at 05:40:31PM -0500, Nathan Stratton Treadway wrote:
> 
> Looks like there we had some discussion here on the list back in
> November 2018 about the unbalanced runs you were seeing, which got as
> far as my theorizing that perhaps you had a few DLEs that were so much
> larger than all the others that Amanda's normal algorithm for adjusting
> the balance is unable to achieve that goal 
> 
> I don't know if there is anyone left on this list who understands the
> inner workings of those algorithms, but if you really want to get to the
> bottom of the problem you'll probably have to spend some time combing
> through the balance calculation section of the planner log files

Asking if there is anyone left is really the purpose of my post.
Following Gene's original thread I started playing with some
configuration edits and "amadmin  balance".  Here is a
sample of the last few lines of the report:

TOTAL   18 975202 704008100572
BALANCED   573868 311613 44516
DISTINCT2714809641187912
  (estimated 7 runs per dumpcycle)

The last value on the 2nd line above (44.5GB) is what amadmin
considers a balanced output of level 0 dumps over the next
7 day dumpcycle.

My default {dump/runsper}cycle is 7 days, set in amanda.conf.
But more than 1/2 of my DLEs have custom dumpcycles (14,21,and
28 days) set in their dumptypes.  You can not set custom values
for runspercycle in dumptypes (bummer!!).

Based on my playing, I do not think the above command considers
custom dumpcycles in calculating a balance value.  I wonder if
the planner also fails to consider custom dumpcycles?

Where is a code reader when you want one :)

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: holding disk too small? -- holding disk RAID configuration

[Jon LaBadie]

On Mon, Dec 23, 2019 at 11:51:11PM -0500, Gene Heskett wrote:
> On Monday 23 December 2019 21:16:26 Nathan Stratton Treadway wrote:
> 
> > On Sun, Dec 22, 2019 at 11:47:21 -0500, Gene Heskett wrote:
> > > The first, /dev/sda contains the current operating system. This
> > > includes /usr/dumps as a holding disk area.
> > >
...
> 
> Sounds good, so I'll try it.
> 
If the sda DLE(s) are small enough to go direct to "tape",
define all holdings, but run sda DLEs with "holdingdisk no".


> Merry Christamas everybody.

mega-dittos!

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: holding disk too small?

[Jon LaBadie]

On Tue, Dec 10, 2019 at 03:27:13PM +0100, Stefan G. Weichinger wrote:
> Am 05.12.19 um 21:47 schrieb Stefan G. Weichinger:
> > Am 03.12.19 um 15:43 schrieb Stefan G. Weichinger:
> >>
> >> Another naive question:
> >>
> >> Does the holdingdisk have to be bigger than the size of one tape?
> > 
> > As there were multiple replies to my original posting and as I am way
> > too busy right now: a quick "thanks" to all the people who replied.
> > 
> > So far the setup works. Maybe not optimal, but it works.
> > 
> > ;-)
> > 
> > stay tuned ...
> 
> Now an additional obstacle:
> 
> one DLE (a Veeam Backup Dir, so I don't want to split it via "exclude"
> or so) is larger than (a) one tape and (b) the holding disk.
> 
> DLE = 2.9 TB
> holding disk = 2 TB
> one tape = 2.4 TB (LTO6)
> 
> It seems that the tape device doesn't support LEOM ...
> 
> Amdump dumps the DLE directly to tape, fills it and fails with
> 
> " lev 0  partial taper: No space left on device, splitting not enabled
> "
> 
> I am unsure how to set LEOM within:
> 
> define device lto6_drive {
> tapedev "tape:/dev/nst0"
> #device-property "BLOCK_SIZE" "2048K"
> device-property "LEOM" "false"
> }
> 
> define changer robot {
>   tpchanger "chg-robot:/dev/sg4"
>   #property "tape-device" "0=tape:/dev/nst0"
>   property "tape-device" "0=lto6_drive"
>   property "eject-before-unload" "yes"
>   property "use-slots" "1-8"
> }
> 
> 
> ... makes amcheck happy.
> 
> additional for your checking eyes:
> 
> define tapetype LTO6 {
> comment "Created by amtapetype; compression enabled"
> length 244352 kbytes
> filemark 868 kbytes
> speed 157758 kps
> blocksize 2048 kbytes
> 
>   part_size 100G
>   part_cache_type memory
>   part_cache_max_size 8G # use roughly the amount of free RAM on your 
> system
> }
> 
> 
> We have 32 GB RAM in there so this should work?

Perhaps my lack of using large devices is causing me to
miss something, but I don't see how.

You are writing 100GB "parts" directly to tape.  At some
point, the tape fills while writing one of these parts.
To repeat that part on a second tape, the 100GB of the
failed part must be saved somewhere.  Certainly not in
memory!  Can the holding disk be used to "cache" the
parts?

Are you sure you can't just plugin another 2TB USB drive
as a second holding disk?

BTW you do have "runtapes" > 1 correct?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: keep one DLE in the holding disk

[Jon LaBadie]

On Fri, Nov 29, 2019 at 07:29:13PM +0100, Stefan G. Weichinger wrote:
> Am 27.11.19 um 21:22 schrieb Debra S Baddorf:
> > 
> > 
> >> On Nov 27, 2019, at 3:29 AM, Stefan G. Weichinger  wrote:
> >>
> >> There could also be a separate cronjob with "amdump --no-taper" when I
> >> think about it.
> >>
> >> I could run that during the day maybe. This would give me time to run
> >> another script find-ing the latest dump in the holdingdisk etc
> > 
> > Actually, I like this idea a lot.
> 
> Thank you ;-)
> 
> > Create a script like this
> > 
> > amdump —no-taper
> >trigger synch process (or just copy to remote area)
> > 
Speaking of "scripts".  Amanda has the ability to run a
user defined script at many points of a run.  One of those
points is "post-dle-backup".  However, nothing is said about
whether "post...backup" is pre- or post- taping.  If it is
post-taping, I suppose you could just access the vtape,
removing the header block from each tape chunk.

Jon

Do people still read signatures?  Did they ever?
I've emperical evidence no one reads mine.  People
I regularly communicate via email still ask me for
my address or phone numbers.

Now you know where to send my Christmas card ;-))
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: rather simple question around dumpcycle

[Jon LaBadie]

On Sun, Nov 17, 2019 at 12:16:18PM +0100, Stefan G. Weichinger wrote:
> Am 15.11.19 um 20:32 schrieb Jon LaBadie:
> 
> > I was not thinking of any algorithm.  Just a weekend crontab entries like:
> > 
> > amadmin ... force ... ; amdump ...
> 
> Yes, I understood that. But that "manual force" always disturbs the
> schedule built by the amanda algorithm, right?
> 

Would "amdump -o no-record ..." or somesuch correct that?

> At least that was my observation, the force didn't work out reliably, I
> assume because of too much data in relation to the media size and number
> of tapes.

True, whether same or separate config ;)

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Best size for vtapes

[Jon LaBadie]

On Mon, Nov 18, 2019 at 02:48:40PM +0700, Olivier wrote:
> Charles Curley  writes:
> 
> > So in theory you could allocate more vtape space that there is room on
> > the partition. Just make sure you never use more vtape space than you
> > actually have.
> 
> In theory. But you are accepting the risk that from time to time all
> your vtapes will be 100% full and you will not have enough space on your
> disk. And I don't think there is any provision in Amanda to prevent
> that.

Do you really expect all (or most) of your vtapes to be 100% full?  If so,
I do not think you have allocated enough total space.

Amanda has one provision for dealing with such situations, the holding
disk.  Mine is dedicated, and about the size of four vtapes.

Another is "runtapes".  Oh, or do you plan to run exactly the number of
vtapes that you need for your chosen dumpcycle?

> 
> So I prefer to stick with the amount of vtapes equal to the real amount
> of disk space.

Then, from my experience, you will be leaving about 1/3 of your disk empty.

> > The answer to that is, "that depends". I have tried to have a vtape
> > size a bit larger than my typical daily backup, and then allow amanda
> > to use enough extra tapes to cover the largest likely backup. So most
> > days I use one vtape, 40-90% filled. Some days I use three or four
> > vtapes. All but the last are almost 100% filled. You can also play
> > with your split size.

My runs mirror Charles' exactly.  I also configure my DLEs' dumpcycles
according to the data.  Unchanging data (music, photos) get 1 month
cycles, other stuff 1 or 2 weeks.

> I don't think that depends at all unless you have a very deterministic
> usage patern. When the size of the daily backup is truly random, it
> becomes a purely mathematical problem:

Is your backup size really even pseudo random?  Mine, over 40+ years,
at many sites, have never been.

> Each day, you are wasting on average 1/2 vtape amount of disk. So you
> could have vtape being half the size of what you are using, wasting 1/4
> of the initial amount, ...

That is based on the assumption that your tapes match the available space
and your runtapes is 1.  Neither of which ?we?/I recommend.  BTW I just
peeked, my disks dedicated to vtapes, even though substantially over-
subscribed are between 79% and 89% full.

> ... but then you are wasting x blocks overheard for declaring new
> directories and using more inodes. When do both fucntions cross?

First, though an unused inode would be allocated, no inodes would be
wasted.  When you create your file system (assuming extX, ???) space
to a set number of inodes is created.

Second, disks have many millions, even billions of data blocks.  Are
you really worried about using another 1 or 3 for a directory?  You
must have more important thing with which to be concerned.

One last thing, when you create your file system(s) for vtapes you may
be able to control how many inodes are created.  Remember each file
takes only one inode.  A 3TB disk of vtapes on my system only has a
total of 947 files.  Yet there were 350,000 inodes created even though
I changed the mkfs options to greatly reduce them.  Another disk where
I forgot to reduce the number of inodes created has 190,000,000 inodes.

So I'm "wasting" about 20,000,000 data blocks as inodes.  Not enough
for a 100GB vtape, but enough for four 5GB chunks.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

amanda lists at yahoo

[Jon LaBadie]

Do any of our readers or poster use the amanda-users and
amanda-hackers lists at yahoo.com?

Apparently yahoo is making major changes to its groups
program and that will cause a loss of all existing data
at the end of the year.

It may have archival value, seems to have the entire
list back to January, 1997.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: rather simple question around dumpcycle

[Jon LaBadie]

On Tue, Nov 12, 2019 at 11:15:24AM +0100, Stefan G. Weichinger wrote:
> Am 12.11.19 um 00:12 schrieb Jon LaBadie:
> 
> > What about a single config incremental only.
> > 
> > Via crontab, force level 0's when you want,
> > maybe every other weekend FRI & SAT.
> > 
> > The SAT dump serves as level 0's for the incrementals.
> > 
> > The FRI dump can be used for your archiving as needed.
> > Take the tape out of use but retain logs etc.  Add a
> > new tape to keep the same # of tapes in use.
> 
> Well, yes, I tried that already.
> 
> But it was always hard to get the algorithm do lev0 on weekends.
> 
I was not thinking of any algorithm.  Just a weekend crontab entries like:

amadmin ... force ... ; amdump ...

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: rather simple question around dumpcycle

[Jon LaBadie]

On Mon, Nov 11, 2019 at 05:34:14PM +0100, Stefan G. Weichinger wrote:
> 
> At a customer we try to run 2 configs in parallel.
> 
> I discussed this setup here in the past, and bring it back up again ;-)
> 
> -
> 
> We run a daily config with 58 tapes.
> This is run on weekdays 1-4 only.
> 
> Until now we had:
> 
> dumpcycle 2 weeks
> runspercycle 8
> 
> Would it be better to have
> 
> dumpcycle 8 days ?
> 
> -
> 
> The 2nd config runs on FR and SAT: "weekly". With full backups only.
> 
> I want to force lev0 here and archive a set of tapes every 3 months ->
> quartal and year archive sets.
> 
> I have 31 active tapes there right now, a full set of data takes around
> 4-5 tapes.
> 
> What parameters do you users use for such a setup?
> 
> dumpcycle? runspercycle?
> 
> -
> 
> I'd appreciate optimization tips here. It seems my dumpcycles are too
> short so I get issues with too much data dumped per week etc
>>> End of included message <<<

What about a single config incremental only.

Via crontab, force level 0's when you want,
maybe every other weekend FRI & SAT.

The SAT dump serves as level 0's for the incrementals.

The FRI dump can be used for your archiving as needed.
Take the tape out of use but retain logs etc.  Add a
new tape to keep the same # of tapes in use.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

order of "optional" and "append" enforced by amcheck

[Jon LaBadie]

Editing my disklist, and adding an exclude statement, I was
surprised to find amcheck required the "optional" and "append" 
arguments in that order.  By listing them in that order, the
amanda.conf manpage does suggest that order is required.  My
warped mind simply thought of them as options where order is
seldom important.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: bad tape?

[Jon LaBadie]

On Tue, Aug 27, 2019 at 11:44:02AM +0200, Diego Zuccato wrote:
> Il 27/08/19 05:55, Olivier ha scritto:
> 
> > But a disk dedicated to vtapes should be doing a lot of sequetial
> > accesses: once it has been formatted and the slots have been assigned,
> > it is writting files the size of one Amanda's chunk. In fact, that would
> > be worth a study: the disk usage for vtapes vs. normal disk usage.
> That's the *perfect* use-case for SMR drives. But they'd need either no
> filesystem (like tapes :) ) or a dedicated one.
> Is it possible to use raw devices as vtapes?

My guess is that vtapes are expecting normal *nix file
names and system/library calls.

I'm pretty sure it would be possible to write a FUSE (file system
in user space extension) to implement what you imagine.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amrecover as root on client NAK'd

[Jon LaBadie]

On Mon, Jun 24, 2019 at 03:03:35PM -0400, Gene Heskett wrote:
> Greetings all;
> 
> I have a pi running stretch with an rt-preempt kernel so it can run 
> linuxcnc, which according to my tests so far it should do it nicely, its 
> realtime enough to handle servo machines with a 1 kilohertz update rate, 
> jitter is about 29 microseconds.
> 
> But I've followed the output of amcheck and it now reports no errors, 
> although the excludes file is empty, something I'll need to fix.

I thought that was the norm.  The file is there so amcheck is happy.
It doesn't check the contents.  That is so you can define an exclude
file for all DLEs and if unneeded, leave empty.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amrecover/amrestore: block size too small

[Jon LaBadie]

I sent the following note to Steve.  I don't have facilities
to check if my comments are correct.  Anyone care to comment
or check?

On Wed, Jun 19, 2019 at 07:37:23PM +0200, Stefan G. Weichinger wrote:
...
> 
> $ amrestore  --config abt -b 2097152 /dev/nst0 jupi smb_revision
> 
> seems to work now ... at least it starts searching.
> 
> I don't know why I have to tell that ... but it seems I have a mismatch:
> 
> tapetype says 32 kbytes:
> 
> define tapetype LTO-4 {
> comment "Created by amtapetype; compression disabled; 2017-10-31
> sgw"
> length 698510208 kbytes
> filemark 0 kbytes
> speed 36696 kps
> blocksize 32 kbytes
> }
> 
> changer def sets "2 mbytes":
> 
> define changer robot {
> tpchanger "chg-robot:/dev/sg1"
> property "tape-device" "0=tape:/dev/nst0"
> device-property "BLOCK_SIZE" "2 mbytes"
> device-property "READ_BLOCK_SIZE" "2 mbytes"
> property "eject-before-unload" "no"
> property "use-slots" "1-24"
> changerfile "/etc/amanda/abt/chg-robot-dev-sg1"
> }
> 
> storage def pulls in both:
> 
> define storage abt {
> tapepool "abt"
> tapetype "LTO-4"
> tpchanger "robot"
> [..]
> }
> 
> Maybe then amrecover would also work (bailed out before as well)
>>> End of included message <<<

I don't think LTO-4 will use a 32K blocksize.  Minimum is probably
over 1M.  32K in tapetype is just what you said via command line
or letting it default.  Amrestore is probably just reading this
and knowing, or testing, that LT0-4 won't take 32K BS.

Pull off of a tape the first file.  It will be named "0.".
I think it will be 1 tape block in size.  But contain < 32K of data.

Pull off the first data file of a DLE and look at the first 32K.
One of mine ends with this:

To restore, position tape at start of file and run:
dd if= bs=32k skip=1 | /usr/bin/gzip -d | /usr/libexec/
amanda/application/amgtar restore [./file-to-restore]+

The "bs=32K" in my command line may be the BS actually used.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: exact-match

[Jon LaBadie]

On Thu, Jun 13, 2019 at 10:07:10AM -0400, Jean-Francois Malouin wrote:
> * Jon LaBadie  [20190613 05:05]:
> > On Tue, Jun 11, 2019 at 04:28:50PM +0200, C. Scheeder wrote:
> > > Errm
> > > sorry to interup this all,
> > > but isn't the option
> > >  "exclude list "
> > > or
> > >  "exclude file "
> > > man amand.conf dosn't know the option "exclude" without being folowed by 
> > > the
> > > word "file" or "list".
> > > 
> > In amanda.conf(5) it shows:
> > 
> >   exclude [list|file] ...
> > 
> > The square brackets typicaly surround optional arguments.
> > Indeed, the next line shows:
> > 
> >   Default: file.  ...
> > 
> > While I would never recommend it, the file/list argument can
> > be omitted.
> 
> I have done that quite often in the past without an itch. 
> For example, a disklist.conf entry for a DLE, I exclude everything below
> path{1,2,3,4} but not the subdirs themselves:
> 
>  client_hostname /some/dle { 
>app-amgtar-span
>estimate server
>index no
>exclude append "./path1/*"
>exclude append "./path2/*"
>exclude append "./path3/*"
>exclude append "./path4/*"
>  }
> 

Its not that I question whether it will work as advertised.

I just don't trust myself to remember which the default is.
Nor to I want a future maintainer of my configuration to
wonder what that "exclude without file/list" is all about.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: exact-match

[Jon LaBadie]

On Tue, Jun 11, 2019 at 04:28:50PM +0200, C. Scheeder wrote:
> Errm
> sorry to interup this all,
> but isn't the option
>  "exclude list "
> or
>  "exclude file "
> man amand.conf dosn't know the option "exclude" without being folowed by the
> word "file" or "list".
> 
In amanda.conf(5) it shows:

  exclude [list|file] ...

The square brackets typicaly surround optional arguments.
Indeed, the next line shows:

  Default: file.  ...

While I would never recommend it, the file/list argument can
be omitted.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: ***UNCHECKED*** exact-match

[Jon LaBadie]

On Tue, Jun 11, 2019 at 11:47:15AM +0100, Nuno Dias wrote:
>  Thanks, but I'm already doing that (the exclude)
> 
> This is what I have in disklist
> 
> MACHINE   /dir/subdir/name/2019   {
>   tar
>   exclude "./subdir/*"
> 
> }
> 
> MACHINE /dir/subdir/name/2019/subdir tar
> 
>  Because this are very big directories, I want to dump only one every
> time, anyone know how I can do this? 
> 

Two ideas

1) instead of trying to exclude everything you don't want,
consider including the one you do want.  That should auto-
exclude all else.

2) have a separate DLE for each directory you want.  Have
them do only full backups.  Run your amdump in such a way
that only one DLE is dumped each day.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: ***UNCHECKED*** exact-match

[Jon LaBadie]

On Fri, Jun 07, 2019 at 11:04:48AM +0100, Nuno Dias wrote:
>  Hi,
> 
>  I'm trying to use amanda to backup only one dir from a list of dirs
> that are in disklist file
> 
> I run amdump like this
> 
> $ /usr/sbin/amdump  -o reserve=0 --no-taper MACHINE
> ^/dir/subdir/name/2019$
> 
> and with ps I can see the amdump running
> 
> /usr/bin/perl /usr/sbin/amdump  -o reserve=0 --no-taper MACHINE
> ^/dir/subdir/name/2019$
> 
> The problem is instead of only one dir I have two dirs in the backup
> 
> MACHINE:/dir/subdir/name/2019 20190606153859 0   486g dumping
> (18g done (3.74%)) (15:39:26)
> 
> MACHINE:/dir/subdir/name/2019/another 20190606153859 1   244g
> wait for dumping
> 
> 
>  From the amanda-match man page
> 
>   /usr$
>   Will match the disks /usr but not /usr/local.
> 
>  So, this is a bug or I'm doing something wrong?
> 
Not wrong, just a misconception.
The directory you give is a "starting" directory.  amdump will
backup everything under that directory unless it could cross
a mount point (file system) or if something is excluded.

So, assuming you have a DLE for ".../2019" and that your "another"
directory is the only problem, add a directive to the DLE to
exclude another.  If you don't have a DLE for 2019, PERHAPS
the -o option to amdump can be used to add an exclude directive.
I don't know if it can or cannot.

Jon
>  I'm using amanda-3.5.1-16.fc29.x86_64
> 
> thanks,
> Nuno
> -- 
> Nuno Dias 
> LIP


>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: How does amanda invoke gnutar on clients?

[Jon LaBadie]

On Wed, May 29, 2019 at 09:50:50AM +0700, Olivier wrote:
> Winston Sorfleet  writes:
> 
> > I'll try that, thanks!  Out of curiosity, would using "include" get
> > around that? 
> 
> My guess is that it won't. But you can explore the manual of gtar :)
> 
I'm pretty sure that "one-file-system" tells tar that during its
recursion don't cross a mount point to a different filesystem.

But if you "include " you have
given it a second starting point, and it is not crossing a
mount point.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: lev 0 FAILED [data timeout]

[Jon LaBadie]

On Tue, May 28, 2019 at 12:57:18PM +1000, Tom Robinson wrote:
> On Tue, 14 May 2019 at 22:14, Nathan Stratton Treadway 
> wrote:
> 
> Hi Nathan,
> 
> Thanks for you reply and help.
> 
> On Mon, May 13, 2019 at 09:59:13 +1000, Tom Robinson wrote:
.
> 
> I am getting a new issue now which is annoying but not a show stopper. I
> think I will have to revisit my threshold settings to fix this but maybe
> you can offer some insight.
> 
> I have a tape robot and the following settings in amanda.conf
> 
> runtapes 3
> flush-threshold-dumped 100
> flush-threshold-scheduled 100
> taperflush 100
> autoflush yes
> 
> There is enough room for all the data to go on three tapes yet after the
> amdump run is complete only two tapes are written and I am left to flush
> the remaining dumps to tape manually.
> 
> I think it's because I'm trying to get a whole tape's worth of data before
> writing to tape. Is my thinking correct?

With "taperflush 100" you are saying do not write to tape unless
the tape will be filled.  So the third tape, being a partial, is
not written.

However you should not need to manually flush the data, "autoflush yes"
will write the leftover dumps onto the first tape of the next run.
> 
> What I'd like to do is make sure there's a tape's worth of data to write to
> the first two tapes in turn and then dump all remaining backup data to tape
> three (this will not be a complete tapes worth).
> 
> Should I be setting taperflush as follows to achieve this?
> 
> taperflush 0

Yes, from the comments in the sample "amanda.conf".

# You want to improve tape performance by waiting for a complete tape
# of data before writing anything. However, all dumps will be flushed;
# none will be left on the holding disk.
#
# flush-threshold-dumped100 # (or more)
# flush-threshold-scheduled 100 # (or more)
# taperflush0

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: if to fork Amanda

[Jon LaBadie]

On Wed, May 08, 2019 at 12:01:42AM +, Chris Hassell wrote:
> I don't allow a script to alter the repo unless it's in big neon.
> 
...
>
> It does (if I recall correctly) require a user "amandabackup" to exist, but 
> that's all.   The user 'amanda' (?), not so sure if that's made in the build 
> or not.

The original ananda user was "amanda", not "amandabackup".  Some of us
old codgers, who don't have staffers named "amanda"  still prefer the
original user name.  I do it by having two users, amanda and amandabackup
with the same uid/gid.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Dump to tape AND keep copy on disk?

[Jon LaBadie]

On Tue, Jan 15, 2019 at 07:38:37PM +, Debra S Baddorf wrote:
> Amanda people:  does the newer amanda have any means to dump to tape
> while still retaining a copy on the holding disk?
> 
> Dumping to tape, and then vaulting BACK to a disk area seems excessive,
> if there is a better way?
> 
> Deb Baddorf   (asking for Stefan Weichinger)
> 

I've never used it, but ISTR amanda had a RAID feature
where you could tape to multiple devices simultaneously.


jon
> 
> 
> > On Jan 15, 2019, at 1:34 PM, Stefan G. Weichinger  wrote:
> > 
> > Am 14.01.19 um 19:05 schrieb Debra S Baddorf:
> > 
> >> You can let the normal job send the dumps to tape,  and then
> >> “vault”  a copy back onto a “vtape”  on disk  (not your holding disk,
> >> but that’s semantics).
> >> That might suit your need.
> >> 
> >> I’ve done vaulting from a tape onto a disk area …..  moved the
> >> disk area to another node,  and vaulted again to copy it to a
> >> different flavor of tape drive.   So the very first part of my task
> >> might suit you to a tee.
> > 
> > I understand that this might work but it seems to way more steps than
> > having a second "storage" to flush the holdingdisk content to. If that
> > works (and the "storage" doesn't allow to filter the lev0 afaik).
> > 
> > I haven't yet found the time and the brain to continue testing that so far.
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Only increasing incrementals

[Jon LaBadie]

On Wed, Dec 12, 2018 at 02:18:57PM +0700, Olivier wrote:
> Nathan Stratton Treadway  writes:
> 
> > On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
> >> Hello,
> >> 
> >> UI am wondering if there is a way to define a DLE that would all
> >> incrementals but only with increasing level:
> >> 
> >> - full (0)
> >> - incremental 1
> >> - incremental 2
> >> - incremental 3
> >> - etc.
> >> 
> >> But never: 0, 1, 1, 1, 2
> >> 
> >> Each back-up level must be above the previous one or be a full back-up.
> >

Have you tried:

  bumpdays0
  bumpmult0
  bumppercent 0
  bumpsize0

I don't know if it will work, or maybe bork somehow.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amcheck-ing root files

[Jon LaBadie]

On Thu, Nov 29, 2018 at 03:06:17PM +0100, Stefan G. Weichinger wrote:
> 
> I know it was topic here already ... long ago, maybe someone knows a
> solution to this already:
> 
> I have files owned by root that get correctly amdump-ed but throw ugly
> errors/warnings when I run my daily amcheck via cron (to remind admins to
> change tapes if necessary).
> 
> So I get for example:
> 
> Amanda Backup Client Hosts Check
> 
> HOST backup ERROR: backup: Could not access samba_DC_meta
> (/root/dc_stuff/backups): Permission denied
> 
> but the actual backup works
> 
> Can I get rid of that? This floods my inboxes and is noise ...

Are the directories not group root and readable/executable
by group?  Maybe make the amanda a secondary member of the group.

Use acl's to specifically let amanda user access.

jl

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Reusable

[Jon LaBadie]

On Wed, Nov 21, 2018 at 11:55:21AM -0800, Chris Miller wrote:
> Hi Folks, 
> 
> I have written some very small DLEs so I can rip through weeks
> of backups in minutes. I've learned some things.

When I'm experimenting with amanda after I create the
test configuration, before the first amdump run, I
copy or tar the entire configuration.

Then I write a little shell script so that after I've
run a few test dumps, I can delete the entire config
and restore it to the original state from the copy or
tarball.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Reusable

[Jon LaBadie]

On Wed, Nov 21, 2018 at 11:55:21AM -0800, Chris Miller wrote:
> Hi Folks, 
> 
> I have written some very small DLEs so I can rip through weeks
> of backups in minutes. I've learned some things. AMANDA can't
> backup a file; only directories. Is there any way around that?
> Suppose I need to backup a large file in a directory with many
> other large files, so backing up the complete directory or
> moving / copying my target would be inconvenient, not to mention
> administratively cumbersome when it comes to restoring. 

Create a DLE that would backup the entire directory.  Then
add the command "include list " in the DLE and
it will exclude all others.

> 
> I'm also still fighting AMANDA about how to manage vtapes, but I'm developing 
> more focused questions. For example, my test runs are reporting: 
> 

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Only increasing incrementals

[Jon LaBadie]

On Thu, Nov 22, 2018 at 11:18:25AM +0700, Olivier wrote:
> Hello,
> 
> UI am wondering if there is a way to define a DLE that would all
> incrementals but only with increasing level:
> 
> - full (0)
> - incremental 1
> - incremental 2
> - incremental 3
> - etc.
> 
> But never: 0, 1, 1, 1, 2
> 
> Each back-up level must be above the previous one or be a full back-up.

You can use "amadmin  force [host] [disk]"
to force a level 0 dump.

You can use "amadmin  force-bump [host] [disk]"
to force a bump to the next level.

Run them from crontab to automate the process.

Jon
> 
> Best regards,
> 
> Olivier
> -- 
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Taper scan algorithm did not find an acceptable volume.

[Jon LaBadie]

On Mon, Nov 19, 2018 at 09:15:31AM -0800, Chris Miller wrote:
> Hi Folks, 
> 
> Last week I created a set of vtapes {slot0..slot21}, (because I didn't know 
> if zero was used). I labeled each of them with a unique label. I ran amdump 
> for each of three clients and I got semi-successful results, meaning two 
> completed and one left artifacts on the holding disk but never got written to 
> vtape. Later that day, I ran a second amdump on all three clients, but this 
> time all three clients failed. amcheck reports: "Taper scan algorithm did not 
> find an acceptable volume." 
> 
> Clearly I don't understand how to manage vtapes. Or maybe my configuration is 
> wrong. How do I know that AMANDA realizes that there are 21 vtapes available? 
> How do I ask why none of these is acceptable? 
> 
> I ran amoverview, which tells me what I already knew, meaning that I 
> completed one level 0 backup for each client. I ran amreport with much more 
> detailed description of what I already knew. 
> 
> I understand that AMANDA takes great pride managing the tape library to avoid 
> operational catastrophes like re-using a tape which happens to be the only 
> backup. I have zero experience with real tapes, and the tape library model is 
> not helpful for someone like me with no tape library experience trying to use 
> vtape. For example, do I have to label my vtapes, or can I tell AMANDA to 
> label them for me since I don't think I really care what the label says. And 
> then how do I tell AMANDA that it is acceptable to re-use the oldest vtape in 
> the collection? How do I take a vtape out of the rotation and how do I add a 
> new tape to the rotation? 
> 
> Please don't be afraid to answer my questions with a short and sweet 
> reference to an article. There is little point explaining the subtleties to 
> me in e-mail, when a perfectly good explanation already exists. 
> 
As a general approach, an admin should not run new software systems
until its dependices are independently verified.  For example 
client backups depand on a working network.  You should confirm
the clients can reach the server and visa-versa.

You are setting up a vtape library, tapes to fill the library slots,
and a changer to manipulate the vtapes in the library.  You should
exercise these components with amanda commands before trying to
backup your clients.

Hmm, does amanda have any commands to exercise the changer and library?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

tape pools, wrapping my mind around them

[Jon LaBadie]

I currently have 240 Vtapes in a single changer.
Suppose I have 3 "departments", sales, research,
and support.

Could I create 3 dumptype definitions so that
all "sales" DLEs are backed up to a tape pool
consisting of vtapes 1-80, "research" to vtapes
81-160, and "support" to vtapes 161-240?

If so, then a single config could handle keeping
the various data separated while still maintaining
balance etc.  Of course, is the balance on a tape
pool basis or on a total configuration basis?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Now I'm confuzzled about disklist entry's for "./[a-g]*" types

[Jon LaBadie]

On Wed, Nov 14, 2018 at 01:06:48PM -0500, Gene Heskett wrote:
> 
> First a snip/paste of my 3 new del's as I try to split a big one up:
> line entry
> 28 coyote /home/gene/PublicA/ Publicag {
> 29comp-coyote-tar
> 30include "./[a-g]*"
> 31exclude "./[h-z]*"
> 32} 1 local
> 33

Chris already got the syntax error.

I'll just note you do not need both the include
and the exclude lines.  The "include" says these
are the only things to backup.

If "comp-coyote-tar" has any common excludes (like
temp files or ???) the exclude above would replace
them.  If you want to retain the excludes, you may
want to make them "exclude append" to add to any
default exclude.

What if you create a "Gene" or a "123go" or a ".save"
file or directory under PublicA?  Your 3 DLEs would
not back them up.  I usually add a "catchall" DLE.
In your case it might be:


> 28 coyote PublicMisc /home/gene/PublicA/ {
> 29comp-coyote-tar
> 31exclude "./[a-z]*"
> 32} 1 local

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: following the "balance" report

[Jon LaBadie]

On Mon, Nov 12, 2018 at 03:44:04PM -0500, Nathan Stratton Treadway wrote:
> On Mon, Nov 12, 2018 at 15:18:05 -0500, Gene Heskett wrote:
> > > So, if you want to do the deep dive to try to debug that quirk... the
> > > next step will be to look through the
> > >   /var/log/amanda/Daily/amdump.20181112*
> > > log file (assuming the run in question started after midnight this
> > > morning; if not the datetime-stamp would be 2018* of course).
> > 
> > It did, I have some other stuff so it isn't started till about 03:05.
> > 
> > > Find the "ANALYZING ESTIMATES..." section, and cut-and-paste the log
> > > lines that discuss coyote:/var from that section on down through
> > > INITIAL SCHEDULE, PROMOTING DUMPS IF NEEDED, and on through the
> > > GENERATING SCHEDULE section  and we'll see if those lines tell us
> > > anything useful...
> > 
> > Ok but my local prefix is /usr/local, and there is no such text 
> > in /usr/local/tmp*, so the .debug files there are quite small, 15 to 20 
> 
> (Look at the top of the output of "amadmin Daily status"; the very first
> line should be 
>   Using: [PATH...]/amdump.1
> , where [PATH...] is the path to the directory containing the amdump.*
> and log.* files.  [That will be a different directory than the one
> containing the *.debug log files.])
> 

I think you meant "amstatus Daily".

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: following the "balance" report

[Jon LaBadie]

On Mon, Nov 12, 2018 at 06:40:56AM -0500, Gene Heskett wrote:
> amanda@coyote:/root$ /usr/local/sbin/amadmin Daily balance
> 
>  due-date  #fsorig MB out MB   balance
> --
> 11/12 Mon1  32963   7875-53.6%
> 11/13 Tue1   7688   7688-54.7%
> 11/14 Wed2  22109  22109+30.3%
> 11/15 Thu4  75027  46623   +174.9%
> 11/16 Fri4   3506   3506-79.3%
> 11/17 Sat   14  12127   7581-55.3%
> 11/18 Sun4  21281  16842 -0.7%
> 11/19 Mon   14  27513  15343 -9.6%
> 11/20 Tue1  49240  25295+49.1%
> 11/21 Wed   22  24718  16774 -1.1%
> --
> TOTAL   67 276172 169636 16963
>   (estimated 10 runs per dumpcycle)
> 
For personal interest I ran the "balance command" and
got a result I've never seen before:

$ amadmin DS1 balance

 due-date  #fsorig MB out MB   balance
--
11/12 Mon1 207811 207094   +534.4%
11/13 Tue1 135316  92641   +183.8%
11/14 Wed0  0  0  ---
11/15 Thu2  75363  46830+43.5%
11/16 Fri3  52415  22130-32.2%
11/17 Sat1  57080  48442+48.4%
11/18 Sun9  93945  29953 -8.2%
later6 249656 228031   +598.5%
--
TOTAL   17 621930 447090 63870
BALANCED   376137 228508 32644
DISTINCT23 871586 675121
  (estimated 7 runs per dumpcycle)
$

I've never noticed a "later" entry in the list.
I'm sure it is because I have some DLEs with
nearly static data and I give those DLEs long
dumpcycles (3 or 4 weeks).

That also is the explanation for the total DLEs
(#fs) being 17 while there are actually 23 DLEs.
Six of them have dumpcycles longer than 7 days,
my "normal" dumpcycle.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Breaking DLEs up

[Jon LaBadie]

On Thu, Nov 08, 2018 at 09:06:47PM +, Debra S Baddorf wrote:
> Ah, good!   What does “file”  do in your include line?
> Deb
> 
Include (and exclude) can take a first argument of "file" or "list".
With "file" the following string is a "glob" expression.
"file" is the default but I like to specify it anyway.

With "include list" the string that follows is the name of a file
containing "globs", one per line.

You can have multiple "include file" globs if all but the first
are "include file append".  So

  include file "./[a-gA-G]*

could also be specified as

  include file "./[a-g]*
  include file append "./[A-G]*

Jon
> 
> 
> > On Nov 8, 2018, at 2:54 PM, Jon LaBadie  wrote:
> > 
> > On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote:
> >> Yeah, I do use includes,  but I only do a single letter at a time
> >>   include "./a*”
> >> 
> >> Perhaps the problem is with the syntax of doing more than one letter.
> >> I only do   [a-f]   on my excludes.   Weird!
> >> 
> >> Deb Baddorf
> > 
> > I have a working entry that matches the OP.
> > 
> >include file "./201[7-9]*"
> > 
> > Jon
> >> 

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Breaking DLEs up

[Jon LaBadie]

On Thu, Nov 08, 2018 at 08:43:49PM +, Debra S Baddorf wrote:
> Yeah, I do use includes,  but I only do a single letter at a time
>include "./a*”
> 
> Perhaps the problem is with the syntax of doing more than one letter.
> I only do   [a-f]   on my excludes.   Weird!
> 
> Deb Baddorf

I have a working entry that matches the OP.

include file "./201[7-9]*"

Jon
> 
> > On Nov 8, 2018, at 2:33 PM, Cuttler, Brian R (HEALTH) 
> >  wrote:
> > 
> >  
> > Interesting, not sure.
> >  
> > For part 2, I will say that it is far easier to exclude files from backup 
> > than include them. You had done an excellent job of exclusion, you’ll 
> > pardon the poor attempt at humor, it is getting late in the day.
> >  
> >  
> > From: Chris Nighswonger  
> > Sent: Thursday, November 8, 2018 3:21 PM
> > To: Cuttler, Brian R (HEALTH) 
> > Cc: amanda-users@amanda.org
> > Subject: Re: Breaking DLEs up
> >  
> > ATTENTION: This email came from an external source. Do not open attachments 
> > or click on links from unknown senders or unexpected emails.
> > 
> > On Thu, Nov 8, 2018 at 1:56 PM Cuttler, Brian R (HEALTH) 
> >  wrote:
> >  
> > Your syntax
> >  
> > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" {
> >   comp-tar
> >   include "./[a-f]*"
> >   estimate server
> > }
> >  
> > my syntax
> >  
> > finsen  /export/home-A /export/home   {
> > user-tar2
> > include "./[a]*"
> > }
> >  
> > finsen  /export/home-AZ /export/home   {
> > user-tar2
> > include "./[A-Z]*"
> > }
> >  
> >  
> > Well, this fixes my problem, though why I do not know.
> >  
> > fileserver CAMPUS_a-f /netdrives/CAMPUS {
> >   comp-tar
> >   exclude file "./[g-z]*"
> >   estimate server
> > } 1
> >  
> > It seems a bit of work compared to the include directive. I tried "include 
> > file" to no avail.
> >  
> > I'll see how the backup runs tonight, but amcheck likes it.
> >  
> > Kind regards,
> > Chris
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Can AMANDA be configured "per client"?

[Jon LaBadie]

On Thu, Nov 08, 2018 at 11:23:44AM -0800, Chris Miller wrote:
> Hi Folks, 
> 
> My thanks to the guys that helped me. You know who you are -- Austin, Gene, 
> Stefan, Nathan. 
> 
> My problem with vtapes was that the name of the slots, "slot..." was not 
> arbitrary, or at least not arbitrary on my part. I fixed that, and backups 
> are happening as we speak. Beers all 'round. I now have a bit of breathing 
> room, and I can review ALL the "HowTo"s, which is a pretty good place to 
> start. 
> 
> So, now I have a couple of "clean-up" questions to conclude this thread: 
> vtape labels: Are these just an artifact of the tape heritage, meaning, How 
> is the label any more restrictive/protective than the path to and filename of 
> the vtape? It's not like you can inadvertently mount the wrong directory, is 
> it?Well, actually, in the case of my NAS configuration, I guess that is 
> possible, but unlikely except in the case of some sort of NAS failure and 
> recovery. Is there a discussion somewhere describing how these are used and 
> what sort of failures they can prevent in the vtape world? 
> 

It is more restrictive/specific than a path to the vtape.
Some sites use portable drives and rotate them off-site.
So the same "slots" are used for many different vtapes.

My own setup has /amanda on the root filesystem.
Under /amanda are directories "Holding", "Storage", and
Slots.  A dedicated disk is mounted on "Holding".
Under Storage are six directories "D1" to "D6".  My six
dedicated vtape storage disks are mounted there.

Slots contains mostly symbolic links to the vtapes under
Storage.  230+ vtapes whose naming is DS1-### and matches
the numeric part of the Slot name.  I.e. slot28 is a
symbolic link to /Storage/D1/DS1-028.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Breaking DLEs up

[Jon LaBadie]

On Thu, Nov 08, 2018 at 01:24:46PM -0500, Chris Nighswonger wrote:
> No question is stupid. I learned that beating my head against the wall for
> long hours. :-)
> 
[snip]
> >
> >
> > Here is the relevant portion of my DLEs:
> >
> > fileserver "/netdrives/CAMPUS/af" "/netdrives/CAMPUS" {
> >   comp-tar
> >   include "./[a-f]*"
> >   estimate server
> > }

One difference I noticed.  Deb, Brian, and myself all use
"disknames" that are not pathnames.  I've not heard of pathnanmes
causing a problem, but you might consider changine /netdrives/CAMPUS/af
to something like CAMPUSa-f.

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: dumporder

[Jon LaBadie]

On Tue, Nov 06, 2018 at 01:18:15PM -0500, Chris Nighswonger wrote:
> This seems to work:
> 
> amplot /var/backups/campus/log/amdump.1
> 
> Running under the amanda user.
> 
> However, the issue now is the attempt to write the output to the X11 terminal:
> 
> 
> gnuplot: unable to open display ''
> gnuplot: X11 aborted.
> 
> Not sure what all that's about. So I'm doing a bit of hacking on the
> gnuplot script to have it write the results out to a png file.
> 
You have the permissions on your X screen restrictive (as they
should probably be).  Thus if you logged in a nathan and ran
a program that needs the screen it cannot.

The "xhost" command controls the access permissions.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: New Installation

[Jon LaBadie]

On Fri, Nov 02, 2018 at 10:46:09AM -0700, Chris Miller wrote:
> Hi Folks, 
> 
> I'm working carefully though " [ 
> https://www.zmanda.com/quick-backup-setup.html | The 15-Minute Backup 
> Solution ] " It is pretty good, but it confuses me a bit and it has raised 
> questions. 
> 
> I am in the first client setup section, "Iron". I'm configuring a "tapedev". 
> The pdf version references this as Section 5.1 and the HTML version doesn't 
> mention it at all. I am instructed to edit /etc/amanda/amanda-client.conf, in 
> particular I need to define "tapedev".When I was also told to do this on the 
> server for the time when I need to backup elements of the server. I have the 
> same questions in that context. 
> 
> So, I have a few questions: 
> 
> 
> * Why is this definition of "tapedev" in /etc/amanda/amanda-client.conf 
> mentioned in the PDF version, but not in the HTML version? Do I not need to 
> do this yet, or is this an oversight in the HTML version? 
> * Assuming that I do have to do this, am I designating capacity to stage 
> backups, or am I designating an interface of some kind through which backups 
> will flow? 
> 
> Now I'm confused about the behavior of AMANDA . I've read about "staging" 
> backups and how this is a good architecture, and to that end I have installed 
> a big disk in the AMANDA server, but it is not clear to me if this is a 
> benefit to me. I will be backing up to NAS, so if I stage my backups on the 
> server, I need to put everything on the net twice -- once from client to 
> server, and once from server to NAS. Couldn't I just tell the client, "Put 
> your backup there." and designate the appropriate NAS? 
> 
> I'm sure my questions are answered somewhere in some explanation of AMANDA 
> operations, but I haven't stumbled across this explanation yet, so if you 
> have one, I'd be most appreciative. 
> 
> Thanks for the help, 
> -- 
> Chris. 

I think the 15 min install doc is dated and not or poorly maintained.

amanda-client.conf is a "relatively" recent addition.  In there tape-dev
can be defined to allow recovery of backups on the client machine.  You
need to tell it the server and which device to recover from.  For small
installations like mine, it probably is not needed.  And if you recover
on the server then transfer to the client, it is not needed either.

By staging, I presume you refer to the "holding disk".  It does have
several benefits, particularly if going to real tape.  Amanda was
developed for tape backup and later added "virtual tapes".  You can
backup directly to the storage device, but as tape drives can only
handle one stream at a time, you can only be backing up 1 DLE.  With
a holding disk you can backup multiple DLEs simultaneously.

Some sites use a separate network from the amanda server to the NAS.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: All level 0 on the same run?

[Jon LaBadie]

On Wed, Oct 31, 2018 at 08:25:39AM -0400, Chris Nighswonger wrote:
> So, looking at this more, it may be self-inflicted. Last week I changed
> blocksize to 512k, and began amrmtape and amlabel with the oldest tape
> first and working backward day by day. I run backups 5 nights per week with
> a cycle of 13 tapes (see below). I would have thought that this would have
> allowed the change in blocksize to run seamlessly. Maybe not. I'm now
> suspecting that by amrmtape --cleanup, this caused Amanda to bork and fall
> back to level 0 backups. She did this two nights in a row!!!
> 
> Anyway, I'm going to hold off any further concerns until I finish a
> complete tapecycle. If the problem continues after that point, I'll pick
> back up.
> 
> Relevant lines from amanda.conf:
> 
> dumpcycle 5 days
> runspercycle 5
> tapecycle 13 tapes
> runtapes 1
> flush-threshold-dumped 50
> bumpsize 10 Mbytes
> bumppercent 0
> bumpmult 1.5
> bumpdays 2
> 
> Kind regards,
> Chris
> 

When I introduce a lot of DLEs into the disklist (or start a
new amanda instance) I typically will comment out all the new
DLEs and only uncomment a few each amdump run.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: temporarily remove a system

[Jon LaBadie]

On Sun, May 20, 2018 at 04:47:36AM -0400, Gene Heskett wrote:
> On Sunday 20 May 2018 02:55:31 Jon LaBadie wrote:
> 
> > I have one system that will be down for an unknown time.
> >
> > How best to stop amanda from trying, and failing, to
> > back up the DLEs of that system.  Is it as simple as
> > commenting out the DLEs?  Then if I do need to recover,
> > or it comes back on line, uncomment the entries?
> >
> Thats how I've been doing it for decades.

Me too, though that has typically been for fairly short
times.  At least short relative to my tape cycle.

I'm going to have to look at some of those "new-fangled"
amanda commands like amvault or am??? and see if I can
save about the last months' worth of dumps from that
system.  Maybe store it in some separate data structure
than my current vtape collection.  Then it would not
matter if they get overwritten.

Suggestions?

jl

> > Alternatively, is there some disktype entry that says
> > basically "never backup this DLE unless forced with
> > amadmin".
> >
> > Jon
> 
> 
> 
> Copyright 2018 by Maurice E. Heskett
> -- 
> Cheers, Gene Heskett
> --
> "There are four boxes to be used in defense of liberty:
>  soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> Genes Web page <http://geneslinuxbox.net:6309/gene>
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: temporarily remove a system

[Jon LaBadie]

On Sun, May 20, 2018 at 08:44:03AM -0600, ghe wrote:
> On 05/20/2018 12:55 AM, Jon LaBadie wrote:
> > I have one system that will be down for an unknown time.
> > 
> > How best to stop amanda from trying, and failing, to
> > back up the DLEs of that system.  Is it as simple as
> > commenting out the DLEs?  
> 
> How about turning the cron job off and on?
> 

There are still 25 other DLEs to back up.
I guess I could modify the cronjob to specify the
hosts to backup.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

temporarily remove a system

[Jon LaBadie]

I have one system that will be down for an unknown time.

How best to stop amanda from trying, and failing, to
back up the DLEs of that system.  Is it as simple as
commenting out the DLEs?  Then if I do need to recover,
or it comes back on line, uncomment the entries?

Alternatively, is there some disktype entry that says
basically "never backup this DLE unless forced with
amadmin".

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: bump* dumptype parameters tuning

[Jon LaBadie]

On Mon, Apr 09, 2018 at 01:20:05PM -0400, Jean-Francois Malouin wrote:
> Hi,
> 
> It's with some shame that I must admit that through the long years of using
> amanda I've never really played with the bump* parameters for DLEs!
> 
> I'm now in a situation that might benefit from some tuning using those
> to aggressively encourage bumping the gnutar incremental levels.  The
> drawback of potentially needing more tapes for recovery is not really a
> constraint for me.
> 
> I have a DLE (~5TB) for which the estimates are done on the server --
> the client, a NAS/filer, doesn't support anything else, so the DLE usage
> info is really just from the historical usage not the current one. Any
> ideas/hints on how should I proceed from what is currently (default
> values I guess) configured?
> 
> bumppercent 20
> bumpsize20480K
> bumpdays1
> bumpmult4.0
> 
> Would setting bumpdays to higher values (2 or 3 days) pull in the
> direction I want to go?
> 
> Since bumppercent != 0, bumpsize is ignored from the man page, so I
> guess the only important parameter left is bumppercent. Should I
> increase it or decrease it?  I'm having a hard time understanding what
> the doc says for amanda.conf says...

As you note, your bumpsize is ignored.

The bumpdays parameter gives the minimum number
of dumps at each level that must be done before
the next higher level is even considered.

bumpdays is often recommended to be set to > 1.
The rationale being that if you have two level 1
dumps, one could be lost without much concern.
But if you only have a single level 1, losing it
could be a big problem.

I question your multiplier in "bumpmult  4.0".
It says to go from level 1 to level 2 the level 2
needs to be 20% smaller than the level 1.  But
to go to level 3, the size of the expected level 3
dump must be 80% (4.0 x 20%) smaller than the
expected size of a level 2.  And it will be really
hard to get to level 4 as it must be 320% smaller
than the expected level 3 (4.0 x 80%).

I'd suggest picking a bumpdays with which you are
comfortable, get that multiplier down (1.0? 1.2?),
comment out bumpsize, then see if 20% gives you
what you want, if not start adjusting it.

BTW, if you make a specific dumptype for that
DLE, you can play with the bump settings in that
one dumptype definition without affecting other
DLEs.  They would be using the global values.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: some suggested config parameters for backups to local disk

[Jon LaBadie]

On Sat, Mar 24, 2018 at 08:17:32AM -0400, hymie! wrote:
> On Fri, Mar 23, 2018 at 05:26:39PM -0400, Jon LaBadie wrote:
> > > On 2018-03-23 08:25, hy...@lactose.homelinux.net wrote:
> > > > "Ryan, Lyle (US)" writes:
> > > > 
> > > > > The server has an 11TB filesystem to store the backups in.  I should
> > > > > probably be fancier and split this up more, but not now.   So I've 
> > > > > got my
> > > > > holding, state, and vtapes directories all in there.
> > > > 
> > > > In this scenario, I would think there's no point to a "holding" disk.
> > > > 
> > 3. If something happens to the data storage device(s), the holding
> > disk (HD) can continue to collect your backups.  My HD is big
> > enough to hold about 4 typical runs.  Should the storage outage
> > be protracted and HD space gets low, amanda switches to "degraded"
> > mode and only does incrementals.
> 
> Umm ... we aren't talking about the benefits of a holding disk.  We're
> specifically talking about the benefits of a holding disk **on the same
> device** as the virtual tapes.
> 
Having overlooked that, apologies.

Looking back at the three benefits mentioned, all would
apply with holding disk on same device.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: some suggested config parameters for backups to local disk

[Jon LaBadie]

On Fri, Mar 23, 2018 at 09:40:34AM -0400, Austin S. Hemmelgarn wrote:
> On 2018-03-23 08:25, hy...@lactose.homelinux.net wrote:
> > "Ryan, Lyle (US)" writes:
> > 
> > > The server has an 11TB filesystem to store the backups in.  I should
> > > probably be fancier and split this up more, but not now.   So I've got my
> > > holding, state, and vtapes directories all in there.
> > 
> > In this scenario, I would think there's no point to a "holding" disk.
> > 
> > I use a holding disk because my actual backup disk is external-USB and
> > (comparatively) slow.  So I backup to a holding disk on my internal
> > SSD, releasing the client and the network as soon as possible, and then
> > copy the backup to the backup drive afterwards.  But in your case, I
> > don't see any benefit.
> There are two other benefits to having a holding disk:
> 
> 1. It lets you run dumps in parallel.  Without a holding disk (or some
> somewhat complicated setup of the vtapes to allow parallel taping), you can
> only dump one DLE at a time because it dumps directly to tape.
> 
> 2. It lets you defer taping until you have some minimum amount of data ready
> to be taped.  This may sound kind of useless when working with vtapes, but
> if the holding disk is on the same device as the final vtape library,
> deferring until the dumps are all done (or at least, almost all done) can
> help improve dumping performance, because the dump processes won't be
> competing with the taper process for disk bandwidth.
>>> End of included message <<<

3. If something happens to the data storage device(s), the holding
disk (HD) can continue to collect your backups.  My HD is big
enough to hold about 4 typical runs.  Should the storage outage
be protracted and HD space gets low, amanda switches to "degraded"
mode and only does incrementals.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Amtapetype wrongly reporting compression?

[Jon LaBadie]

On Fri, Jan 12, 2018 at 03:20:51PM -0500, Luc Lalonde wrote:
> Hello Gene,
> 
> Would a variant like this:
> 
>     /usr/sbin/mtx load 1 0;
>     /usr/bin/mt -f /dev/st0 compression 0;
>     /usr/bin/mt -f /dev/st0 setblk 524288;
>     /usr/bin/dd if=/dev/zero of=/dev/st0 bs=512k count=1;
>     su amandabackup -c "/usr/sbin/amlabel -f Monthly-LTO7 000321L7 slot
> 1";
>     /usr/sbin/mtx unload 1 0;
> 
> work?
> 
> During my tests with this, the hardware compression stays disabled when I
> load a new tape.
> 
The critical event is whether it remains disabled after the
amanda program opens the tape drive for reading (or R/W).
All amanda programs that deal with the tape drive do a read
of the tape header before doing anything else.

> Thanks!
> 
> 
> On 2018-01-11 10:51 PM, Gene Heskett wrote:
> > 1. rewind the tape.
> > 1a. Do NOT remove tape from drive, or cause it to read the tape other
> > than what I write here until after step 5.
> > 2. read the label out to a 32k file.
> > 3. rewind the tape.
> > 4. Turn the compression off, probably with mt.
> > 5. Immediately re-write that label while the tape is rewound, and the
> > hidden tape id block in front of the label will get rewritten too, with
> > that compression flag off.
> 
> -- 
> Luc Lalonde, analyste
> -
> Département de génie informatique:
> École polytechnique de MTL
> (514) 340-4711 x5049
> luc.lalo...@polymtl.ca
> -
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

ambackup -- usage message error

[Jon LaBadie]

The usage message of ambackup(8) shows "dump" as a
valid command.  It should show "backup" instead.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: First run...

[Jon LaBadie]

On Fri, Jan 05, 2018 at 11:23:49AM -0800, Chris Miller wrote:
> Hi Jean-Louis, 
> 
> | runtapes 1  # number of tapes to be used in a single run of amdump
> 
> | Because you tell it to use one tape.
> As advertised, Amanda behaviors that I don't yet understand... 
> 
> So, I suspect that vtape behavior is a big piece of Amanda and from this low 
> point on the learning curve, it looks formidable. Is there some document that 
> describes it sufficiently that I don't have to make a lot of mistakes and ask 
> a lot of stupid questions? 
> 
> So, having been so helpful with that question, I have two more: 
> 
> 1. If I am backing up a client to a NAS appliance, then I don't see how 
> it makes sense to have a holding disk, so I think I want to configure that 
> out. Am I right? 

No you are not.

Amanda does not just store backups on random access devices.  Some
amanda sites still use real tape.  The core sections of amanda are
written to be storage device agnostic.  With real tape, only one
DLE can have its backup being written to the storage device at a
time.  If a backups are written directly to the storage device each
DLE will have to wait its sequential turn.

In my small environment, 6 hosts and 25 DLEs, that could take as long
as 10+ hours ('Dump Time' in the daily report).  Yet the longest the
entire run has taken has been 3.5 hours ('Run Time').  That is the
first benefit of a local random access device, called a holding disk,
to collect DLE dumps before sending them to the storage device.  The
holding disk can be collecting multiple simultaneous dumps, even while
sending a previously completed dump to the storage device.

I configure my setup to allow a maximum of 4 simultaneous dumps, not
more than 2 from the same host or disk spindle.

Perhaps, though I doubt it, you are fortunate enough to never have
your connection to the NAS down.  That is a second benefit of a
holding disk.  Where do you put backups when the storage device is
not available?  My holding disk is sufficiently large to save 2-5
days worth of normal backups.  For extended outages, when holding
disk space gets tight, amanda switches to doing only incrementals.

BTW the normal amanda recovery tools know if a backup is on the
holding disk or on the storage device and can do recoveries from
either.

So yes, you can run amanda without a holding disk.  But I wouldn't.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Serious tape error

[Jon LaBadie]

On Fri, Jan 05, 2018 at 08:54:36AM -0500, Jean-Louis Martineau wrote:
> This is an issue in 3.4, 3.4.1 and 3.4.2
> 
> It is fixed in 3.4.3

Ahh yes, I do [now] recall that fix being reported.

Thanks, I will look for an available update.
> 
> Jean-Louis
> 

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda tape algorithm

[Jon LaBadie]

On Thu, Jan 04, 2018 at 08:59:32PM -0500, Ned Danieley wrote:
> On Thu, Jan 04, 2018 at 05:16:59PM -0700, Steven Backus wrote:
> > Ned Danieley  wrote:
> > 
> > > ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> > > the rated capacity would be accurate. I'll give it a try; in the meantime,
> > > has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.
> > 
> > I did and got:
> > 
> > define tapetype LTO6 {
> > comment "Created by amtapetype; compression disabled"
> > length 2442954880 kbytes
> > filemark 7456397 kbytes
> > speed 154519 kps
> > blocksize 32 kbytes
> > }
> 
> thanks. that fairly well matches what I was using
> 
> define tapetype LTO6comp {
>   length 244352 kbytes
>   filemark 868 kbytes
>   speed 157129 kps
>   blocksize 2048 kbytes
>   }
> 
> except for filemark; can anyone comment on that? looking at the tapetype
> definitions on the zmanda wiki, it seems that most of the LTO entries have
> zero kbytes for filemark...
> 
It is the space (if any) left between files written to the tape.

Suppose you write a continuous stream of data to a tape
and you can write exactly 100GB.  If no space was left
between files you should be able to write 100 x 1GB files.
When you try it (as amtapetype does) and you find you can
only write 98 x 1GB files, 2GB was used by "filemarks".
Divide 2GB/98 files and you have your filemark.

I believe Amanda does consider filemarks in its determination
of what will fit on a tape.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Serious tape error

[Jon LaBadie]

On Thu, Jan 04, 2018 at 05:12:27PM -0500, Jon LaBadie wrote:
> I use vtapes of 100GB and runtapes == 3.  Most days only
> one tape is used.  About once every 2 or 3 weeks a second
> tape is needed.  Yesterday was one of those days.  But
> amanda chose the same vtape for the second tape and over
> wrote the data it had just written to the "first" tape.
> 
> >From the report:
> 
>   Tape Size (meg) 179621.1   171549.4 8071.8
> 
> Need multiple tapes.
> 
>   USAGE BY TAPE:
> Label Time Size  %  DLEs Parts
> DS1-215   1:45  179621M  175.42558
> 
> Only one tape listed, filled to 175%.
> 
>   Notes:
> taper: tape DS1-215 kb 104856364 fm 34 [OK]
> taper: tape DS1-215 kb 79075697 fm 24 [OK]
> 
> Tape DS1-215 selected twice.  Only the 24 files from the
> later run are present on the vtape, not the 58 total files.
> The first run was overwritten.
> 
> 
> JLM I've saved all the log files I could find from this run.
> What might you need to analyze this further?
> 
> I looked at /var/log/amanda/server/DS1/taper. and it
> appears to only contain information about the 2nd usage of
> the tape.
> 

I was wrong, there is info from both uses of the same tape in
the taper logfile.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda tape algorithm

[Jon LaBadie]

On Thu, Jan 04, 2018 at 04:58:13PM -0500, Ned Danieley wrote:
> On Thu, Jan 04, 2018 at 09:43:35PM +, Debra S Baddorf wrote:
> > 
> > > On Jan 4, 2018, at 11:59 AM, Ned Danieley  wrote:
> > > 
> > > 
> > > I'm using
> > > 
> > > taperalgo largestfit
> > > 
> > > which I assume means that amanda will write to tape the largest DLE
> > > available. I have 'runtapes' set to 2, and occasionally I'll see amanda
> > > move on to the second tape when the first tape is reporting around 80% 
> > > full.
> > > I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
> > > that would fit in the remaining 20%. is there any way to find out (after
> > > the fact) what DLEs were in the queue? I've looked at the taper debug file
> > > but it just seems to have a list of each DLE as it is written.
> > > 
> > > I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
> > > about the ability to split DLEs, but I'm not ready to take that step.
> > > 
> > 
> > Did you actually test the size of your own tapes? 
> > 
> 
> ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> the rated capacity would be accurate. I'll give it a try; in the meantime,
> has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.

That 2.5TB is likely marketing-speak using decimal number.
Switching to binary TB would immediately reduce the size
by nearly 250GB (about 9% of the listed capacity).

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Serious tape error

[Jon LaBadie]

I use vtapes of 100GB and runtapes == 3.  Most days only
one tape is used.  About once every 2 or 3 weeks a second
tape is needed.  Yesterday was one of those days.  But
amanda chose the same vtape for the second tape and over
wrote the data it had just written to the "first" tape.

>From the report:

  Tape Size (meg) 179621.1   171549.4 8071.8

Need multiple tapes.

  USAGE BY TAPE:
Label Time Size  %  DLEs Parts
DS1-215   1:45  179621M  175.42558

Only one tape listed, filled to 175%.

  Notes:
taper: tape DS1-215 kb 104856364 fm 34 [OK]
taper: tape DS1-215 kb 79075697 fm 24 [OK]

Tape DS1-215 selected twice.  Only the 24 files from the
later run are present on the vtape, not the 58 total files.
The first run was overwritten.


JLM I've saved all the log files I could find from this run.
What might you need to analyze this further?

I looked at /var/log/amanda/server/DS1/taper. and it
appears to only contain information about the 2nd usage of
the tape.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: VTape configuration

[Jon LaBadie]

On Wed, Jan 03, 2018 at 06:22:41PM -0500, Winston Sorfleet wrote:
> On 2018-01-03 05:46 PM, Chris Miller wrote:
> > Hi Jean-Louis,
> >
> > A vtape do not reserve the space, it only use the space of the
> > dump you put on it.
> > The vtape size should be the maximum size of any days or larger,
> > they could be 120GB or 2TG, the result will be the same.
> > Some vtape will use 5GB, some will use 120GB.
> >
> > Thanks very much. That is quite helpful.
> >
> > Is it possible to re-use a vtape for several successive backups? If
> > the vtape were twice the size of a level 0 backup, then the level 1
> > backups would be appended, but the next level 0 would be too large and
> > would trigger a new tape.
> >
> > Additionally, How do I configure larger cycles, meaning, I will use
> > 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that
> > after 13 weeks, she should start over by deleting the oldest vtape?
> 
> That's exactly what it will do, unless you explicitly tell Amanda that a
> tape is out of commission.  So, just like your traditional robot, it
> will cycle.
> 

One parameter you specify is how many tapes (minimum) you have in
rotation.  Amanda will not reuse any tape until it has used that
many tapes.

I presently have 240 vtapes of 100GB each spread over 6 hard disks.
I have specified my "tapecycle" as 140, not 240.  All 240 tapes are
used in sequence and then things rotate back to the beginning.  But
the lower number is my protection against one of the disks failing
and amanda not having anywhere to put new backups.

> HOWEVER, note that Amanda will not write to a tape that has been
> recently written, whether it's 5 GB or 150 GB.  It does /not/ append to
> tapes (barring some painful and unnecessary manual configuration, that
> is).  So, you're better off using small vtapes.  Because if it only
> writes 5 GB to 150 GB in a 2 TB NAS, that remaining 145 GB is
> unavailable to Amanda until that vtape comes up for overwriting.
> 
Perhaps I misunderstand Winston's comment here.  But if you write 5GB
to a "150GB" vtape, the "remaining 145GB" are available for other vtapes.
My 240 x 100GB vtapes (nominally 24TB) are on 20TB of disk.  None of the
six disks are over 80% full.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: VTape configuration

[Jon LaBadie]

On Wed, Jan 03, 2018 at 01:08:37PM -0800, Chris Miller wrote:
> Hi Winston, 
> 
> | I highly suggest a read of this FAQ: [
> | 
> http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
> | ] ; particularly the section about Amanda's planning strategy. If you 
> "insist"
> | on constraining Amanda to one-volume-per-backup, you are basically going
> | against the strategy; without that capability, I don't think that Amanda's
> | overhead gives you anything you can't do with tar and a cron job.
> 
> I understand how Amanda wants to try to "smooth" the mix of backup
> levels and filesystem sizes so that backup costs about the same amount
> of time and storage each cycle, and that is a very worthwhile goal,
> so I don't want to impede that. I also understand that tape discipline
> is already built-in to Amanda at a fundamental level, so I don't want
> to mess with that, either. So, I seek advice.
> 
> Suppose "amanda.example.com" is backing-up "client.example.com" to
> "NAS0.example.com". My level 0 backups are typically 120GB and my level 1
> backups are typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 
> 150GB/week,
> meaning I have sufficient room for thirteen weeks of backup. This seems to me
> like it might be a pretty common scenario and that there might be example
> configs floating around that would size the vtapes for optimal use. Is there 
> one? 

I'm not sure you do understand amanda's approach Chris.  Amanda does not
try to smooth things across a dump cycle, but across each run within a
dump cycle.  If your dumpcycle is 7 days with daily amdump runs, amanda's
goal would be 5GB incrementals plus 120GB/7 (about 18GB) for a daily
dump of about 23GB.

That ideal might be achievable if you had about 14 DLE averaging about
5GB each.  Then amanda could decide which 1,2,3, or 4 DLEs to do level 0
dumps each run.  But in real life you have some small and some large
DLEs and amanda can't meet that ideal.  Assuming you have at least a
few DLEs in those 120GB, some days might have a 60GB level 0 DLE to
dump, other days maybe a 5 and a 10 GB, and still others maybe none
and you will only have incrementals.

In such a scenario I would try to size my vtapes so the largest (60GB
in my example) would fit comfortably in about 3 tapes including a set
of incrementals.  So something like 25GB/vtape and tell amanda it can
use up to 3 tapes per amdump run.  The day that large DLE gets a full
dump, 3 tapes will be used.  Most other days probably only 1 tape will
be needed.

> I have some questions: 

Again, based on the questions below, you don't yet get the amanda scheme.
There will not be A (as in "a single") level 0, each DLE will get its
own schedule of level 0's.

Amanda is nothing if not flexible.  You can force it to do things in a
"non-amanda" way.  My comments after each question are about how you
might achieve the goal, but then why use amanda?  .
> 
> 1. Can I make my vtapes all 150GB, and then instruct Amanda to put
> one cycle (one level 0, and six level 1) on one vtape, meaning re-use a
> vtape multiple time in a backup cycle? I like this approach quite a bit,
> if it is possible. It "packages" one level 0 with all the attendant
> level 1 differentials and eliminates my strongest reservations about
> vtapes -- namely that I don't know where anything is. 

Amanda will not write a new amdump to the end of an existing tape containing
a previous dump.

I've not seen mention of a "holding disk" in this thread.  If you are using
one (strongly recommended) and it is sufficiently large, you could run each
amdump with the option to not write to tape.  The dumps will then collect
on the holding disk.  Then one day a week have your cronjob also do an
amflush which will move all the dumps, full and incr., to one or more vtapes.

You could even do your traditional all level 0's the same day if you
want.  Specify a dumptype of incremental only and on the day you choose
specify a forced full dump.

> 2. Failing that, should I make my vtapes 120GB, so I can fit a
> level 0 backup on one vtape, but then will Amanda truncate level 1 backups,
> so that the vtapes storage requirement is NOT 120G, but more like 5GB? 

You could make a 100 120GB or 150GB vtapes.  They are just directories
and take no space.  Their size is the maximum amount of data amanda will
write to a single vtape.  So if only 10GB get written to each tape it
those 100 tapes will only take 1/2 of your 2TB space on NAS0.

Amanda will not truncate any dumps.  Again assuming you are using a
holding disk, any dumps that don't fit the available vtape(s) will
remain on the holding disk until flushed to a vtape.  This can happen
automatically on the next amdump run.

> 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
> have to span fourteen vtapes for the level 0? This might be optimal use of
> storage, but it scares me with added complexity. I won't know where anything
> is, meaning 

Re: bumpsize vs bumppercent

[Jon LaBadie]

On Sat, Dec 30, 2017 at 03:17:14PM -0500, Jon LaBadie wrote:
> Several "bump" parameters affect how aggressively
> amanda bumps the backup level 0->1, 1->2, etc.
> 
> "bumpsize" and "bumppercent", defaulting to 20MB
> and 20% respectively, seem to have the same purpose,
> provide a threshold of savings before a level bump
> can occur.  But if both are defined they could
> conflict.
> 
> For example, assume estimated dump sizes of 120MB
> for level 1 and 22MB for level 2.  That meets the
> bumpsize threshold, but not the bumppercent.  Or
> estimates of 50 and 15 would meet the percent
> threshold but not the size.
> 
> Does anyone know how the bump- size and percent
> parameters interact?  Must both threholds be met
> or it exceeding one threshold sufficient to
> trigger a level bump (assuming other things are
> met as well, eg. bumpdays)?
> 

One should always RTFM before posting!!

If bumppercent is set to a non-zero value then
it controls the bumping and bumpsize is ignored.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

bumpsize vs bumppercent

[Jon LaBadie]

Several "bump" parameters affect how aggressively
amanda bumps the backup level 0->1, 1->2, etc.

"bumpsize" and "bumppercent", defaulting to 20MB
and 20% respectively, seem to have the same purpose,
provide a threshold of savings before a level bump
can occur.  But if both are defined they could
conflict.

For example, assume estimated dump sizes of 120MB
for level 1 and 22MB for level 2.  That meets the
bumpsize threshold, but not the bumppercent.  Or
estimates of 50 and 15 would meet the percent
threshold but not the size.

Does anyone know how the bump- size and percent
parameters interact?  Must both threholds be met
or it exceeding one threshold sufficient to
trigger a level bump (assuming other things are
met as well, eg. bumpdays)?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda planner and level 0 promotion

[Jon LaBadie]

On Sat, Dec 30, 2017 at 08:33:42AM -0500, Ned Danieley wrote:
> 
> and on the other side of the coin, is there any way to encourage the planner
> to do more level 0 dumps in a given day?
> 
> I occasionally find, often after a kernel update/reboot, that amanda
> thinks that a large number of DLE are 'new', and fairly quickly moves them up
> to level 2. but then the planner leaves them at L2 for days or weeks, at the
> same time using less than 1 TB of my 2.5 TB tape.
> 
> I know that I can force a level 0, but that has to be done on each DLE. it
> seems like there should be a way to tell just amanda to use a larger
> fraction of the tape, but I can't figure it out.
> 

As an ideal, amanda tries to get consistant size dumps
each run of amdump.  As an approximation this size can be
estimated (using post-compression data) as the sum of
all level 0's divided by the number of runs per dumpcycle
plus the average daily level 1 sizes.

So you can get larger runs by three means, get more data
(bigger level 0's), become more active (bigger level 1's)
or shorten the dumpcycle (or runs/dumpcycle).

Sounds like you may want to shorten your dumpcycle.

Note, dumpcycle is a "dumptype" parameter.  I.e. thought
there is a global default value for dumpcycle, new dump-
types can be defined with a different dumpcycle.

In my case I have my photos and my music backed up in
their own DLEs, 5 DLEs in the case of photos.  These are
nearly static DLEs, so no need to back them up each week.
I defined a separate dumptype to have a 3 week dumpcycle
and these 6 DLEs use that dumptype.

You may want to define a dumptype for your more precious
DLEs with a shorter dumpcycle which will cause them to
be dumped at level 0 more frequently.  Or just shorten
your default dumpcycle.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: VTape configuration

[Jon LaBadie]

On Fri, Dec 29, 2017 at 12:28:33PM -0800, Chris Miller wrote:
> Hi Folks, 
> 
> Just setting things up, and I think I ran out of docs... 
> 
> I have no tape drives, so that means "vtape", which is fine, as long as I can 
> size the vtape volume to be no larger than and no smaller than the current 
> backup under consideration. Is this possible? Seems like a silly question, 
> but physical tapes have a specific size and a small backup will result in 
> surplus tape and a large backup will result in multiple volumes. I'd like to 
> avoid both situations. 
> 
> I found very little documentation. In fact there was so little, that I think 
> I saw all of it this morning. I have found: 
> http://wiki.zmanda.com/index.php/User_documentation Is there a hidden archive 
> that I haven't found? 

Welcome to amanda Chris.  Two comments first.

The size of a vtape is its "maximum" size.  A 100GB vtape does not take
up 100GB of disk unless it contains a backup of that size.

Working with multiple vtapes and virtual changers is a breeze in amanda.
Don't try to avoid multiple tape backups.

Suppose you were used to working with 100GB physical tapes.  You may
feel inclined to create 100GB vtapes in amanda.  You may do that and
if your disk is 1TB, you would probably create 10 vtapes.  Explore
2 possibilities.

Using 100GB vtapes, telling amanda to use only 1 vtape per run, you
may discover most (all?) your vtapes are not full.  After you gain
some data from backups you may find your tapes average well under
100% full and you can allocate more vtapes than 10 x 100GB to your
1TB drive.

As a real example, I have 20TB of storage for vtapes and 240 x 100GB
vtapes.  The 6 disks average 75-80% full even though I have 20% more
vtapes than I "should".

Another possibility is to allocate vtapes of 10 (or 20GB) and tell
amanda that it can use up to 10 (or 5) vtapes per run.  Also tell
amanda that DLEs* may be split across multiple tapes using "chunks"
of about 10% or 20% the size of the vtape.

This would be an efficient use of your storage.  A backup totalling
30GB would take 3 vtapes, 90GB would take 9.  Each vtape except the
last would be filled to within 1 "chunk"-size.

Good Luck,
HTH,
Jon

* DLE == Disk List Entry, the unit of backup for amanda.  A DLE
  may be a file system (say / the root fs) or other more complex
  entries (like root but not /var or /opt or /usr/local which are
  backed up in other DLEs
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda is skipping over available tapes [mea culpa]

[Jon LaBadie]

Top-posting as there is no longer anything interesting below.

This was a self-inflicted boondoggle.  When I created the extra
vtapes, the ones that were being skipped, amanda kept reordering
the tapelist file in ways I did not want.  I found if I marked
the new tapes as "no-reuse" (even though they had not been used),
they remained in the positions I wanted.

My plan was to switch them to "reuse" just in time for them to
be used in the my desired sequence.  But I missed the window by
a day and amanda used tape DS1-010 rather than DS1-208.

Until JLM asked about the taperscan algorithm I was unaware that
there were 3 (traditional, oldest, and lexical).  One of the
properties of the lexical algorithm I was using is that it tries
to use the "next" tape, i.e. after 010 then 011 then 012, etc.,
even if older or unused tapes are available.  Thus, even after I
retagged my 40 new vtapes as "reuse" amanda continued to ignore them.

So the problem was not amanda, but the amanda administrator -- ME.


The only thing to come out of this that may need fixing is that
"amadmin  tape" and "amtape  taper" do not use
the same procedures to determine the next tapes to be used and
thus can report different "next tape(s)".  This should probably
be addressed.

Sorry for the noise,

Jon

On Wed, Dec 27, 2017 at 05:43:22PM -0500, Jon LaBadie wrote:
> On Wed, Dec 27, 2017 at 05:26:04PM -0500, Jean-Louis Martineau wrote:
> > Jon,
> > 
> > 'amadmin tape' list the oldest reusable label, it doesn't use the 
> > taperscan algorithm and it doesn't check what is currently loaded in the 
> > changer.
> > 'amcheck' and 'amtape' run the taperscan algorithm and verify they are 
> > in the changer.
> > 
> > Which taperscan are you using?
> 
> I didn't realize there were multiple.
> 
> Just checked and my amanda.conf say "taper_lexical".
> 
> jon
> > 
> > Jean-Louis
> > 
> > 
> > On 27/12/17 05:12 PM, Jon LaBadie wrote:
> > > A while back one of my small older drives containing
> > > vtapes failed.  I replaced it and its twin with larger
> > > drives.  I moved the still good vtapes from the good
> > > twin to the new drives, created empty vtapes for the
> > > lost backups from the crashed drive, and added a good
> > > number of new vtapes.
> > >
> > > However, none of the previously unused vtapes are
> > > being used by amanda and I can not figure out why.
> > >
> > > Here are relevant entries from my current tape list.
> > >
> > > 20171227025501 DS1-018 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > > ...
> > > 20170705025501 DS1-019 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > > 20170627205502 DS1-009 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > > ...
> > > 20170618085502 DS1-209 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > > 20170618025502 DS1-208 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > >
> > > DS1-018 is the vtape it used last night, 019 will be
> > > used tonight.  DS1-208 is at the bottom of the list
> > > and amanda is skipping all 40 tapes between it and
> > > DS1-009.
> > >
> > >
> > > There are 3 ways to check what tapes will be used.
> > > However their results differ.
> > >
> > > $ amanda amadmin DS1 tape
> > > The next Amanda should go onto tape DS1-208 or a new tape.
> > > tape DS1-209 or a new tape.
> > > tape DS1-210 or a new tape.
> > >
> > > $ amanda amtape DS1 taper
> > > slot 19: volume 'DS1-019'
> > > Will write to volume 'DS1-019' in slot 19.
> > >
> > > $ amcheck DS1
> > > Amanda Tape Server Host Check
> > > -
> > > NOTE: Holding disk '/amanda/Holding': 293752 MB disk space available, 
> > > using 293652 MB
> > > slot 19: volume 'DS1-019'
> > > Will write to volume 'DS1-019' in slot 19.
> > >
> > > [ looks the same as amtape ]
> > >
> > >
> > > Any clues as to why the 40+ tapes are not being used?
> > >
> > > Why do amadmin and amtape/amcheck use different
> > > algorithms when looking up the next tape(s) to be used?
> > >
> > > Jon
> > This message is the property of CARBONITE, INC. and may contain 
> > confidential or privileged information.
> > If this message has been delivered to you by mistake, then do not copy or 
> > deliver this message to anyone.  Instead, destroy it and notify me by reply 
> > e-mail
> >>> End of included message <<<
> 
> -- 
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (703) 935-6720 (C)
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda is skipping over available tapes

[Jon LaBadie]

On Wed, Dec 27, 2017 at 06:07:41PM -0500, Jon LaBadie wrote:
> On Wed, Dec 27, 2017 at 05:26:04PM -0500, Jean-Louis Martineau wrote:
> > Jon,
> > 
> > 'amadmin tape' list the oldest reusable label, it doesn't use the 
> > taperscan algorithm and it doesn't check what is currently loaded in the 
> > changer.
> > 'amcheck' and 'amtape' run the taperscan algorithm and verify they are 
> > in the changer.
> > 
> > Which taperscan are you using?
> > 
> 
> Just a bit more info (I already posted "lexical" is my algorithm.
> 
> I did an "amtape DS1 inventory".  It lists all 240 tapes.
> 
> The output has two values in the last column, "[retention-tapes]"
> and "[retention-no]".  However these do not match the in-use vs.
> skipped tapes.
> 
>   DS1-000 to DS1-009 [retention-no] # recreated after crash
> # currently being skipped
>   DS1-010 to DS1-018 [retention-tapes]  # most recently used
> # after looping from 207
>   DS1-019 to DS1-077 [retention-no] # oldest used and I assume
> # will be used next
>   DS1-078 to DS1-207 [retention-tapes]  # used after 077 and
> # looped to 010
>   DS1-208 to DS1-239 [retention-no] # newly created, never used
> # skipped over
Further update,

I changed the taper scan algorthm from lexical to oldest.
This changed the next tape from DS1-019 to DS1-208.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda is skipping over available tapes

[Jon LaBadie]

On Wed, Dec 27, 2017 at 05:26:04PM -0500, Jean-Louis Martineau wrote:
> Jon,
> 
> 'amadmin tape' list the oldest reusable label, it doesn't use the 
> taperscan algorithm and it doesn't check what is currently loaded in the 
> changer.
> 'amcheck' and 'amtape' run the taperscan algorithm and verify they are 
> in the changer.
> 
> Which taperscan are you using?
> 

Just a bit more info (I already posted "lexical" is my algorithm.

I did an "amtape DS1 inventory".  It lists all 240 tapes.

The output has two values in the last column, "[retention-tapes]"
and "[retention-no]".  However these do not match the in-use vs.
skipped tapes.

  DS1-000 to DS1-009 [retention-no] # recreated after crash
# currently being skipped
  DS1-010 to DS1-018 [retention-tapes]  # most recently used
# after looping from 207
  DS1-019 to DS1-077 [retention-no] # oldest used and I assume
# will be used next
  DS1-078 to DS1-207 [retention-tapes]  # used after 077 and
# looped to 010
  DS1-208 to DS1-239 [retention-no] # newly created, never used
        # skipped over

Jon
> Jean-Louis
> 
> 
> On 27/12/17 05:12 PM, Jon LaBadie wrote:
> > A while back one of my small older drives containing
> > vtapes failed.  I replaced it and its twin with larger
> > drives.  I moved the still good vtapes from the good
> > twin to the new drives, created empty vtapes for the
> > lost backups from the crashed drive, and added a good
> > number of new vtapes.
> >
> > However, none of the previously unused vtapes are
> > being used by amanda and I can not figure out why.
> >
> > Here are relevant entries from my current tape list.
> >
> > 20171227025501 DS1-018 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > ...
> > 20170705025501 DS1-019 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > 20170627205502 DS1-009 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > ...
> > 20170618085502 DS1-209 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > 20170618025502 DS1-208 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> >
> > DS1-018 is the vtape it used last night, 019 will be
> > used tonight.  DS1-208 is at the bottom of the list
> > and amanda is skipping all 40 tapes between it and
> > DS1-009.
> >
> >
> > There are 3 ways to check what tapes will be used.
> > However their results differ.
> >
> > $ amanda amadmin DS1 tape
> > The next Amanda should go onto tape DS1-208 or a new tape.
> > tape DS1-209 or a new tape.
> > tape DS1-210 or a new tape.
> >
> > $ amanda amtape DS1 taper
> > slot 19: volume 'DS1-019'
> > Will write to volume 'DS1-019' in slot 19.
> >
> > $ amcheck DS1
> > Amanda Tape Server Host Check
> > -
> > NOTE: Holding disk '/amanda/Holding': 293752 MB disk space available, using 
> > 293652 MB
> > slot 19: volume 'DS1-019'
> > Will write to volume 'DS1-019' in slot 19.
> >
> > [ looks the same as amtape ]
> >
> >
> > Any clues as to why the 40+ tapes are not being used?
> >
> > Why do amadmin and amtape/amcheck use different
> > algorithms when looking up the next tape(s) to be used?
> >
> > Jon
> This message is the property of CARBONITE, INC. and may contain confidential 
> or privileged information.
> If this message has been delivered to you by mistake, then do not copy or 
> deliver this message to anyone.  Instead, destroy it and notify me by reply 
> e-mail
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda is skipping over available tapes

[Jon LaBadie]

On Wed, Dec 27, 2017 at 05:26:04PM -0500, Jean-Louis Martineau wrote:
> Jon,
> 
> 'amadmin tape' list the oldest reusable label, it doesn't use the 
> taperscan algorithm and it doesn't check what is currently loaded in the 
> changer.
> 'amcheck' and 'amtape' run the taperscan algorithm and verify they are 
> in the changer.
> 
> Which taperscan are you using?

I didn't realize there were multiple.

Just checked and my amanda.conf say "taper_lexical".

jon
> 
> Jean-Louis
> 
> 
> On 27/12/17 05:12 PM, Jon LaBadie wrote:
> > A while back one of my small older drives containing
> > vtapes failed.  I replaced it and its twin with larger
> > drives.  I moved the still good vtapes from the good
> > twin to the new drives, created empty vtapes for the
> > lost backups from the crashed drive, and added a good
> > number of new vtapes.
> >
> > However, none of the previously unused vtapes are
> > being used by amanda and I can not figure out why.
> >
> > Here are relevant entries from my current tape list.
> >
> > 20171227025501 DS1-018 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > ...
> > 20170705025501 DS1-019 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > 20170627205502 DS1-009 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > ...
> > 20170618085502 DS1-209 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> > 20170618025502 DS1-208 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
> >
> > DS1-018 is the vtape it used last night, 019 will be
> > used tonight.  DS1-208 is at the bottom of the list
> > and amanda is skipping all 40 tapes between it and
> > DS1-009.
> >
> >
> > There are 3 ways to check what tapes will be used.
> > However their results differ.
> >
> > $ amanda amadmin DS1 tape
> > The next Amanda should go onto tape DS1-208 or a new tape.
> > tape DS1-209 or a new tape.
> > tape DS1-210 or a new tape.
> >
> > $ amanda amtape DS1 taper
> > slot 19: volume 'DS1-019'
> > Will write to volume 'DS1-019' in slot 19.
> >
> > $ amcheck DS1
> > Amanda Tape Server Host Check
> > -
> > NOTE: Holding disk '/amanda/Holding': 293752 MB disk space available, using 
> > 293652 MB
> > slot 19: volume 'DS1-019'
> > Will write to volume 'DS1-019' in slot 19.
> >
> > [ looks the same as amtape ]
> >
> >
> > Any clues as to why the 40+ tapes are not being used?
> >
> > Why do amadmin and amtape/amcheck use different
> > algorithms when looking up the next tape(s) to be used?
> >
> > Jon
> This message is the property of CARBONITE, INC. and may contain confidential 
> or privileged information.
> If this message has been delivered to you by mistake, then do not copy or 
> deliver this message to anyone.  Instead, destroy it and notify me by reply 
> e-mail
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

amanda is skipping over available tapes

[Jon LaBadie]

A while back one of my small older drives containing
vtapes failed.  I replaced it and its twin with larger
drives.  I moved the still good vtapes from the good
twin to the new drives, created empty vtapes for the
lost backups from the crashed drive, and added a good
number of new vtapes.

However, none of the previously unused vtapes are
being used by amanda and I can not figure out why.

Here are relevant entries from my current tape list.

20171227025501 DS1-018 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
...
20170705025501 DS1-019 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
20170627205502 DS1-009 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
...
20170618085502 DS1-209 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1
20170618025502 DS1-208 reuse BLOCKSIZE:32 POOL:DS1 STORAGE:DS1 CONFIG:DS1

DS1-018 is the vtape it used last night, 019 will be
used tonight.  DS1-208 is at the bottom of the list
and amanda is skipping all 40 tapes between it and
DS1-009.


There are 3 ways to check what tapes will be used.
However their results differ.

$ amanda amadmin DS1 tape
The next Amanda should go onto tape DS1-208 or a new tape.
   tape DS1-209 or a new tape.
   tape DS1-210 or a new tape.

$ amanda amtape DS1 taper
slot 19: volume 'DS1-019'
Will write to volume 'DS1-019' in slot 19.

$ amcheck DS1
Amanda Tape Server Host Check
-
NOTE: Holding disk '/amanda/Holding': 293752 MB disk space available, using 
293652 MB
slot 19: volume 'DS1-019'
Will write to volume 'DS1-019' in slot 19.

[ looks the same as amtape ]


Any clues as to why the 40+ tapes are not being used?

Why do amadmin and amtape/amcheck use different
algorithms when looking up the next tape(s) to be used?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: application amgtar ignore messages

[Jon LaBadie]

On Mon, Dec 11, 2017 at 07:30:43AM -0500, Jean-Louis Martineau wrote:
> On 09/12/17 02:11 PM, Jon LaBadie wrote:
...
> >
> > The original question still stands though.  Is it sufficient
> > to list messages for amgtar to ignore on the server even though
> > the amgtar is running on the client?  Or must an ignore be
> > created on the client running the amgtar?
> >
> > Jon
> 
> You can add application property on the server or the client. You do 
> what you prefer.
> 
> Jean-Louis

Thank you.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: application amgtar ignore messages

[Jon LaBadie]

On Fri, Dec 08, 2017 at 07:14:54AM -0500, Austin S. Hemmelgarn wrote:
> On 2017-12-07 22:26, Jon LaBadie wrote:
> > If I want amgtar to ignore certain messages, is it
> > sufficient to list the message on the amanda server
> > or must the ignored message also be listed in
> > amanda-client.conf?
> > 
> > I've done it several times, only on the server, and
> > it seemed to work fine.  But I'm now trying to ignore
> > one message that appears on only one client and I'm
> > having no success.
> > 
> > Do I need to set up an "application amgtar" stanza
> > on the client?
> > 
> > Doesn't affect the question, but the problem is
> > caused by the "gnome virtual file system directory",
> > /home/user/.config/.gvfs.  This is a fuser mountpoint
> > not accessible by root.  So it generates a "can not
> > stat" error message from amgtar.
> > 
> The better approach to this is to add that to the exclude file for that
> particular disk.  It's a well known path, so nothing else should be using
> it, and it's an area that shouldn't be dumped anyway, for a lot of the same
> reasons you shouldn't be dumping /sys or /dev/shm (and in fact, it isn't
> getting dumped, because amgtar can't see inside it).

Thought I had tried that and was still getting "Can not stat"
messages.  At the time I reasoned that amanda had to "stat"
the file before deciding to exclude it.  However, I recently
split /home from the root file system when I added a disk and
I was editing the wrong exclude file.

Before that realization I manually deleted the .gvfs directory
and that "cured" the problem.

The original question still stands though.  Is it sufficient
to list messages for amgtar to ignore on the server even though
the amgtar is running on the client?  Or must an ignore be
created on the client running the amgtar?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

application amgtar ignore messages

[Jon LaBadie]

If I want amgtar to ignore certain messages, is it
sufficient to list the message on the amanda server
or must the ignored message also be listed in
amanda-client.conf?

I've done it several times, only on the server, and
it seemed to work fine.  But I'm now trying to ignore
one message that appears on only one client and I'm
having no success.

Do I need to set up an "application amgtar" stanza
on the client?

Doesn't affect the question, but the problem is
caused by the "gnome virtual file system directory",
/home/user/.config/.gvfs.  This is a fuser mountpoint
not accessible by root.  So it generates a "can not
stat" error message from amgtar.
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: power down hard drives

[Jon LaBadie]

On Mon, Nov 13, 2017 at 02:04:42PM -0500, Gene Heskett wrote:
> On Monday 13 November 2017 13:42:13 Jon LaBadie wrote:
> 
> > On Mon, Nov 13, 2017 at 11:40:17AM -0500, Austin S. Hemmelgarn wrote:
> > > On 2017-11-13 11:11, Gene Heskett wrote:
> > > > On Monday 13 November 2017 10:12:47 Austin S. Hemmelgarn wrote:
> > > > > On 2017-11-13 09:56, Gene Heskett wrote:
> > > > > > On Monday 13 November 2017 07:19:45 Austin S. Hemmelgarn wrote:
> > > > > > > On 2017-11-11 01:49, Jon LaBadie wrote:
> > > > > > > > Just a thought.  My amanda server has seven hard drives
> > > > > > > > dedicated to saving amanda data.  Only 2 are typically
> > > > > > > > used (holding and one vtape drive) during an amdump run.
> > > > > > > > Even then, the usage is only for about 3 hours.
> > > > > > > >
> > > > > > > > So there is a lot of electricity and disk drive wear for
> > > > > > > > inactive drives.
> > > > > > > >
> > > > > > > > Can todays drives be unmounted and powered down then
> > > > > > > > when needed, powered up and mounted again?
> > > > > > > >
> > > > > > > > I'm not talking about system hibernation, the system
> > > > > > > > and its other drives still need to be active.
> > > > > > > >
> > > > > > > > Back when 300GB was a big drive I had 2 of them in
> > > > > > > > external USB housings.  They shut themselves down
> > > > > > > > on inactivity.  When later accessed, there would
> > > > > > > > be about 5-10 seconds delay while the drive spun
> > > > > > > > up and things proceeded normally.
> > > > > > > >
> > > > > > > > That would be a fine arrangement now if it could
> > > > > > > > be mimiced.
> > > > > > >
> > > > > > > Aside from what Stefan mentioned (using hdparam to set the
> > > > > > > standby timeout, check the man page for hdparam as the
> > > > > > > numbers are not exactly sensible), you may consider looking
> > > > > > > into auto-mounting each of the drives, as that can help
> > > > > > > eliminate things that would keep the drives on-line (or make
> > > > > > > it more obvious that something is still using them).
> >
> > ...
> >
> > > > But if I allow the 2TB to be  unmounted and self-powered down,
> > > > once daily, what shortening of its life would I be subjected to? 
> > > > In other words, how many start-stop cycles can it survive?
> > >
> > > It's hard to be certain.  For what it's worth though, you might want
> > > to test this to be certain that it's actually going to save you
> > > energy.  It takes a lot of power to get the platters up to speed,
> > > but it doesn't take much to keep them running at that speed.  It
> > > might be more advantageous to just configure the device to idle
> > > (that is, park the heads) after some time out and leave the platters
> > > spinning instead of spinning down completely (and it should result
> > > in less wear on the spindle motor).
> >
> > In my situation, each of the six data drives is only
> > needed for a 2 week period out of each 12 weeks.  Once
> > shutdown, it could be down for 10 weeks.
> >
> > Jon
> 
> Which is more than enough time for stiction to appear if the heads are 
> not parked off disk.
> 
Don't today's drives automatically park heads?

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: power down hard drives

[Jon LaBadie]

On Mon, Nov 13, 2017 at 11:40:17AM -0500, Austin S. Hemmelgarn wrote:
> On 2017-11-13 11:11, Gene Heskett wrote:
> > On Monday 13 November 2017 10:12:47 Austin S. Hemmelgarn wrote:
> > 
> > > On 2017-11-13 09:56, Gene Heskett wrote:
> > > > On Monday 13 November 2017 07:19:45 Austin S. Hemmelgarn wrote:
> > > > > On 2017-11-11 01:49, Jon LaBadie wrote:
> > > > > > Just a thought.  My amanda server has seven hard drives
> > > > > > dedicated to saving amanda data.  Only 2 are typically
> > > > > > used (holding and one vtape drive) during an amdump run.
> > > > > > Even then, the usage is only for about 3 hours.
> > > > > > 
> > > > > > So there is a lot of electricity and disk drive wear for
> > > > > > inactive drives.
> > > > > > 
> > > > > > Can todays drives be unmounted and powered down then
> > > > > > when needed, powered up and mounted again?
> > > > > > 
> > > > > > I'm not talking about system hibernation, the system
> > > > > > and its other drives still need to be active.
> > > > > > 
> > > > > > Back when 300GB was a big drive I had 2 of them in
> > > > > > external USB housings.  They shut themselves down
> > > > > > on inactivity.  When later accessed, there would
> > > > > > be about 5-10 seconds delay while the drive spun
> > > > > > up and things proceeded normally.
> > > > > > 
> > > > > > That would be a fine arrangement now if it could
> > > > > > be mimiced.
> > > > > 
> > > > > Aside from what Stefan mentioned (using hdparam to set the standby
> > > > > timeout, check the man page for hdparam as the numbers are not
> > > > > exactly sensible), you may consider looking into auto-mounting each
> > > > > of the drives, as that can help eliminate things that would keep
> > > > > the drives on-line (or make it more obvious that something is still
> > > > > using them).
> > > > 
...
> > 
> > But if I allow the 2TB to be  unmounted and self-powered down, once
> > daily, what shortening of its life would I be subjected to?  In other
> > words, how many start-stop cycles can it survive?
> >
> It's hard to be certain.  For what it's worth though, you might want to test
> this to be certain that it's actually going to save you energy.  It takes a
> lot of power to get the platters up to speed, but it doesn't take much to
> keep them running at that speed.  It might be more advantageous to just
> configure the device to idle (that is, park the heads) after some time out
> and leave the platters spinning instead of spinning down completely (and it
> should result in less wear on the spindle motor).
> > 
In my situation, each of the six data drives is only
needed for a 2 week period out of each 12 weeks.  Once
shutdown, it could be down for 10 weeks.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

power down hard drives

[Jon LaBadie]

Just a thought.  My amanda server has seven hard drives
dedicated to saving amanda data.  Only 2 are typically
used (holding and one vtape drive) during an amdump run.
Even then, the usage is only for about 3 hours.

So there is a lot of electricity and disk drive wear for
inactive drives.

Can todays drives be unmounted and powered down then
when needed, powered up and mounted again?

I'm not talking about system hibernation, the system
and its other drives still need to be active.

Back when 300GB was a big drive I had 2 of them in
external USB housings.  They shut themselves down
on inactivity.  When later accessed, there would
be about 5-10 seconds delay while the drive spun
up and things proceeded normally.

That would be a fine arrangement now if it could
be mimiced.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: manual updating tapelist and tape headers after disk crash

[Jon LaBadie]

On Fri, Nov 10, 2017 at 08:18:03PM +, Debra S Baddorf wrote:
> 
> > On Nov 10, 2017, at 2:02 PM, Jon LaBadie <j...@jgcomp.com> wrote:
> > 
> > On Wed, Nov 08, 2017 at 06:10:12PM +, Debra S Baddorf wrote:
> >> 
> >>> Am 2017-11-06 um 19:49 schrieb Jon LaBadie:
> >>>> My vtapes are spread across several disks.  One of the
> >>>> smallest recently died in a most audible way.
> >>>> 
> >>>> I'm replacing the disk with a larger one.
> >>>> 
> >>>> I would like to recreate the lost vtapes (no backup
> >>>> data, that is gone!) and have them reused in the
> >>>> order they would have been used.
> >>>> 
> >>>> Also, I would like to add vtapes and have them used
> >>>> after the currently highest numbered vtape.
> >>>> 
> >>>> I.e. I am succumbing to that human fraility of wanting
> >>>> things to run in numerical order when it really doesn't
> >>>> matter.
> >>> 
> >> 
> >> I haven’t tried this with vtapes, but with physical tapes,  I’ll just edit 
> >> the tapelist file.
> >> I move the desired tape into the correct slot and edit the date field to 
> >> be just after
> >> the previous line (so it fits, sequentially, with prior and subsequent 
> >> lines).
> >> 
> >> For blank tapes, which are at the end of the file,  I just move the line 
> >> to the end of
> >> the file, so it uses this tape next.
> >> “amadmin   tape”   will tell me the order it plans to use the next 
> >> few tapes,
> >> so I can tell if I got it right.
> >> 
> >> Note:  don’t change the line for a GOOD tape until just before you want it 
> >> to be used.
> >> I’m not sure what these changes would do to amrestore  or amrecover  
> >> processes.
> >> 
> > 
> > I'm surprised at amanda's reordering the tapelist file
> > at several points.  But the leading '0' files, whether
> > "reuse" or "no-reuse" keep being moved to the bottom.
> > 
> > I think I'm going to try setting the dates manually.
> > I'm concerned about the "tape header file",
> > 0-.  It has a DATE last used field
> > which is "X", I guess "never used".  I wonder if that
> > must match the tapefile date field?
> > 
> > Jon
> 
> Well,  for overwriting the tape,  it has never mattered for me.  My tapelist 
> edits (above)
> have worked.  Now,  for READING the tape and doing restores,  it may very 
> well matter.
> But for re-using the tape, it has to re-write that record anyway, with the 
> current date.  
> 
> Deb 

Makes sense.  I just finished changing the "0's" to datestamps,
reordering the list, and in the header file replacing the "X"
with the matching tapelist time stamp.  There were about 50,
so I wrote a script for some of the work.

I've left them at "no-reuse" but in the desired order.
If they stay that way for a few days I'll remove the "no-".

Thanks,

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: manual updating tapelist and tape headers after disk crash

[Jon LaBadie]

On Wed, Nov 08, 2017 at 06:10:12PM +, Debra S Baddorf wrote:
> 
> > Am 2017-11-06 um 19:49 schrieb Jon LaBadie:
> >> My vtapes are spread across several disks.  One of the
> >> smallest recently died in a most audible way.
> >> 
> >> I'm replacing the disk with a larger one.
> >> 
> >> I would like to recreate the lost vtapes (no backup
> >> data, that is gone!) and have them reused in the
> >> order they would have been used.
> >> 
> >> Also, I would like to add vtapes and have them used
> >> after the currently highest numbered vtape.
> >> 
> >> I.e. I am succumbing to that human fraility of wanting
> >> things to run in numerical order when it really doesn't
> >> matter.
> > 
> 
> I haven’t tried this with vtapes, but with physical tapes,  I’ll just edit 
> the tapelist file.
> I move the desired tape into the correct slot and edit the date field to be 
> just after
> the previous line (so it fits, sequentially, with prior and subsequent lines).
> 
> For blank tapes, which are at the end of the file,  I just move the line to 
> the end of
> the file, so it uses this tape next.
> “amadmin   tape”   will tell me the order it plans to use the next 
> few tapes,
> so I can tell if I got it right.
> 
> Note:  don’t change the line for a GOOD tape until just before you want it to 
> be used.
> I’m not sure what these changes would do to amrestore  or amrecover  
> processes.
> 

I'm surprised at amanda's reordering the tapelist file
at several points.  But the leading '0' files, whether
"reuse" or "no-reuse" keep being moved to the bottom.

I think I'm going to try setting the dates manually.
I'm concerned about the "tape header file",
0-.  It has a DATE last used field
which is "X", I guess "never used".  I wonder if that
must match the tapefile date field?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

amrmtape --dryrun defect

[Jon LaBadie]

Before deciding to use amrmtape I tried using its
"--dryrun" option.  According to the manpage:

--dryrun, -n
 Generate new tapelist(5) and database files
 with label removed, but leave them in /tmp
 and do not update the original copies.

While the originals were not touched, there were no
modified versions in /tmp, /tmp/amanda, or /var/tmp.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

manual updating tapelist and tape headers after disk crash

[Jon LaBadie]

My vtapes are spread across several disks.  One of the
smallest recently died in a most audible way.

I'm replacing the disk with a larger one.

I would like to recreate the lost vtapes (no backup
data, that is gone!) and have them reused in the
order they would have been used.

Also, I would like to add vtapes and have them used
after the currently highest numbered vtape.

I.e. I am succumbing to that human fraility of wanting
things to run in numerical order when it really doesn't
matter.

So, if I create new tapes and manually move their
tapelist entry to the position I want, will amanda
still use the "new" tape over the one at the bottom
of the list?

For those already existing vtapes lost in the crash,
can I create the directory and hand craft the tape
header.  Then just leave those tapelist entries alone.

Considering the unlikely possibility that I will attempt
recovery of something from the lost vtapes should I just
"amrmtape" the lost vtapes and recreate them from scratch?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: List Contents of gnutar-list files

[Jon LaBadie]

On Wed, Oct 25, 2017 at 01:33:53PM +1100, Jobst Schmalenbach wrote:
> Hi
> How can I see the contents of gnutar-list files?
> 
> I tried tar/gunzip, ckecked amadmin etc but it seems I cannot read the 
> content.
> 
> Need to know whats in those files to see whether my inclusion/exclusion list 
> is correct and why it takes too much time to backup one particular host.
> 
gnutar has had several formats for the gnutar list incremental files.
Starting with about version 1.16 the format included null bytes.

It is not pretty, but this awk command can make the file readable.

  awk 'BEGIN{RS="[\n\0]" ; ORS="\n"} {print}'  

The output is grouped by directories, here are two examples.

0
1483174597
191920908
64768
2374981
./usr/share/doc/amanda-3.3.3   <= the directory path
YCOPYRIGHT <=   ordinary files
YNEWS  <=   in the directory
YREADME


0
1471672919
831712838
64768
5505376
./usr/src/kernels/3.10.0-327.28.3.el7.x86_64/arch/mips/bcm63xx
YKconfig
YMakefile
Dboards<= a subdirectory


HTH
Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)