Re: LVM for vtapes

[Olivier]

Jon,

> Interesting discussion in other threads got me wondering
> whether I should have made some other choices when setting
> up my vtape environment.  Particularly whether I should
> have used LVM (Logical Volume Management) to create one
> large filesystem covering my multiple dedicated disks.
>
> Its a topic I do not recall being discussed, pros & cons.

I am using 7 disks of 3 (or is that 4) and 6 TB (should upgrade them all to 6TB
soon) almost dedicated to vtapes (the last disk also has a copy of the
deleted accounts). I have them configured as individual disks. The size
of my vtapes is also about 100GB and I am using a small chunck size, so
my disks end up being 80% full at least.

When I designed my vtape architecture, I decided to keep each disk
individual so that it can be put offline after usage. My idea was to
have a system that could prompt an operator to "mount a disk" before the
backup and the disk could be manually unmounted and safe stored each
day. It is taking advantage of the automount service on FreeBSD.

Mounting could be USD disk, or hot-swap. I never went very far in the
implementation. I wrote all that many years ago when vtapes were new and
limited to a single directory, that is why I wrote my own tape changer.

I knew about the risk of loosing a disk and it being a good portion of
consecutive backups. But what I had in mind was:

- have the system as simple and as portable as possible, so I can shove
  a disk in another machine and extract contents manually (during the
  great flood of Bangkok in 2011, I moved all the servers and also took
  all my hard disks from Amanda backup, but I did not need to move the
  rack mounted server itself);

- a side advantage of my own tape changer is that I can keep the older
  disks (each disk has an individual label, like any vtape has a label)
  (I have updated them from 500GB to 1TB to 3TB and soon to 6TB) and the
  vtapes are still known into tapelist (they are marked noreuse). If the
  need arise, I can still remount that old disk.

So far (10+ years) the only disk I had failling was the disk having the
holding partition, I guess it was because of excessive usage.

I understand that vtapes have evolved since I started using them, but my
system works for me so I never took the time to search any further.

Best regards,

Olivier

Re: backing up the catalogue for Amrestore

[Charles Curley]

On Mon, 20 Sep 2021 15:59:31 +
David Simpson  wrote:

> Is there a definitive guide on backing up the catalogue/indexes?
> 
> (ideally appending to tape as an epilogue of sorts)

Oddly enough, we've just been having a discussion related to this.
Start with Gene Heskett's email in your inbox:

Subject: Re: DLE splitting
Date: Mon, 20 Sep 2021 01:27:43 -0400

(I went looking for the list archives; that link appears to be
broken. However, http://www.amanda.org/support/mailinglists.php)

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

LVM for vtapes

[Jon LaBadie]

Interesting discussion in other threads got me wondering
whether I should have made some other choices when setting
up my vtape environment.  Particularly whether I should
have used LVM (Logical Volume Management) to create one
large filesystem covering my multiple dedicated disks.

Its a topic I do not recall being discussed, pros & cons.

Let me describe my current vtape environment as a starting
point.  I have 6 hard drives dedicated to vtapes, 3x3TB
and 3x4TB, 21TB total.  I have 240 vtapes of 100GB (0.1TB)
allocated.  That works out to 11-12 vtapes/TB.

When I first set things up, I did not like the idea of
one drive having all consecutive vtapes meaning it would
exclusively be used for about a month and not used again
for about 4 months.

Instead I allocated blocks of 8-12 vtapes on one drive
and the next block on the next drive ...  In part I felt
on drive failure, it would minimize my loss exposure of
recent backups to about a week rather than a month.

I wonder whether I might have instead just pooled the
six drives into one logical volume and not worry about
specific allocation ov vtapes to specific drives.

Hmmm, 21TB filesystem, hell of a beast to fsck ;)

Has anyone used LVM for their vtapes?  Positive or
negative results to report?

Jon

--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: DLE splitting

[Chris Hoogendyk]

With respect to having the current amanda directory backed up, what I do is simply run scripts from 
a crontab entry:


45 22 * * * /usr/local/adm/backup/daily-amanda; 
/usr/local/adm/backup/daily-amanda-config

The first script runs amanda and then runs a script I wrote called amchanger that accesses the tape 
library and unloads the tape that amanda just wrote to. The second script in the crontab entry does 
incremental and full backups of the amanda home directory on a weekly cycle to another server using 
cpio. The semicolon in the crontab entry says to wait until the first one is completed before 
calling the second one. I get an email report from amanda when the backups are done, and I get an 
email report from cron when the scripts are all completed, because they generate output to standard 
output.


Running on Ubuntu these days. Originally set up on Solaris.

I think all of this and details are back in the list archives somewhere. 
Possibly on the wiki as well.


On 9/20/21 1:27 AM, Gene Heskett wrote:

On Sunday 19 September 2021 23:07:54 Jon LaBadie wrote:


is there a way to run a user supplied program just
before amanda's taper opens each tape file?  I.e.,
before it actually opens and creates a new file on
the tape.

Do I recall correctly that it opens a tmp file on
the vtape and when complete renames it?  Or am I
thinking of the chunks on the holding disk?

jon

My GenesAmandaHelper does not do anything to the vtape, all it does is
run amanda as a child so it knows when amanda is done. at which point it
does its thing copyiing the now uptodate indices and configs to that
same vtape amanda just made. so they are valid for the vtape just
remade, as opposed to always being a day out of date.

No one else has publicly discussed how they've handled that so I have no
clue what others are doing, if anything to prevent a days indexes loss
if the drive or the amandatapes drive pukes and you have to recover to
bare metal. I also delete from the index database I keep, those files
that represent a vtape that has been re-used. So I have a duplicate of
amandas index stuff that takes up a few gigabytes on the main drive in
addition to a backup copy on the vtape. Belt and suspenders approach.
Recovering the latest indice and putting it where amanda keeps them,
enables access to the whole /amandatapes drive in one swell foop.

Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett


--
---

Chris Hoogendyk

-
   O__   Systems Administrator, Retired
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center III
~~ - University of Massachusetts, Amherst



---

Erdös 4

RE: DLE splitting

[Cuttler, Brian R (HEALTH)]

Debra, Charles, Jon,

I had some 'large' vtapes for a while, and had a requirement for N-days of 
recovery and ran into an issue were I was getting deadlocked on space during 
backup.
If the vtapes were too small I consumed more of them, if they were too large 
then I'd might not have enough free space when one of them was reused, not 
certain why but it may have been that level 0 was advanced more frequently 
because of the "available" tape capacity, which is measured by tape size and 
does not take vtape pool volume into account.

100% right, vtapes leave unused space in the pool, but some vtape size tuning 
may be needed.

Thanks,
Brian


-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Debra S Baddorf
Sent: Friday, September 17, 2021 5:28 PM
To: Charles Curley 
Cc: Debra S Baddorf ; amanda-users 
Subject: Re: DLE splitting

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.


> On Sep 17, 2021, at 12:57 PM, Charles Curley 
>  wrote:
>
> On Fri, 17 Sep 2021 11:55:09 -0400
> Jon LaBadie  wrote:
>
>> I'm seeing instances of wasted VTape with amanda switching
>> to a new tape for the last DLE even though there is room
>> left on the current tape.
>
> I wouldn't worry about it. Unlike physical tapes, vtapes all pull
> storage out of the same pool. You might create some more vtapes so you
> have plenty for your storage.
>

And, as I understand it,  vtapes don’t actually require more space than
is actually ON them.   I.E.  Empty space is still owned by the vtape manager.
Which is what Charley said,  but in different words.

Deb Baddorf
Fermilab





> --
> Does anybody read signatures any more?
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__charlescurley.com=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=HMrKaRiCv4jddln9fLPIOw=PwvWx6Y9I4KUy_b5D3jsXMnkbLiq7ulFG5J1JzobJeY=lljOS8dvUO1Mot2mr72DNZxuu8f_j1tqh0kSizkJc18=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__charlescurley.com_blog_=DwICAg=gRgGjJ3BkIsb5y6s49QqsA=HMrKaRiCv4jddln9fLPIOw=PwvWx6Y9I4KUy_b5D3jsXMnkbLiq7ulFG5J1JzobJeY=3vxiQwgUx4Am7JAOXEQ7z-J2yfWqyCtBxy2fvA7FyJs=

Re: backing up the catalogue for Amrestore

[Gene Heskett]

On Monday 20 September 2021 11:59:31 David Simpson wrote:

> Is there a definitive guide on backing up the catalogue/indexes?
>
> (ideally appending to tape as an epilogue of sorts)
>
That is basicly what GenesAmandaHelper does.

If comfy with shell scripts there is an older version that will need to 
be tweaked a bit for 3.5.1 on my web page in the sig. Its a bit out of 
date, written originally for a much older amanda that I started out with 
in 1999, but shouldn't be too hard to adapt. Zero consideration for the 
fixed capacity of real tapes, but I've been using vtapes on a big hd for 
well over a decade with relatively few problems that didn't resolve to 
hdwe in the final fix. They don't have a fixed tape length, only the 
disks capacity. 

This is NOT suitable for businesses requiring off site storage, but as a 
backup for a 6 machine home network, 4 of which are driving metal 
carving machines. Be my guest and I'll help if I can.

> -
> David Simpson - Senior Systems Engineer
> ARCCA, Redwood Building,
> King Edward VII Avenue,
> Cardiff, CF10 3NB
>
> David Simpson - peiriannydd uwch systemau
> ARCCA, Adeilad Redwood,
> King Edward VII Avenue,
> Caerdydd, CF10 3NB
>
> simpso...@cardiff.ac.uk
> +44 29208 74657



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

backing up the catalogue for Amrestore

[David Simpson]

Is there a definitive guide on backing up the catalogue/indexes?

(ideally appending to tape as an epilogue of sorts)

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

RE: Encryption information

[David Simpson]

Yes this sort of thing. Server-side.

Aside from safekeeping of secrets used are there any other pitfalls?

What is the experience of this generally?

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

-Original Message-
From: Jens Berg  
Sent: 20 September 2021 10:01
To: David Simpson ; amanda-users@amanda.org
Subject: Re: Encryption information

Maybe 
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.zmanda.com%2Findex.php%2FHow_To%3ASet_up_data_encryptiondata=04%7C01%7CSimpsonD4%40cardiff.ac.uk%7C1d0bb264ba7b4f168ad008d97c151be6%7Cbdb74b3095684856bdbf06759778fcbc%7C1%7C0%7C637677252382787331%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=uTXU43CkpH00JyAOdeL3mhow%2FHowEMRPmfNOxAdMig0%3Dreserved=0
 is what you are looking for?

On 20.09.2021 10:39, David Simpson wrote:
> Looking for any useful information and scripts on encryption with 
> Amanda (and not hardware encryption done by the tape library itself).
> 
> thanks
> 
> -
> 
> David Simpson - Senior Systems Engineer
> 
> ARCCA, Redwood Building,
> 
> King Edward VII Avenue,
> 
> Cardiff, CF10 3NB
> 
> David Simpson - peiriannydd uwch systemau
> 
> ARCCA, Adeilad Redwood,
> 
> King Edward VII Avenue,
> 
> Caerdydd, CF10 3NB
> 
> simpso...@cardiff.ac.uk 
> 
> +44 29208 74657
> 

--
Jens Berg  R  VTech IAD GmbH
Max-Giese-Str. 22, 24116 Kiel, Germany
Phone: +49 431 99081811, Fax: +49 431 99081850 PGP key Id 0x45B55BB1
Location: Kiel, Local Court Kiel, HRB 10298 KI Managing Director: Andreas Zipp
---
Please be informed that this email and any materials attached herewith may 
contain confidential or legally privileged information that are intended solely 
for the use by its named recipient. If you have received this email in error, 
please notify us immediately by reply email and delete this email from your 
system. Any disclosure, use, copying, printing, distribution or reliance upon 
the contents of this email is strictly prohibited. Thank you for your 
attention. - This mail is sent via VTech IAD GmbH

Re: amanda with Debian 11

[Stefan G. Weichinger]

Am 16.09.21 um 16:58 schrieb David Simpson:

It is now released under the name "amanda_deploy" on github

Looking forward to seeing usage/comments/improvements etc.


I didn't yet *use* your roles so far, but I looked through them ... 
looks quite good already and covers not only the basics but additional 
stuff like ssh etc


Nice work!

Re: DLE splitting

[Gene Heskett]

On Monday 20 September 2021 08:38:36 Charles Curley wrote:

> On Mon, 20 Sep 2021 01:27:43 -0400
>
> Gene Heskett  wrote:
> > No one else has publicly discussed how they've handled that so I
> > have no clue what others are doing, if anything to prevent a days
> > indexes loss if the drive or the amandatapes drive pukes and you
> > have to recover to bare metal.
>
> An interesting subject.
>
> I have three rotating USB drives for offsite backup. I swap them out
> to a nearby site weekly and take another with me when I travel. So if
> I've done everything else right, I should be good against anything
> short of the Yellowstone Caldera going or an asteroid strike if I'm
> near home when it occurs.
>
Which is far more than my wrapper does. But if yellowstone goes up (Its 
way overdue) or an asteroid strike, like in the Yucatan 65 million years 
go, its all over anyway. I don't think its possible to sterilize the 
planet, but we've no clue what future lifeform will dominate 10 million 
years after that event. I doubt that it will be human, we seem bound and 
determined to exterminate us in any event with fauci leading the march. 
We have also discovered we can vote us a subsistence life, forgetting 
TANSTAAFL. We do that at our species risk.

> These I rsync manually from a RAID array on my backup server. That
> array includes my VTAPEs. I also have a script that calls amanda, then
> backs up amanda's metadata into the RAID array. So everything gets
> backed up to the offsite drives.



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: DLE splitting

[Charles Curley]

On Mon, 20 Sep 2021 01:27:43 -0400
Gene Heskett  wrote:

> No one else has publicly discussed how they've handled that so I have
> no clue what others are doing, if anything to prevent a days indexes
> loss if the drive or the amandatapes drive pukes and you have to
> recover to bare metal.

An interesting subject.

I have three rotating USB drives for offsite backup. I swap them out to
a nearby site weekly and take another with me when I travel. So if I've
done everything else right, I should be good against anything short of
the Yellowstone Caldera going or an asteroid strike if I'm near home
when it occurs.

These I rsync manually from a RAID array on my backup server. That array
includes my VTAPEs. I also have a script that calls amanda, then backs
up amanda's metadata into the RAID array. So everything gets backed up
to the offsite drives.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Encryption information

[Jens Berg]

Maybe https://wiki.zmanda.com/index.php/How_To:Set_up_data_encryption is 
what you are looking for?


On 20.09.2021 10:39, David Simpson wrote:
Looking for any useful information and scripts on encryption with Amanda 
(and not hardware encryption done by the tape library itself).


thanks

-

David Simpson - Senior Systems Engineer

ARCCA, Redwood Building,

King Edward VII Avenue,

Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau

ARCCA, Adeilad Redwood,

King Edward VII Avenue,

Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk 

+44 29208 74657



--
Jens Berg  R  VTech IAD GmbH
Max-Giese-Str. 22, 24116 Kiel, Germany
Phone: +49 431 99081811, Fax: +49 431 99081850
PGP key Id 0x45B55BB1
Location: Kiel, Local Court Kiel, HRB 10298 KI
Managing Director: Andreas Zipp
---
Please be informed that this email and any materials attached herewith
may contain confidential or legally privileged information that are
intended solely for the use by its named recipient. If you have
received this email in error, please notify us immediately by reply
email and delete this email from your system. Any disclosure, use,
copying, printing, distribution or reliance upon the contents of this
email is strictly prohibited. Thank you for your attention. - This mail
is sent via VTech IAD GmbH

Encryption information

[David Simpson]

Looking for any useful information and scripts on encryption with Amanda (and 
not hardware encryption done by the tape library itself).

thanks

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657