Re: bad tape?

[Olivier]

Gene Heskett  writes:

> On Monday 26 August 2019 23:55:31 Olivier wrote:
>
>> Gene Heskett  writes:
>> > Generally speaking, only because the disc is random access.
>>
>> But a disk dedicated to vtapes should be doing a lot of sequetial
>> accesses: once it has been formatted and the slots have been assigned,
>> it is writting files the size of one Amanda's chunk. In fact, that
>> would be worth a study: the disk usage for vtapes vs. normal disk
>> usage.
>>
>> That is just gross figures but:
>>
>> Users' home directories:
>> FilesystemSizeUsed   Avail Capacity  Mounted on
>> /dev/da1p12.9T851G1.8T31%/home
>> 2565312 files, 223129681 used, 556890331 free
>> (564355 frags, 69540747 blocks, 0.1% fragmentation)
>
> My Mail dir is on /home, dates back to about my 2nd install in 2001, with 
> probably north of 20 GB of maildirs, so I'd expect that is more than 1% 
> fragmented, but except for tde's kmail occasionally bucking about it, 
> has not been a major problem. Copying it to a new Maildir usually fixes 
> it for that particular list.  Reducing the keep time for those lists 
> deemed not so important has also helped. About 3 lists I keep forever, 
> but 50 more are expired every 3 or 6 months.

That was given as an example only, to show that fragmentation grows
faster on a file system supporting users' home directory tha on a file
system supporting Amanda'a vtapes. I had rebuild the users' home
directory on a new RAID array recently, that is why it is so little
fragmented, but still much more than an Amanda disk that is much older.

>> Amanda vtape disk:
>> FilesystemSizeUsed   Avail Capacity  Mounted on
>> /dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
>> 475 files, 582393950 used, 127171372 free
>> (84 frags, 15896411 blocks, 0.0% fragmentation)
>>
>> The vtape disk is slightly older than the users' home, definitely
>> fuller and less fragmented, so I would guess big sequetial files with
>> little head movement.
>>
> I wondered about fragmentation myself, but it has never reared its head 
> in well over 15 years.

Best regards,

Olivier


-- 

Re: bad tape?

[Olivier]

Diego Zuccato  writes:

> Il 27/08/19 05:55, Olivier ha scritto:
>
>> But a disk dedicated to vtapes should be doing a lot of sequetial
>> accesses: once it has been formatted and the slots have been assigned,
>> it is writting files the size of one Amanda's chunk. In fact, that would
>> be worth a study: the disk usage for vtapes vs. normal disk usage.
> That's the *perfect* use-case for SMR drives. But they'd need either no
> filesystem (like tapes :) ) or a dedicated one.
> Is it possible to use raw devices as vtapes?

vtape do rely on a Unix file system, it needs a directory and a couple
of sub-directory per vtape.

Olivier
-- 

Re: bad tape?

[Olivier]

Jon LaBadie  writes:

> On Tue, Aug 27, 2019 at 11:44:02AM +0200, Diego Zuccato wrote:
>> Il 27/08/19 05:55, Olivier ha scritto:
>> 
>> > But a disk dedicated to vtapes should be doing a lot of sequetial
>> > accesses: once it has been formatted and the slots have been assigned,
>> > it is writting files the size of one Amanda's chunk. In fact, that would
>> > be worth a study: the disk usage for vtapes vs. normal disk usage.
>> That's the *perfect* use-case for SMR drives. But they'd need either no
>> filesystem (like tapes :) ) or a dedicated one.
>> Is it possible to use raw devices as vtapes?
>
> My guess is that vtapes are expecting normal *nix file
> names and system/library calls.

They do, indeed.

> I'm pretty sure it would be possible to write a FUSE (file system
> in user space extension) to implement what you imagine.

Or write another device for Amanda to use, it would not be vtape, it
would be ... something.

Olivier
-- 

Re: amdump (3.5.1) run from script does not send email report

[Chris Hoogendyk]

Thanks, Nathan, that's a good guide for troubleshooting this issue. However, it leaves out one 
crucial item: sysadmin blind spots. ;-)  Looking through my emails and the dates, I realized Amanda 
had hung for several days and I hadn't noticed it because of other problems I've been focused on . . .


(We did release upgrades from Ubuntu 14.04 to Ubuntu 16.04 on two departmental servers that were 
running drupal web sites with MySQL backends as well as sendmail, squirrelmail, request-tracker4, 
proftpd, samba, and all the other things an academic department depends on. I've spent the last week 
patching up services and dealing with the switch from upstart to systemd among other things, trying 
to get things running smoothly before the semester begins.)


Anyway, I had added a line in my script that did `echo "Amanda already running"` and it got lost in 
all the output from the rest of the script documenting the nightly backup of the Amanda 
configuration and indexes and the trimming of the SSDs. Looking back at cron emails, I see the 
message in the output, but I had overlooked it in my quick scanning. Otherwise, I would have looked 
into the status of Amanda more closely and sooner.



On 8/27/19 3:59 PM, Nathan Stratton Treadway wrote:

On Tue, Aug 27, 2019 at 15:15:57 -0400, Chris Hoogendyk wrote:

When I upgraded one of my servers to Amanda 3.5.1, I had to run
amdump from a script (out of cron), because . . .

Now it doesn't send out an email report when it is finished.

Did you switch to using the script at the same time you upgraded to
3.5.1?  That is, did you ever have working reports on this server, either
pre-3.5.1 called from your script, or 3.5.1 called directly from cron?

(Obviously calling it from a script should work; I have a 3.5.1 system
running on Ubuntu which sends email reports from a script-invoked amdump
just fine.)

  

What is my workaround? I've looked through the amdump and amanda man

(What OS is running on this system, and what was the origin of your
Amanda binaries?)

What do you get when you run the following commands?

   $ amgetconf  mailto
   $ amgetconf  send-amreport-on
   $ amgetconf  mailer

Does the executable shown in the "mailer" parameter actually exist?



pages. I don't see an option to tell it, "please DO send an email
report." I also did check all the various places that might hint at
an email getting hung up, but didn't find any evidence in any of the
logs or spool directories that any had ever been sent.

Do you have any "amreport.*.debug" files in your
/server// directory?

If so, look in those log files for a line containing the string "invoking mail
app:", and see if anything interestinng shows up between that line and
onne saying "waiting for child process to finish.." (which normally is
the next line in the log, I think)

If those amreport debug files don't exist, look in the amdump.*.debug
files for lines with the string "amreport" to see if it's trying
and failing to run amreport, or completely skipping that step.

Nathan

Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239


--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: bad tape?

[Jon LaBadie]

On Tue, Aug 27, 2019 at 11:44:02AM +0200, Diego Zuccato wrote:
> Il 27/08/19 05:55, Olivier ha scritto:
> 
> > But a disk dedicated to vtapes should be doing a lot of sequetial
> > accesses: once it has been formatted and the slots have been assigned,
> > it is writting files the size of one Amanda's chunk. In fact, that would
> > be worth a study: the disk usage for vtapes vs. normal disk usage.
> That's the *perfect* use-case for SMR drives. But they'd need either no
> filesystem (like tapes :) ) or a dedicated one.
> Is it possible to use raw devices as vtapes?

My guess is that vtapes are expecting normal *nix file
names and system/library calls.

I'm pretty sure it would be possible to write a FUSE (file system
in user space extension) to implement what you imagine.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amdump (3.5.1) run from script does not send email report

[Nathan Stratton Treadway]

On Tue, Aug 27, 2019 at 15:15:57 -0400, Chris Hoogendyk wrote:
> When I upgraded one of my servers to Amanda 3.5.1, I had to run
> amdump from a script (out of cron), because . . .
> 
> Now it doesn't send out an email report when it is finished.

Did you switch to using the script at the same time you upgraded to
3.5.1?  That is, did you ever have working reports on this server, either
pre-3.5.1 called from your script, or 3.5.1 called directly from cron?

(Obviously calling it from a script should work; I have a 3.5.1 system
running on Ubuntu which sends email reports from a script-invoked amdump
just fine.)

 
> What is my workaround? I've looked through the amdump and amanda man

(What OS is running on this system, and what was the origin of your
Amanda binaries?)

What do you get when you run the following commands?

  $ amgetconf  mailto 
  $ amgetconf  send-amreport-on
  $ amgetconf  mailer

Does the executable shown in the "mailer" parameter actually exist?


> pages. I don't see an option to tell it, "please DO send an email
> report." I also did check all the various places that might hint at
> an email getting hung up, but didn't find any evidence in any of the
> logs or spool directories that any had ever been sent.

Do you have any "amreport.*.debug" files in your
/server// directory? 

If so, look in those log files for a line containing the string "invoking mail
app:", and see if anything interestinng shows up between that line and
onne saying "waiting for child process to finish.." (which normally is
the next line in the log, I think)

If those amreport debug files don't exist, look in the amdump.*.debug
files for lines with the string "amreport" to see if it's trying
and failing to run amreport, or completely skipping that step.

Nathan

Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

amdump (3.5.1) run from script does not send email report

[Chris Hoogendyk]

When I upgraded one of my servers to Amanda 3.5.1, I had to run amdump from a script (out of cron), 
because . . .


Now it doesn't send out an email report when it is finished.

What is my workaround? I've looked through the amdump and amanda man pages. I don't see an option to 
tell it, "please DO send an email report." I also did check all the various places that might hint 
at an email getting hung up, but didn't find any evidence in any of the logs or spool directories 
that any had ever been sent.


The reason I'm running it out of a script instead just a straight call from the crontab line is that 
3.5.1 will now run more than one instance of amanda in parallel, and this server can't handle that. 
So, I have a script now that checks to see if amanda is already running before launching a new run. 
That script is called from cron. I have another server that manages just fine with Amanda 3.5.1's 
behavior, because it has a tape library with two LTO7 drives and also with a pair of 4TB server 
grade SSDs for holding disk space.



--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: bad tape?

[Diego Zuccato]

Il 27/08/19 05:55, Olivier ha scritto:

> But a disk dedicated to vtapes should be doing a lot of sequetial
> accesses: once it has been formatted and the slots have been assigned,
> it is writting files the size of one Amanda's chunk. In fact, that would
> be worth a study: the disk usage for vtapes vs. normal disk usage.
That's the *perfect* use-case for SMR drives. But they'd need either no
filesystem (like tapes :) ) or a dedicated one.
Is it possible to use raw devices as vtapes?

-- 
Diego Zuccato
Servizi Informatici
Dip. di Fisica e Astronomia (DIFA) - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
mail: diego.zucc...@unibo.it

Re: bad tape?

[Gene Heskett]

On Monday 26 August 2019 23:55:31 Olivier wrote:

> Gene Heskett  writes:
> > Generally speaking, only because the disc is random access.
>
> But a disk dedicated to vtapes should be doing a lot of sequetial
> accesses: once it has been formatted and the slots have been assigned,
> it is writting files the size of one Amanda's chunk. In fact, that
> would be worth a study: the disk usage for vtapes vs. normal disk
> usage.
>
> That is just gross figures but:
>
> Users' home directories:
> FilesystemSizeUsed   Avail Capacity  Mounted on
> /dev/da1p12.9T851G1.8T31%/home
> 2565312 files, 223129681 used, 556890331 free
> (564355 frags, 69540747 blocks, 0.1% fragmentation)

My Mail dir is on /home, dates back to about my 2nd install in 2001, with 
probably north of 20 GB of maildirs, so I'd expect that is more than 1% 
fragmented, but except for tde's kmail occasionally bucking about it, 
has not been a major problem. Copying it to a new Maildir usually fixes 
it for that particular list.  Reducing the keep time for those lists 
deemed not so important has also helped. About 3 lists I keep forever, 
but 50 more are expired every 3 or 6 months.

> Amanda vtape disk:
> FilesystemSizeUsed   Avail Capacity  Mounted on
> /dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
> 475 files, 582393950 used, 127171372 free
> (84 frags, 15896411 blocks, 0.0% fragmentation)
>
> The vtape disk is slightly older than the users' home, definitely
> fuller and less fragmented, so I would guess big sequetial files with
> little head movement.
>
I wondered about fragmentation myself, but it has never reared its head 
in well over 15 years.

> Good luck with your health.
>
Thank you.

> Olivier



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page