Re: command line syntax host !disk
On Wed, Oct 13, 2010 at 12:25 AM, Jon LaBadie j...@jgcomp.com wrote: I wonder if a syntax extension that would be backward compatible is possible. Blue-skying here, maybe a single argument with a separator between host and disk. Something like 'host|disk' or 'host/disk'. For those playing along at home, by the way, this is all summed up nicely in the amanda-match(7) manpage. http://wiki.zmanda.com/man/amanda-match.7.html Back when I was implementing 'amadmin holding ..', I considered defining a dumpspec of the form host:d...@level/date, but it was rejected because the longer dumpspec syntax is already unambiguous. It's only the dlespecs that are ambiguous, and those are only used for amdump and amflush. Any merit (or problems) to the idea? Or maybe the need does not occur often enough to matter. Honestly, the problem doesn't occur often enough to try to *complicate* things by supporting both an ambiguous and unambiguous syntax -- Amanda's match expressions are pretty complex already. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: command line syntax host !disk
On Wed, Oct 13, 2010 at 10:08 AM, Jon LaBadie j...@jgcomp.com wrote: Surprisingly, the amanda-match(7) manpage is not included in the pre-built binaries I've installed (version 3.1.2-1). I added it in 3.2 However the installed amanda(8) manpage does include the same Host and Disk Expression section as amanda-match(7). It's been improved in amanda-match(7), and fleshed out with more context, but yes - that was the source of most of that material. Surprisingly, the amanda(8) manpage is not available on the zmanda wiki under Most Current Manpages. sure it is - the link takes you directly to amanda(8), which serves as a table of contents for the other manpages http://wiki.zmanda.com/man/amanda.8.html Dustin
Re: amvault - version 3.2.0beta3
On Fri, Oct 8, 2010 at 10:11 AM, Dustin J. Mitchell dus...@zmanda.com wrote: Does such a configuration exist? I thought your config was named HANSA? Did you sort this out? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: command line syntax host !disk
On Tue, Oct 12, 2010 at 10:53 PM, Jon LaBadie j...@jgcomp.com wrote: The manpage Amanda(8) where Host and Disk Expression are documented indicates some wildcarding is allowed. Thus, for a host where I wanted all DLEs excpet one that began with the letter A I tried: amdump config host '[!A]*' This was a no-starter, giving an argument of: Argument [!A]* cannot be both a host and a disk What have I overlooked in the docs? The problem is that [!A]* matches both a host and a disk. This [host [disk .. ] .. ] syntax is ambiguous. You could have either meant all disks not beginning with A on host HOST or all DLEs on host HOST and all hosts not beginning with A IMHO, an ambiguous syntax like this is stupid, because it may work fine until one day you change the set of available items. For example, if you had no hosts beginning with A, then the expression would work. But as soon as you added such a host, the syntax would stop working. That said, it's the syntax we've got, and backward compatibility is job #1. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: No space left on device question
On Mon, Oct 11, 2010 at 10:47 AM, McGraw, Robert P rmcg...@purdue.edu wrote:
hertz -ss/export/users-w 0 50143 50143 -- PARTIAL
25:45 33234.0
says that when it went to the next tape that the complete {user-a, users-m,
users-w} were written on the next tape.
Actually, all that says is that the dump to the first tape (that
filled up) was partial. If you see that DLE again in the list, then
you know it was successfully re-flushed later. The 3.2.0beta3
amreport also puts a note at the top of the report to indicate this.
Dustin
--
Open Source Storage Engineer
http://www.zmanda.com
Re: No space left on device question
looks like they did... On Mon, Oct 11, 2010 at 1:03 PM, McGraw, Robert P rmcg...@purdue.edu wrote: 2010-10-09 23:10:45 hertz /gauss/export/users-w 0 A00184 1 1/1 OK 2010-10-09 23:10:45 hertz /gauss/export/users-w 0 A00180 6 1/-1 PARTIAL PARTIAL Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault report - *** THE DUMPS DID NOT FINISH PROPERLY!
On Sat, Oct 9, 2010 at 12:22 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: ok then I must overide the tapetype def with -o option in the amvault command. I'm not sure about that. Your amvault run demonstrated that your tapes can actually hold almost 6 times the data you've suggested in your default tapetype. That is, amvault stuffed 550% of the configured tapetype 'length' parameter onto a single tape. So I think the length should be changed for *all* runs, not just amvault. But there are cases where you want to use a different length for vaulting runs, and you're right - in those cases you will want to define a new tapetype and use amvault ... -otapetype=MYNEWTAPETYPE As for the waiting-to-flush: from the amdump.1 you sent, I see that the dumper gets a schedule from the planner after 2364 seconds, and first starts writing to tape at 2365 seconds (svksdev3:/extra1, a 1kb DLE). It takes a while for the taper to find its new tape (EMS1-13), which it does at 2372 seconds, and it finishes writing the part about 0.055 seconds later. Sure enough, after that point, the taper remains idle - the driver does not assign it any additional work. This looks like it's related to the multi-taper implementation, so perhaps Jean-Louis can jump in here? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault report - *** THE DUMPS DID NOT FINISH PROPERLY!
On Sat, Oct 9, 2010 at 4:02 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Sat Oct 9 22:23:08 2010: amgtar: Spawning /usr/local/bin/tar /usr/local/bin/tar --create --verbose --file - --directory / --one-file-system --no-check-device --listed-incremental /var/opt/lib/amanda/gnutar-lists/hansabck_1.new --sparse --ignore-failed-read --totals . in pipeline but in 3.1.1 snapshot is created and used - the config is the same. Sat Oct 9 20:10:26 2010: amgtar: Spawning /usr/local/bin/tar /usr/local/bin/tar --create --verbose --file - --directory //.zfs/snapshot/amanda-_-current --one-file-system --no-check-device --listed-incremental/var/opt/lib/amanda/gnutar-lists/tx5019__0.new --sparse --ignore-failed-read --totals . in pipeline Ah, so this is using the amzfs-snapshot scripts? Jean-Louis, did something change here? I don't recall if we have a way for a script to change the diskdevice for a DLE? Did that work in 3.1 and not in 3.2? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Testing amvault
On Fri, Oct 8, 2010 at 4:19 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Good work releasing beta3. Few thing that come to my mind while testing amvault regarding slections of dumps. Maybe I'm misunderstanding come to mind, but these are all possible now. Have I failed to document these well enough? Or are they not working for you? - latest dump with --src-timestamp latest Works fine. - all dumps without specifying --src-timestamp Works fine. - parts by specifying dates eg. --src-timestamp 20100926-20101008 You can specify a range of *dump* timestamps (the dates the backups were taken from the client) with a range in the datestamp parameter, e.g., amvault ... '*' '*' '0-9' '20100926-20101008' There's no way to specify a range of write timestamps (the dates the backups were written to secondary media). I would likte to select all but only the most recent version. What does most recent version mean in this case? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Re:
On Fri, Oct 8, 2010 at 8:40 AM, McGraw, Robert P rmcg...@purdue.edu wrote: Dustin, thanks very much for your help in resolving this. Sure thing. Jean-Louis has, in the interim, fixed the problem of not being able to eject when a fatal error occurs in the tape device. So this one is fixed twice :) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault - version 3.2.0beta3
On Fri, Oct 8, 2010 at 9:01 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: I'm vaulting from config hansa and I'm getting parse error about hansa-vault config ? Does such a configuration exist? I thought your config was named HANSA? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Testing amvault
Thanks for the tip about --autolabel any, by the way. I accidentally implemented it as --autolabel all. The fix is easy :) On Fri, Oct 8, 2010 at 11:17 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: ok this will vault all full dumps in my configuation but can I select only the most resent for each filesystem ? amvault --dry-run --fulls-only --dst-changer vault \ --label-template EMS1-7% --autolabel volume-error hansa Ah, no, there's not an easy way to do that currently. This is a reasonable request, and wouldn't be difficult to implement in 3.3, but I'm concerned that we not build every possible use-case into amvault as an option. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.1.2
On Fri, Oct 8, 2010 at 10:24 AM, Brian Cuttler br...@wadsworth.org wrote: Feature request (?) - There are checks for DLE tape, seems almost silly but a check for SUM(DLEs) (tape_capacity * runtapes) One DLE with a full size larger than available tape is always a problem: there's no way to set up a dump run that will get that DLE on tape. So we warn about it. However, it's not a bug for the sum of the full size of each DLE to be greater than tape length, because the planner can conceivably rearrange fulls and partials to fit everything on tape anyway. When the planner *fails* to pull this off, it will warn. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault report - *** THE DUMPS DID NOT FINISH PROPERLY!
On Fri, Oct 8, 2010 at 1:11 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Is this because amvault is considered a flush ? Tape usage statistic is rather odd as well. Can you send the trace log that generated this report? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.1.2
On Fri, Oct 8, 2010 at 12:26 PM, Brian Cuttler br...@wadsworth.org wrote: I realize that there IS NO Solution within amanda. This is a capacity planning issue for the humans to solve, but amanda can warn via the amdump report, that problems are imminent and that human capacity planning is required. Well, by the time this happens, the planner has already run and generated the best schedule it can think of, which may be larger than the available tape. So it's too late for a warning that would allow you to fix something before the run. And if the dumps actually *don't* fit on tape, you'll get errors to that end. Adding a warning here would only be useful in the narrow margin when the planner has scheduled dumps whose estimates are more than tapelength*runtapes, but where those dumps actually *do* fit on tape at the end of the day. Depending on your actual tape size, that margin may be only a few megabytes wide - easy enough to miss. I wonder if the taper could add warnings to the trace log (such that they would show up in amreport) when it starts bumping DLEs to keep its total under tapelength*runtapes. Jean-Louis, what do you think? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: long running amandad on client
On Fri, Oct 8, 2010 at 1:07 PM, John Hein jh...@symmetricom.com wrote: I have a client with an amandad that has been running since Sep 23... .. 2000 APPLICATION 36 |;auth=BSD;compress-fast;index;exclude-file=.no-amanda-backup;exclude-file=.nobak;exclude-file=.noback;exclude-file=.nodump; I would avoid BSD auth if possible. It uses UDP, so there is a bit of a race where amandad must wait around long enough to catch any new packets, but must exit at some point. Currently, this timeout is set to 30 seconds, but there may be (or have been) a particular case in which amandad can get stuck waiting for a new packet. The file-descriptor leaking is still a problem - amandad is in awful shape, to be honest - but zombie collection has improved somewhat (with band-aids, not a fix, but..) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault report - *** THE DUMPS DID NOT FINISH PROPERLY!
On Fri, Oct 8, 2010 at 1:11 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Is this because amvault is considered a flush ? .. *** THE DUMPS DID NOT FINISH PROPERLY! No, it was a bug - amvault didn't write the FINISH driver line before calling amreport, so amreport didn't think it was finished. Tape usage statistic is rather odd as well. .. EMS1-71 1:59 261951M 532.9 462 462 It seems to have really, truly written all that data to that tape. What sort of tape is it? A vtape? If so, does it contain 262MB of data? That's odd, because vtapes should cut you off when they reach their configured length (this will become configurable in 3.3). Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amvault report - *** THE DUMPS DID NOT FINISH PROPERLY!
On Fri, Oct 8, 2010 at 5:42 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: No is a HP Ultrium LTO 3 tape drive: Yes I think the data was written to tape I will verify that next week. OK. You should adjust the length in your tapetype definition, then - your tape appears to be at least 550% longer than you have configured. On Fri, Oct 8, 2010 at 5:52 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: No I was wrong about it only delayed the flushing until at the end. Is that a new behaviour in versioh 3.2.0 ? Not sure what you mean by this - what was delayed? The only problem I got was on the client side for the root filesystem zfs. /-- hansabck / lev 1 FAILED [/usr/local/bin/tar exited with status 2: see /tmp/amanda/client/hansa/amgtar.20101008223221000.debug] sendbackup: info BACKUP=APPLICATION sendbackup: info APPLICATION=amgtar sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/opt/sfw/libexec/amanda/application/amgtar restore [./file-to-restore]+ sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? /usr/local/bin/tar: ./devices/ramdisk-root: Cannot savedir: No such file or directory ? /usr/local/bin/tar: ./devices/ramdisk-root/: Warning: Cannot stat: No such file or directory You probably should be excluding ./devices? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Thu, Oct 7, 2010 at 1:49 AM, Titl Erich erich.t...@ruf.ch wrote:
I guess this finding should give you something to chew on
Indeed! If I'm reading the hex dumps correctly, then, the first file
is padded at the beginning with exactly 28424 bytes of zero. The file
lengths differ by the same quantity. Can you verify that the files
are identical after that offset is removed?
I wondered if the zeroes could come from padding to a multiple of the
block size (presumably 32,768). Sort of!
14392979208 % 32768 # unreadable
28424
14392950784 % 32768 # readable
0
So the *readable* file is an even multiple of the block size (and may
include some trailing all-zero tar records for that purpose), while
the *unreadable* file is not. So this offers no clue as to where that
offset comes from.
Let's see if those extra bytes are coming from the server, or if
amrecover is adding them, somehow. Can you add the following patch to
Amanda/Recovery/Clerk.pm (you can just add the line in place, without
the +)
diff --git a/perl/Amanda/Recovery/Clerk.pm b/perl/Amanda/Recovery/Clerk.pm
index 40c9ff1..f344893 100644
--- a/perl/Amanda/Recovery/Clerk.pm
+++ b/perl/Amanda/Recovery/Clerk.pm
@@ -330,6 +330,8 @@ sub _xmsg_part_done {
my ($src, $msg, $xfer) = @_;
my $xfer_state = $self-{'xfer_state'};
+Amanda::Debug::debug(@@@ read a $msg-{'size'}-byte part);
+
my $next_label = $xfer_state-{'next_part'}-{'label'};
my $next_filenum = $xfer_state-{'next_part'}-{'filenum'};
also, please set
auth_debug 9
on the server *and* the client. Then do another run with amrecover.
The amrecover, amidxtaped, and (server-side) amandad debug logs will
then have sufficient information to determine how many bytes passed
through each component.
If I recall, there's no filtering (crypto or compression) going on, right?
Dustin
--
Open Source Storage Engineer
http://www.zmanda.com
Re: Re:
On Thu, Oct 7, 2010 at 11:01 AM, McGraw, Robert P rmcg...@purdue.edu wrote: I had been using 2048KB, which I had defined in tapetype, when I was running Amanda 2.x and never got this error. Yes, it's a new error for something that used to be handled quietly. Basically, the kernel takes the 2MB write and breaks it up into two 1MB writes, only one of which succeeds. So since is thinks that I need to give it 1M I will give it 1M block size. Good plan. Q1 Is defining the block size in the changer section an ok place to add this parameter as below? Yes. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Wed, Oct 6, 2010 at 10:10 AM, Jon LaBadie j...@jgcomp.com wrote: Again, if trying from the server, might this be yet another tar version incompatibilities? This is unlikely, but worth checking out all the same - can the version of tar that amrecover is using successfully extract the amfetchdump'd tarfiles? If so, can you wrap that tar in a shell script so that it also writes the datastream to a permanent file, and then compare the known-good and known-bad datastreams to see what's going on? Your hypothesis about a difference in offsets makes sense, but maybe there's just random corrupt data in there. It would be good to know at what offset the corruption begins. Dustin -- Open Source Storage Engineer http://www.zmanda.com
3.2.0 beta3 release
Thank you to everyone who tested beta2 - we caught a number of bugs, none of which are huge, which gives me confidence that the upcoming 3.2.0 release will be a sturdy one. As promised[1], we have another beta release available this morning. Please test! Those of you who have been holding off on testing amvault based on my warning last week: it's fixed, so please have at it. (By the way: three releases in one week! Whew!) Find the tarball and pre-built binaries in the usual places: - http://amanda.org/download.php - http://www.zmanda.com/download-amanda.php The following fixes are included in 3.2.0 beta3: - fix dumper to process encryption parameter correctly (Stefan) - always set the current slot, even when searching by label (Gene) - print all DLEs in the postscript labels (Paul Yeatman) - process events before timeouts, in case Amanda blocks for a long time - minor fixes for convert-zd-mtx-to-robot.sh - amvault: use tapetype splitting parameters, rather than splitting unconditionally at 2 MB (therm) - fix amtape --help (Jon LaBadie) - report re-flushed dumps (Gunnar) - recovery-limit fixes (therm) - handle negative taper durations better (mezzie) - better cleanup on device finish (Robert) - test threading on FreeBSD better (John Hein) I'd like to see if we can get Erich's bug fixed, although I'm not yet convinced that it's a new bug in 3.2. NOTE: none of the 3.2.0 beta releases exhibit the vulnerability announced yesterday for 3.1.2. At this point the Amanda team is targetting a 3.2.0 release around Monday the 18th, but that is of course subject to change. Also, there are strong rumors of a 3.1.3 ZWC release coming soon that will fix a number of bugs, including the two Jon has been chasing down. Dustin [1] Almost: I messed up the beta2 release, so we're on to beta3 already -- Open Source Storage Engineer http://www.zmanda.com
Re: [Amanda-users] amrecover 3.1.1 issue
On Tue, Oct 5, 2010 at 9:29 AM, Jean-Louis Martineau martin...@zmanda.com wrote: The value of '-1' show that something is corrupted, the backup image is probably not valid, that's why you can't restore it. Can you post the complete log.timestatmp.0, the amdump.? log file and the dumper.*.debug file. You're right - this suggests something very wrong in the original backup. That said, I'll adjust Amanda::DB::Catalog to not die on such logfiles. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: [Amanda-users] amrecover 3.1.1 issue
On Tue, Oct 5, 2010 at 11:39 AM, mezzie amanda-fo...@backupcentral.com wrote: The logs here are on a different date than the original post but I have verified that it is still happening for the files attached. So it looks like mvsu.edu:/home/tk20/fileshighered is a problematic DLE. Looking at when that particular DLE was dumped: 7998 driver: result time 666.284 from dumper2: DONE 02-00019 1290 33 0 [sec 0.295 kb 33 kps 111.8 orig-kb 1290] it was compressed from 1290k to 33k! In fact, all of your dumps are compressing to virtually nothing. I don't think bzip2 is doing the job for you. Does bzip2 need an argument to compress stdin? I think this is the core of your problem. I'm not sure why the taper wrote 'sec -1.', when the taper run actually took 0.045 seconds, but that's secondary to the problem, and has, I believe, been remedied in newer versions. Dustin -- Open Source Storage Engineer http://www.zmanda.com
[SECURITY ALERT] Vulnerability in Amanda-3.1.2 - Upgrade to 3.1.3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The Amanda development team has discovered a security vulnerability introduced in Amanda-3.1.2. The vulnerability affects both Amanda servers and clients, and could lead to remote execution of code as the Amanda user. The problem is fixed in Amanda-3.1.3, which is released today. It is a patch release in the 3.1.x series, and contains the fix for this security vulnerability as well as fixes for several other significant bugs in 3.1.2. This upgrade should be considered a high priority upgrade. Updated images and tarballs are available at - http://www.amanda.org - https://sourceforge.net/project/showfiles.php?group_id=120 - http://www.zmanda.com/download-amanda.php A permanent copy of this alert is at - http://www.amanda.org/secalert-3-1-2.php The MD5 sum for the release tarball is f72fa06d4f90f7997b17ded1d7b4314e Dustin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyrazUACgkQdiVAPX8NFbFtLwCfW+Sb1MRoPTU9KSMaIud3Pai2 Ol0AoOKUknQQs20TqBImE231GzDKuUCN =oT6b -END PGP SIGNATURE-
3.2.0beta2 tomorrow - last call!
We're planning another beta tomorrow, so if you have anything you'd like to see fixed in that beta, let me know ASAP. That includes manpage typos and other actual nitpicks :) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re:
On Tue, Oct 5, 2010 at 1:33 PM, McGraw, Robert P rmcg...@purdue.edu wrote: hertz /gauss/export/users-s lev 0 partial taper: Error writing block: Mysterious short write on tape device: Tried 2097152, got 1052672 as amanda hit the end of tape. .. and you're using a block size (2M) that's bigger than your tape drive can handle (1M). As for the device-busy error, do you get this error if you don't see the Mysterious short write? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Tue, Oct 5, 2010 at 5:01 PM, Charles Curley charlescur...@charlescurley.com wrote: Done. I also upgraded to amanda-3.2.0beta1.svn.3489.tar.gz, per today's announcement. I just ran 15 amchecks in a row, then an amdump, and no problems. Great! I should be clear - and I'll reiterate in tomorrows beta2 announcement - 3.2.0beta1 does not exhibit the security vulnerability in 3.1.2, and 3.2.0beta2 will not, either. As for options to configure, major ones (and other goodies) are documented at http://wiki.zmanda.com/index.php/Installation/Installing_Amanda_Source Good; now I can get rid of a lot of stuff I don't need. Ah, cool - thanks! Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amreport version 3.1.2
On Sun, Oct 3, 2010 at 1:46 PM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Yes this is much better - log is attached. Fixed up in http://github.com/djmitche/amanda/commit/z12092.patch Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amreport version 3.1.2
Incidentally, I can't help but notice that you're backing up disks named /var/amanda/amandatapes/ems/slotNN. Will amvault be able to help you out? If so, could you find some time to test it before I set sail in a week and a half? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amreport version 3.1.2
On Mon, Oct 4, 2010 at 1:59 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: Yes I hope so. What version should I go for ? This - or rather a version improved by Jean-Louis' suggestion to print the more precise flush successfully retried - will be in the next 3.2.0 beta, which we're considering releasing midweek. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Compile time enhancement: make the amanda user directory owned by the amanda user
On Mon, Oct 4, 2010 at 2:09 PM, Charles Curley charlescur...@charlescurley.com wrote: It might be nice if the installation process not only made the directory but changed its ownership to the amanda user. The only reason that Amanda references its home directory is for .amandahosts, which is modeled on .rhosts (and you've all disabled rhosts and telnet, right?), and for encryption keys (where gpg tends to use $HOME unless instructed otherwise). I consider both an unfortunate legacy. So I'm loathe to tread on territory that should belong to the system itself. That directory should be created and populated by useradd or whatever utility you use to create the necessary user account. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Compile time enhancement: make the amanda user directory owned by the amanda user
On Mon, Oct 4, 2010 at 3:28 PM, Charles Curley charlescur...@charlescurley.com wrote: Understandable. Yes, useradd should set it up correctly. However, on debian and Ubuntu, the default user directory, /var/backups, is already created by the time you run make install, and so is the backup user. And debian uses that directory for something else. Well, /var/backups is an odd home directory, and is a debian-specific choice, so the deb/ubuntu folks should take care of making sure that works. I think the right solution here is to somehow optionally configure Amanda to use /etc/amanda/amandahosts, and to update amcrypt to pass explicit paths to gpg. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Mon, Oct 4, 2010 at 9:45 PM, Charles Curley charlescur...@charlescurley.com wrote: I'm no expert on configure, so that could be wrong. BTW, are the options for configure documented anywhere? Only in ./configure --help. It looks like you got things figured out, though. That done, I was able to get an amckeck, a backup and a recovery, verified by diffing the two trees. With 1 DLE, amcheck takes about 7 seconds to check the client. Great! However, approximately every other use of the client results in no service at the amanda port: # netstat -a | grep amanda # This sounds like an xinetd bug, somehow. Are you using UDP or TCP? Are the stream/dgram settings appropriate to that? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Mon, Oct 4, 2010 at 10:30 PM, Charles Curley charlescur...@charlescurley.com wrote: I plan to go back to tcp and auth=bsdtcp once I get this issue solved. Of course it could be that making those changes will solve this issue. I'd recommend that. I consider BSD and BSDUDP authentications to be legacy backward-compatibility modes. TCP/IP is the wave of the future! (also, SSL which incidentally looks like it will wait until Amanda-3.3). Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amreport version 3.1.2
On Sun, Oct 3, 2010 at 7:22 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: I upgraded Amanda version from 2.6.1 to 3.1.2 and amreport reports: ... But they are written on the next tape, I'm not using tape spanning. OK .. so what's the problem? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Confused by amanda's 'planner': why multiple level 0?
On Sat, Oct 2, 2010 at 5:01 PM, Robert Heller hel...@deepsoft.com wrote: I picked a 'virtual' tape size to match the capacity of a DVD-R: 4.3Gig, with the idea of migrating the fulls and the more major incrs to DVD-Rs for long-term archival. Ah! You should take a look at both the dvdrw device (for writing to DVDs) and at amvault (for copying dumps to other volumes). In 3.2, amvault will be able to do exactly what you want - pluck the fulls off vtapes and put them on DVDs. Then you can adjust your vtape size to something that fits your backup schedule better. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amreport version 3.1.2
On Sun, Oct 3, 2010 at 10:26 AM, Gunnarsson, Gunnar gunnar.gunnars...@svk.se wrote: It is reported as failed see below and those parts are added twice - filesystem 66 parts 68. In earlier version it is reported correctly. Perhaps this is a change in behavior, but I think that this version is arguably more correct: the taper *did* tape 68 parts, two of which were partial. And there *were* two failures, athough the report should also mention that they were successfully retried, e.g., hansabck /var/amanda/amandatapes/ems1/slot13 lev 0 partial taper: No space left on device hansabck /var/amanda/amandatapes/ems1/slot13 lev 0 successfully retried hansabck /var/amanda/amandatapes/ems1/slot65 lev 0 partial taper: No space left on device hansabck /var/amanda/amandatapes/ems1/slot65 lev 0 successfully retried Does that sound about right? Can you send a copy of the trace log that produced this, and I will add the successfully retried? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Confused by amanda's 'planner': why multiple level 0?
On Sun, Oct 3, 2010 at 11:01 AM, Robert Heller hel...@deepsoft.com wrote: Also I am pretty much stuck with 2.5.0, since that is what comes with CentOS... (I don't at this point want to 'experiment' with a bleeding edge self-built package, not for something like this.) Yikes, you may be *very* out of luck, then. 2.5.0 is four and a half years old - long before I began working on Amanda. You should consult the manpages for 2.5.0 (presumably included with your distro) to see how various parameters work - much has changed since then. In particular, look at the bump parameters, and change your runtapes to 1. You're in a pretty constrained set of circumstances, and I'm not sure 2.5.0 was flexible enough to get you what you want. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Confused by amanda's 'planner': why multiple level 0?
On Sun, Oct 3, 2010 at 12:35 PM, Robert Heller hel...@deepsoft.com wrote: One of the downsides of using a very *stable* Linux distro with long term stable support. OTOH, it avoids the fun of re-installing everything every 6-12 months and then spending a couple of months getting all of the settings tweaked just right... Sure, I understand - I just can't offer you much help in that case :) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Sat, Oct 2, 2010 at 5:16 PM, Dustin J. Mitchell dus...@zmanda.com wrote: OK, in trying to duplicate this, I'm getting dumper segfaults, which is probably the same bug -- it looks like dumper is printing random memory in the error message above. So consider it replicated - and I'll try to get a fix put together this weekend. OK, here's a partial fix. Can you confirm that this works for you? http://github.com/djmitche/amanda/commit/z12091.patch Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Sun, Oct 3, 2010 at 4:30 PM, Stefan G. Weichinger s...@amanda.org wrote: Sorry, it doesn't apply when using my amanda-3.2.0_beta1.ebuild ;-( Will look into it tomorrow. There's a version rebased onto the 3.2.0_beta1 tag at http://github.com/djmitche/amanda/commit/z12066.patch Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Sun, Oct 3, 2010 at 5:37 PM, Stefan G. Weichinger s...@amanda.org wrote: applies and seems to work. encrypted DLE dumped. Great! I'll wait to hear back from Jean-Louis about the potential memory leak, then, before committing. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: bumpdays clarification
On Sat, Oct 2, 2010 at 9:21 AM, Jon LaBadie j...@jgcomp.com wrote: Thinking about a recent post caused me to wonder about the bumpdays directive. Is it truely measured in days as opposed to amdump runs? Neither, actually - it's measured in tapes / runtapes. So if your runtapes is large but seldom reached, you may go significantly longer than bumpdays days even with only one run a day. Note that ordinarily Amanda's concept of day is related to runs by the ratio dumpcycle (days) : runspercycle (runs), although that is not the case for bumpdays. The difference between runs and days only became apparent with the addition of usetimestamps, and wasn't completely integrated. A similar thing happend with the difference between parts and dumps when splitting was introduced, which led to a lot of difficult-to-reproduce quirks of amreport (for example, in some cases it considers a dump to be on a particular tape if the *last* part of the dump is on the tape, while in other cases it considers a dump to be on a tape if the *first* part of the dump is on the tape). Not to get all nostalgic already, but as Amanda development continues, it's important to try to incorporate changes like these *fully* into Amanda, preferably through the creation of better abstractions, rather than loosely bolting them on. Jon, if you and I and others can keep an eagle-eye on changes to Amanda for that purpose, I think it would do the project a world of good. To that end, we should all probably have a hard look at the results of amvault (there are some known gotchas in the manpage, and probably some unknown) and the multi-taper support (I'm sure there are some edge cases here; I just haven't found them). That's a big part of the rationale for the 3.2.0beta1 release. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Confused by amanda's 'planner': why multiple level 0?
On Sat, Oct 2, 2010 at 9:15 AM, Jon LaBadie j...@jgcomp.com wrote: runtapes 2 # number of tapes to be used in a single run of amdump I hope very few dumps take more than one tape, like none :) To be clear on the planner's reasoning: it considers itself to have runtapes * tapetype:length kb available to play with. So it very well may be *promoting* level-0's in order to try to fill both tapes. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Confused by amanda's 'planner': why multiple level 0?
On Sat, Oct 2, 2010 at 2:46 PM, Robert Heller hel...@deepsoft.com wrote: So I would be better off setting runtapes to 1? And then manually 'flushing' at the beginning of the cycle to deal with the larger fulls? Arg... This sort of thing is not really well explained in the man pages... True. The problem is, there's really nobody who knows the details except by looking at the code. The discussions of tapecycle about a year ago are a good example: I think a lot of long-time users were shocked to read amanda-taperscan(7) and learn that their ideas of how taperscan worked were very inaccurate. Anyway, after you've resolved things, I'd love to see additions either to the wiki or to the manpages. Amanda is very difficult to configure if you want to trick the planner into doing something specific. Instead, look at the problem and see how you can get Amanda to solve it for you. As I understand it, you have a number of large level-0's, you want to retain your backups for a long time, and your disk space is very limited. First, I would recommend making your tapes a little bit larger, so that each night's level-0's and level-1's can fit comfortably. This may require reducing your retention period slightly, or getting more disk. Second, since you want to use the space on your tapes to their fullest, consider using the flush-threshold parameters to fill tapes to 100% - there's a How To on the wiki. If you do this, you may want to divide your vtapes by three or four so Amanda has a finer-grained tape size to work with (that is, create three times as many vtapes, each one-third their current size, and use runtapes=3). Finally, lengthening your dumpcycle will allow Amanda to do more incrementals. I don't recall the current relationship of tapecycle to dumpcycle, but you can work that out. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: trivial amtape defect
On Sat, Oct 2, 2010 at 1:03 PM, Jon LaBadie j...@jgcomp.com wrote: When running amtape --help the usage message is printed twice. A *real* nitpick! Cool! Fix is here: http://github.com/djmitche/amanda/commit/z12091 If that works for you, let me know and I will commit. You should be able to apply it directly to /usr/sbin/amtape. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0 beta1 release
On Thu, Sep 30, 2010 at 12:33 PM, Stefan G. Weichinger s...@amanda.org wrote: ? encrypt: dumper: error: couldn't exec server encryptionÈÏ HÃ 8 . OK, in trying to duplicate this, I'm getting dumper segfaults, which is probably the same bug -- it looks like dumper is printing random memory in the error message above. So consider it replicated - and I'll try to get a fix put together this weekend. Dustin -- Open Source Storage Engineer http://www.zmanda.com
amvault oversight
FYI, in 3.2.0beta1 amvault always uses a part size of 2MB, which can cause *huge* logfiles for large (or even medium-sized) dumps. This is an oversight on my part, and I'll get it fixed up, but in the interim, don't vault anything too large in 3.2.0 or your logfiles will get huge and thus your recoveries slow. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Wed, Sep 29, 2010 at 7:39 AM, Gene Heskett gene.hesk...@gmail.com wrote: For SG, I removed the data link from the test setup. Running amcheck test does restore it. Removing it from the running setup and running amcheck Daily does not. But it doesn't seem to effect amdump or amcheck in any way. So we need to figure out what's different between these two. Any ideas? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Wed, Sep 29, 2010 at 1:12 AM, Titl Erich erich.t...@ruf.ch wrote: It takes a while and I can observe data transfer with tcpdump. OK, I suspect, then, that this is a timing-related bug in 3.1.2, that is fixed in the 3_1 branch. Do you have the capacity to build from a tarball on the server? I don't have any pre-built packages for that branch at the moment, but I may be able to get one for you if necessary. If you can build from a tarball, please pick the latest amanda-3.1.3 snapshot from http://www.zmanda.com/community-builds.php and see if that fixes things. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Wed, Sep 29, 2010 at 10:29 AM, Titl Erich erich.t...@ruf.ch wrote: Downloading the current snapshot, could you provide compiling and packaging information? The instructions for compiling from a tarball are here: http://wiki.zmanda.com/index.php/Installation/Installing_Amanda_Source If you want to build a .deb file, you can do so easily (hopefully) by running ./packaging/deb/buildpkg from the root of the untarred tarball. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: iSCSI holding disk: slow read.
On Wed, Sep 29, 2010 at 11:46 AM, Valeriu Mutu vm...@pcbi.upenn.edu wrote: What do you mean by shoe-shining? shoe-shining is when a tape drive must stop the tape repeatedly while it buffers more deta. It creates a lot of wear on the tape, and also kills performance. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Wed, Sep 29, 2010 at 11:33 AM, Gene Heskett gene.hesk...@gmail.com wrote:
Note the --with-config=Daily \, but the subdir when running am* Daily is
/amandatapes/Dailys, note the plural. Is this line even required?
No, --with-config is largely ignored, and the name of the vtape
directory need not match the name of the configuration.
The test amanda.conf is the Daily/amanda.conf copied to the $configdir/test
tree, with all instances of 'Daily' or in the case of vtape paths, Dailys,
edited to be 'test' in those strings. And the test checks are running the
code built by the above gh.cf file. Could this missmatch be a problem with
the new code?
It's certainly possible that there's a bug here - I'm just baffled as
to what's going on.
Now, I do see one item that will need moved to the
/usr/local/var/amanda/configname directory, amandates exists as
/usr/local/var/amanda/amandates that will not do for a test run of
'test'.
Humm, that location is in the Makefiles as a default. So is it definable in
the /configdir/amanda,conf? The man pages for amanda.conf don't mention it.
If this move of amandates can be done by moving it to
/usr/local/var/amanda/configname/amandates, I believe this is the last
connection that exists between the Daily and test configs.
Please add that ability to parse /configname/amanda.conf for the location of
the amandates file.
The location for 'amandates' can only be set at configure time.
However, amandates is rarely, if ever, used, so I don't think that the
overlap is a problem.
Can you add some debug prints to Amanda/Changer/disk.pm?
494 # TODO: locking
Amanda::Debug::debug(set_current: symlinking $curlink to slot$slot)
495 symlink(slot$slot, $curlink);
and
471 if (-l $curlink) {
472 my $target = readlink($curlink);
Amanda::Debug:debug(get_current: $curlink = $target);
473 if ($target =~ ^slot([0-9]+)/?) {
and see what appears in the amcheck-device debug log?
Dustin
--
Open Source Storage Engineer
http://www.zmanda.com
Re: Corrupted dumps from windows client
On Wed, Sep 29, 2010 at 12:48 PM, Christian Kratzer ck-li...@cksoft.de wrote: it looks like I might have found the problem. Both windows hosts had the shadow copy service disabled. I startetd the service and will check tomorrows full dumps. Can you post some info to the wiki about how to check and enable VSS? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Wed, Sep 29, 2010 at 12:53 PM, Chris Nighswonger cnighswon...@foundations.edu wrote: Line 473 should have a double colon rather than a single... ie. Amanda::Debug::debug rather than Amanda::Debug::debug Sorry about that, Gene. You can just edit the file under /usr, rather than re-making Amanda, if that's easier. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Wed, Sep 29, 2010 at 1:51 PM, Gene Heskett gene.hesk...@gmail.com wrote: Wed Sep 29 14:41:23 2010: amcheck-device: set_current: symlinking /amandatapes/test/data to slot1 And is this link in place on the filesystem now? What's the difference between the 14:31 and 14:41 runs of amcheck? Dustin -- Open Source Storage Engineer http://www.zmanda.com
3.2.0 beta1 release
As promised, we have the first beta version of Amanda-3.2.0 out today. It's available from http://www.zmanda.com/download-amanda.php (binary packages; version 3.2.0beta1) http://amanda.org/download.php (tarball; under Development) The more testing this release can get, the better. And the more that can happen before I leave (two weeks from today) the better for Jean-Louis' sanity. In particular, please test-drive the new features. If you can email the list to let everyone know you're testing, and of course let us know whether things succeed or fail, that would be great. As always, I'm in #amanda on freenode if you'd like interactive help testing or debugging, or just want to yell at me for introducing so many new bugs. - multiple simultaneous tapers (!!) This is a big one, obviously. See http://wiki.zmanda.com/index.php/How_To:Write_to_Multiple_Volumes_in_Parallel - LEOM support (lossless splitting) If you're using tapes, you will need to set the LEOM property to TRUE to take advantage of this. If you're using vtapes, it's enabled by default. In concert with the new splitting parameters, this can make your backups faster and eliminate the need for part caches. http://wiki.zmanda.com/man/amanda-devices.7.html - new, simpler splitting parameters These are part-cache-size, part-cache-type, part-cache-dir, and part-cache-max-size. They are described in amanda.conf(5), including in a special section at the very end. http://wiki.zmanda.com/man/amanda.conf.5.html - massive amvault improvements Give it a whirl! http://wiki.zmanda.com/index.php/How_To:Copy_Data_from_Volume_to_Volume - retirement of most old changers Not much to test here, but FYI.. - per-client recovery permissions A last-minute addition! See http://wiki.zmanda.com/index.php/How_To:Limit_the_Hosts_That_Can_Recover_a_DLE I'd appreciate not only a test-drive, but some additional code review here, since it's a security-sensitive change: http://github.com/zmanda/amanda/commit/8b874f92df6ee17a335a1ba9089bf94d43a09269 - SSL authentication this didn't make it in yet, although we're considering adding it during the beta period. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
When you do the recovery with amrecover, does it take a while for the tar error to appear, or is it immediate? I'd like to figure out if it's transferring any data at all. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Sun, Sep 26, 2010 at 6:03 PM, Dustin J. Mitchell dus...@zmanda.com wrote: That said, the new chg-disk *is* supposed to duplicate the data symlink quirk, so I'll take a look at why it's not doing that. And I just verified that this works for me -- it's also tested for by the installchecks. Can you do an ls -l of your vtape root directory (the one containing slot1, slot2, ..), then use 'amtape' to switch to a different slot, and do another ls -l? Also, what are tpchanger and tapedev set to in your configuration? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Tue, Sep 28, 2010 at 6:26 PM, Gene Heskett gene.hesk...@gmail.com wrote: I now see a 'data' link to slot 5, I nuked that 2 days ago because nothing was paying an attention to it, and neither amcheck nor amdump was updating it, and the loss causes no errors, none, nada, zip. My helper script no longer pays any attention to it either. That's really weird. I can't replicate this behavior at all. Having reviewed the code, chg-disk actually looks at what the 'data' symlink is pointing to in order to determine the current slot. In particular, I don't see why amtape would set the symlink, but amcheck and amdump would not. Reasons I can think of off the top of my head: - permissions problems - wrapper scripts? - multiple copies of Amanda installed? - voodoo? Can you set up a smaller config without using your wrapper script and with a fresh set of vtapes and a simple DLE (/boot or something like that), and see how amtape and amcheck behave? Don't run amdump.. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Tue, Sep 28, 2010 at 9:06 PM, Gene Heskett gene.hesk...@gmail.com wrote: Anyway, its setup, and awaiting orders Cap'n. Next? Can you replicate any of the failure-to-update-data errors with this test config? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Mon, Sep 27, 2010 at 1:12 AM, Titl Erich erich.t...@ruf.ch wrote: Just before you drop the ball completely, have you or someone else had time to look at the debug files I had skimmed it briefly, and didn't see anything obvious. I suspect that you've fixed the basic problem by changing FSF_AFTER_FILEMARK. With that in place, can you try running amfetchdump (which is basically a command-line version of amidxtaped) for that dump and see what comes out? Is the result a truncated version of the dumpfile? Is there any extra data in the dumpfile? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: FW: amrecover bug?
On Mon, Sep 27, 2010 at 12:05 PM, Titl Erich erich.t...@ruf.ch wrote: Here for your reference the outcome of amfetchdump does not look good subversion:/backup# amrecover that's amrecover. Is there any extra data in the dumpfile? Not that I know of. During amrecover I can observe the data stream and I _hope_ it is not garbled to an extent that it is not processable. It takes the amount of time I would expect to extract the file and send it across the network to the client. On the server I can extract the respective file easily and also feed it successfully to tar (I have not tried dump/restore anymore). Meaning with 'mt' and 'dd'? That's good to know! Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amanda and selinux
On Tue, Sep 28, 2010 at 12:21 AM, Jon LaBadie j...@jgcomp.com wrote: Including appropriate selinux policies for Fedora builds might be another feature enhancement request or nitpick. I'd like to see them, too, but I'm not sure how quickly they'd get out of date. They'd need someone savvy in the ways of selinux who was willing to write them in a maximally flexible way, and keep them up to date. That's probably more than just using the permissive mode and recording what Amanda does. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Sun, Sep 26, 2010 at 6:05 AM, Gene Heskett gene.hesk...@gmail.com wrote: At some point in the last roughly a month, amanda stopped using the link named 'data' to point at the correct slot in a vtape directory. I think you figured out, in a previous thread, that you switched to the new chg-disk around the beginning of August. I have now patched my script for new means of assuring my data files go into the correct subdir of a vtape setup, but it will take a month or so to get my database all in step again. Is anyone else actually using GenesAmandaHelper-0.6? I can send you the two scripts that are changed privately, just yelp. There's a copy of the tarball on the wiki, so perhaps you can upload a new copy? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Amanda changes bite me again.
On Sun, Sep 26, 2010 at 4:34 PM, Gene Heskett gene.hesk...@gmail.com wrote: Yes I did, but that didn't give me a clue that the 'data' link to whatever had been deprecated and that for cleanliness, I should go around an nuke it. No mention of it in the ChangeLog, I read it again early this morning from about the middle of July. The new changer was added sometime before that. YOU changed your configuration to use it in August. In retrospect, we should have named it something different -- perhaps chg-vfs or the like -- to avoid this confusion. It's a completely reimplemented changer, and isn't intended to reproduce every quirk of the old one. That said, the new chg-disk *is* supposed to duplicate the data symlink quirk, so I'll take a look at why it's not doing that. Dustin -- Open Source Storage Engineer http://www.zmanda.com
3.2.0 beta on Wednesday .. and a personal note
As we've hinted in various posts here, there's a lot of exciting new stuff coming up in 3.2: - multiple simultaneous tapers (!!) - LEOM support (lossless splitting) - new, simpler splitting parameters - massive amvault improvements - retirement of most old changers - a great deal of code cleanup and bug fixes - (hopefully) per-client recovery permissions - (hopefully) SSL authentication On a personal note: I've enjoyed working on Amanda, and hopefully served the project well, but the time has come for me to move on. I've accepted a position with Mozilla, and that means, sadly, a significant reduction in the time I have to spend on Amanda. In the interests of giving me a chance to fix some of the bugs I have introduced during my time here, we will be building a beta version of 3.2.0 on Wednesday the 29th. Since I will only be around for two weeks afterward, it is critical that we get as much testing as possible on this beta. I know Gene and Stefan are already debugging the bleeding edge, and I very much appreciate their hard work. It would be great if the rest of the regulars on the list could set aside some time and a virtual machine or other test environment for next week. Lest you were thinking of celebrating - you're not completely rid of me! It's my turn to practice what I've been preaching for so long: contributing to Amanda in spare time. I will keep submitting patches to Amanda, and I will continue to provide technical advice to other contributors on IRC and on the amanda-hackers list. To keep my sanity, though, I will have a much lower profile on this mailing list. This is a great support community, and I expect that to continue unabated in my absence! I'd also like to extend my gratitude to Zmanda for funding, supporting, and making public my work, and that of others, on Amanda. I know the company, through the work of Jean-Louis and others, will continue its dedication to Amanda and I expect to see more exciting new developments in upcoming versions. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: not a nitpick, feature question
On Thu, Sep 23, 2010 at 8:17 AM, Gene Heskett gene.hesk...@gmail.com wrote: This discussion has been a positive, showing that there are indeed several ways to approach this problem, all of which, when TSHTF, contain the seeds for an expedited full recovery. Great! This probably deserves a HowTo and/or FAQ on the wiki. Do you want to get that started, and maybe the others in this conversation can add their suggestions? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Gentoo SVN ebuild buglet
Stefan -- With the way the ebuild is structured right now, Amanda's built with dummy manpages: dus...@euclid ~/code/amanda/t/amanda [z11904 *] $ man amreport DUMMY You may want to consider adding configure flag --enable-manpage-build to the ebuild, and adding =app-text/docbook-sxl-stylesheets-1.72.0 app-text/docbook-xml-dtd to DEPENDS (you may need mutiple docbook-xml-dtd versions, but I don't know how to specify a slot in DEPENDS) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: not a nitpick, feature question
On Thu, Sep 23, 2010 at 9:08 AM, Gene Heskett gene.hesk...@gmail.com wrote: Do I have editing rights on the wiki? Everybody does (even spammers, ugh) - you may need to create an account first, but that's easy. Here's the direct link: http://wiki.zmanda.com/index.php?title=Special:UserLogintype=signupreturnto=Main_Page Everyone on the list should feel encouraged to edit the wiki, either fixing mistakes or adding new content. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpicks?
On Thu, Sep 23, 2010 at 1:28 PM, Stefan G. Weichinger s...@amanda.org wrote: A flag to set in amanda.conf to force an mt -f $tapedev offl after successful amdump/amflush ... This has been a standing bug in Zmanda's bugzilla for a while, but it's never become clear where this should happen. Changers already support an eject() operation as well as an eject = 1 argument to the reservation release, so maybe it belongs in the taper? By the way, amvault just grew an --export option that will move tapes to import-export slots. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpicks?
On Thu, Sep 23, 2010 at 2:20 PM, Stefan G. Weichinger s...@amanda.org wrote: I would like to see it in my smaller installations as well, where there is no changer ... Well, there's always a changer, but some of them (like chg-manual and chg-single) do not support ejecting. That might be a good thing to fix. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: amrecover bug?
On Wed, Sep 22, 2010 at 3:17 AM, Titl Erich erich.t...@ruf.ch wrote: A few weeks ago I reported in the forum a problem with amrecover and compressed dump files. Meanwhile I changed to uncompressed backup, still no luck and, unfortunately not much replies either. So please bear with me when I repeat this here. No worries about the repetition - this mailing list is the more actively-monitored spot. My first guess is that Amanda is not seeking to the required file appropriately. Can you let us know what OS you're using, what type of tape, and also send along the amrecover debug log file from the client and the amidxtaped debug log file (with matching datestamp) from the server? Have a look at the FSF_AFTER_FILEMARK property in amanda-devices(7), as that's my first guess for what's going wrong. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: not a nitpick, feature question
On Wed, Sep 22, 2010 at 10:07 AM, Jon LaBadie j...@jgcomp.com wrote: Any thoughts on how desireable you feel be a separate copy of amanda data would be and other approaches? This comes up often, and I've never found a solution I'm happy enough with to make the official solution. Gene's approach is the obvious one, but has a few limitations: - What do you do if you run out of space on that tape? Start a new tape? How do you reflect the use of that new tape in the catalog? - How does recovery from that metadata backup work? There's a chicken-and-egg problem here, too - you'll need an Amanda config to run any Amanda tools other than amrestore. Let's break down metadata into its component parts, too: 1. configuration 2. catalog (logdir, tapelist) 3. indexes 4. performance data (curinfo) 5. debug logs Configuration (1) can be backed up like a normal DLE. The catalog (2) should technically be recoverable from a scan of the tapes themselves, although the tool to do this is still awaiting a happy hacker to bring it to life. Indexes (3) are only required for amrecover, and if your Amanda server is down, you likely want whole DLEs restored, so you only need amfetchdump. Performance data (4) will automatically regenerate itself over subsequent runs, so there's no need to back it up. Similarly, debug logs (5) can get quite large, and generally need not be backed up. So, to my mind, the only component that needs special handling is the catalog, and we have a menu of ways to handle that: - append a copy of the entire catalog to the last tape in each run (hey, what is the last tape now that we have multiple simultaneous tapers?) - append a copy of only the catalog for that run to the last tape in each run - finally get around to writing 'amrecatalog' - rsync the entire catalog to another machine nightly I just stuck that last one in because it was my technique back when I managed a fleet of Amanda servers. Each would simply rsync its config and catalog to the other servers. Since they were all backing up to shared storage (a SAN), I could do a restore / amfetchdump / recovery of any dump on any server without trouble. It's a very non-Amanda-centric solution, but it's *very* effective. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: not a nitpick, feature question
On Wed, Sep 22, 2010 at 8:05 PM, Christ Schlacta aarc...@aarcane.org wrote: Why keep a backup server up 24/7 when it only works durring bakup hours? Often the backup backup server is a server ordinarily devoted to other purposes, with a few megs available for catalogs. Why not append the entire catalog to each tape? Or at least the entire catalog to date? Well, the second question suggests a strict interpretation of entire as all tapes written past, present and future. And obviously that's impossible without time-travel. :) I did outline some of the downsides of writing the present catalog, or (worse) past and present, to tape. In particular, the EOM problem worries me, since a lot of Amanda users now utilize the flush-threshold-* parameters to fill tapes to 100%, leaving 0% over for the catalog. Tapes don't generally give you any indication of how close they are to EOM, so there's no good way to say my catalog's about 30M, so only write dumps up until 30M from the end. It's a bit of a wonky solution to get catalog data that may be better backed or regenerated up in other ways. That's the bottom line, IMHO. By the way, the amrecatalog utility that I mentioned earlier would be pretty easy to write. Amrestore would provide a nice basis for it. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Mon, Sep 20, 2010 at 11:20 PM, gene heskett ghesk...@wdtv.com wrote: with taper_debug 9 I had a 174 meg taper log, which still didn't make any sense as it was reporting about 6 lines for every 32k write, so I dropped it to 2. I grepped for the differences in the man tree but didn't see anything about the verbosity diffs. So now it's writing to tape. So the problem is solved. Does it not write to tape with a lower debug level? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpick - amreport statistics
On Mon, Sep 20, 2010 at 6:56 PM, Jon LaBadie j...@jgcomp.com wrote: Looks better to my eye. Any one else? With Jean-Louis' review, committed in r3428. Thanks! Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
Gene, this is counterproductive at this point. You've named a few things that seem to be wrong, all of which are unrelated. You've changed things between each email, and claimed a number of different symptoms. PLEASE: slow down. Find *one* failure, describe it fully, and track it down to its cause, using logfiles if possible. Be as methodical as possible. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpicks?
On Tue, Sep 21, 2010 at 1:26 PM, Chris Nighswonger cnighswon...@foundations.edu wrote: This may be a little late, but it would be nice to have a switch on most status, etc. utils to format the output for texting to a mobile device rather than email. Almost the exact same thing was requested earlier by Douglas K. Rand. There are a number of suggestions to change the way notifications are done in Amanda. And from experience, if those are implemented, it will generate an even larger number of requests (HTML emails, mobile-friendly HTML emails, Growl notifications, ..). As I mentioned earlier, I think the better approach is to design a generic notification interface into Amanda, and let other applications do the required formatting/filtering. That said, I wouldn't object to a patch to amcheck to generate pages in the case of an error. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Mon, Sep 20, 2010 at 3:21 PM, Dustin J. Mitchell dus...@zmanda.com wrote: It's impractical to run this check in ./configure, but what do you think about running this check even on 'make all'? I committed this - on FreeBSD, this test will occur on 'make install'. Do you mind adding a FreeBSD problem report, if there's not one already? Can you give me a pointer so I can add myself as cc? This would still be great.. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Mon, Sep 20, 2010 at 2:44 PM, Stefan G. Weichinger s...@amanda.org wrote: Added that debug-parameter and re-ran amflush, nothing special in the logs. What does this mean? Did the taper start up but not write anything? And what revision are you using? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sun, Sep 19, 2010 at 4:48 PM, Stephen Corbesero corbes...@ptd.net wrote: That did it Or may have done it. Sounds like this has worked out. I had added a test to Amanda, which runs in 'make check' that should have detected this: 536 thread-check: libTests.la 537 $(PERL) -I$(builddir) -I$(builddir)/.libs -I$(srcdir) \ 538 -MAmanda::Tests -e 'alarm(10); Amanda::Tests::try_threads' \ 539 || echo Perl cannot run extensions which use threads; consider linking perl \ 540 with -pthreads or compiling perl with threading enabled I suppose we should be more aggressive about running this test, at least on FreeBSD, and at least until this is fixed or worked around in the FreeBSD ports. It's impractical to run this check in ./configure, but what do you think about running this check even on 'make all'? Do you mind adding a FreeBSD problem report, if there's not one already? Can you give me a pointer so I can add myself as cc? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Mon, Sep 20, 2010 at 3:25 PM, Stefan G. Weichinger s...@amanda.org wrote: does that help? Perhaps. I'm strongly suspecting that this is a problem with the new multi-tape support. Jean-Louis, do you see anything related here? I've upgrade my production Amanda to the latest trunk (thanks to the handy-dandy ebuild!), so we'll see if I encounter similar problems myself. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpick - amreport statistics
On Thu, Sep 16, 2010 at 4:52 PM, Dustin J. Mitchell dus...@zmanda.com wrote:
Are you working on this? Perhaps someone else on the list can jump in?
Hmm, well, I just took care of this:
http://github.com/djmitche/amanda/commit/z11908.patch
Total Full Incr. Level:#
Estimate Time (hrs:min) 0:00
Run Time (hrs:min) 0:00
Dump Time (hrs:min) 0:00 0:00 0:00
Output Size (meg)0.00.00.0
Original Size (meg) 0.00.00.0
Avg Compressed Size (%)100.0-- 100.0
DLEs Dumped1 0 1 1:1
Avg Dump Rate (k/s) 3750.0-- 3750.0
Tape Time (hrs:min) 0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%)0.00.00.0
DLEs Taped 1 0 1 1:1
Parts Taped1 0 1 1:1
Avg Tp Write Rate (k/s)300.0-- 300.0
Jon, what do you think?
Dustin
--
Open Source Storage Engineer
http://www.zmanda.com
Re: 3.2.0svn3411
On Sun, Sep 19, 2010 at 5:32 AM, Gene Heskett gene.hesk...@gmail.com wrote: One thing that catches my eye, in the taper.log, it is searching for: [ama...@coyote Daily]$ cat taper.20100919052724.debug Sun Sep 19 05:27:24 2010: taper: pid 5083 ruid 501 euid 501 version 3.2.0alpha.svn.3416: start at Sun Sep 19 05:27:24 2010 Sun Sep 19 05:27:24 2010: taper: pid 5083 ruid 501 euid 501 version 3.2.0alpha.svn.3416: rename at Sun Sep 19 05:27:24 2010 Sun Sep 19 05:27:24 2010: taper: Amanda::Taper::Scan::traditional stage 1: search for oldest reusable volume Sun Sep 19 05:27:24 2010: taper: Amanda::Taper::Scan::traditional oldest reusable volume is 'Dailys-21' Sun Sep 19 05:27:24 2010: taper: Amanda::Taper::Scan::traditional stage 1: searching oldest reusable volume 'Dailys-21' Sun Sep 19 05:27:24 2010: taper: Amanda::Taper::Scan::traditional result: 'Dailys-21' on file:/amandatapes/Dailys/drive0 slot 21, mode 2 Sun Sep 19 05:27:24 2010: taper: pid 5083 finish time Sun Sep 19 05:27:24 2010 [ama...@coyote Daily]$ Where is that spurious '/drive0' coming from in the file:string? False alarm - that's how the new changer works, and also explains why you were not seeing the data/ link updated. So the driver didn't even *try* to write anything to tape in the run for which you just pasted a debug log. Any idea why? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sun, Sep 19, 2010 at 11:16 AM, Stephen Corbesero corbes...@ptd.net wrote: It still seems to hang right after the message about the 'Final linkage'. That is coming from the link_elements() subroutine in xfer-src/xfer.c. Which is basically where the threading starts.. I am not sure what to do next to get more evidence. Have you checked that libc_r, glib, perl, and Amanda are all compiled with the same threading library? John Hein dug up a thread from when we tried to work this out back in March: http://lists.freebsd.org/pipermail/freebsd-perl/2010-March/002511.html the thread went un-answered at the time. The solution we found was to build a threaded perl, using the same underlying threading library that everything else is built against. If I recall, John ended up doing a *lot* of custom builds for this. I'm not sure what the right solution is here. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sun, Sep 19, 2010 at 4:48 PM, Stephen Corbesero corbes...@ptd.net wrote: I upgraded my FreeBSD sources to the latest 8.x stable and recompiled the world (and kernel). I then compiled and installed Perl-5.10.1 for my FreeBSD by hand, specifically enabling its threaded option. Finally, I recompiled and reinstalled Amanda 3.1.2. I'm glad it worked! There's some info at http://wiki.zmanda.com/index.php/Installation/OS_Specific_Notes/Installing_Amanda_on_FreeBSD which apparently could use some additional information. Can you fill in what you know there, to help the next guy? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Sun, Sep 19, 2010 at 5:40 PM, Gene Heskett gene.hesk...@gmail.com wrote: Your guess is as good as mine, there simply isn't anything to indicate the failure, or a hint of a reason in the logs I've grepped. All I know at this point is that I have to back up to about svn3341 just to get amflush to work. The current 3419? taps the drive, once for a read that may be the tape header file, and once about 2 seconds later for a very quick write which might be the status file, then exits silently in another second or so even when amflush is run with the -f option. I think it might help if taper was made a lot more chatty. Is there a debug level option I could pass to it? Sure - add 'debug taper 9' to amanda.conf. But the taper doesn't have anything to chat about - the driver never asks it to do anything after starting it up. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpick: Enterprise features
On Thu, Sep 9, 2010 at 9:24 PM, Dustin J. Mitchell dus...@zmanda.com wrote: So I'm invoking my that's complicated escape clause. The only one of these that I could conceivably accomplish in the 3.2 timeframe is #2. I could add 'amadmin $conf hosts' to list all of the hosts, avoiding the need to grep the multiline output of 'amadmin $conf disklist'. I'll put that in my queue right behind Chris H's suggestion (with which I'm almost finished). And I've done this now. Please have a look at: http://github.com/djmitche/amanda/commit/z11906 and let me know if you think this will meet your needs. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Sun, Sep 19, 2010 at 7:00 PM, Gene Heskett gene.hesk...@gmail.com wrote: But that kills amcheck, when amcheck is svn3341. How new do I have to be to have that work? Oops, sorry, it's debug_taper 9 not debug taper Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sun, Sep 19, 2010 at 11:05 PM, John Hein jh...@symmetricom.com wrote: (Currently, I don't think the amanda client executes threaded code, but that may change in the future - or may have already in 3.1/3.2 code - Dustin?). There is now a good bit of Perl running on the client side, but none of it uses threads. We haven't foresworn threads on the client, though, so I would expect that to change eventually. At some point, there were a few folks thinking about branching an older version of Amanda like 2.5.2 to create a maintained minimal client that could be installed quite widely, even if it didn't support fancy features like the Application API. I'm not sure what happened to that particular project, but it could be a way around problems with increasing client-side requirements. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Fri, Sep 17, 2010 at 8:35 PM, Gene Heskett gene.hesk...@gmail.com wrote: At this point I have a WTH expression on my ancient face. Go read changelog if all else fails. I see a lot of changes on about 08/29-30-31 but no mention of chg-disk-slot? And, indeed, that hasn't changed. Now that you've experimented a bit, can you put your finger on exactly what symptoms you're seeing? I assume that you're still using the old chg-disk, or have you changed that somewhere along the line to use chg-disk:/path/to/something? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Another nitpick / feature request
On Sat, Sep 18, 2010 at 9:44 AM, Jon LaBadie j...@jgcomp.com wrote: When a large DLE doesn't tape all the split parts before running out of tapes, could the subsequent flush (amflush or autoflush) pick up where the original taping left off? I just had a large DLE (23 3GB parts) fail on the last part by just 800MB. On the subsequent amdump the DLE autoflushed the entire 23 parts where only one part was needed. Amanda has historically avoided this for, as I understand it, reliability reasons. Once the dump fails during one run, it's in the catalog as a failed run and can't easily be extended into a successful run. That said, the multi-taper support now handles switching dumps from one in-progress tape to another, so maybe the reliability reasons are gone now? At any rate, this would be pretty difficult to implement, but it's a cool idea! Dusitn -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.2.0svn3411
On Sat, Sep 18, 2010 at 11:26 AM, Gene Heskett gene.hesk...@gmail.com wrote: 1. The backup runs normally, to the holding disk, but times out writing to the vtape, so it is all left sitting in the holding disk. I have enough I can do this for several days. Can you send the taper debug log? Taper doesn't time out, so that's pretty odd. Or is the timeout something you've added externally? 2. amcheck looks like its working just fine, but is not touching /config/chg- disk-slot, nor is it updateing the 'data' link in the vtapes slot# directory. This has been since 08/29 according to the dates on those files in the config dir, and the data link was left pointing at slot2, which would correspond to the 08/30 run. Can you send the amcheck output and `find /my/vtapes`? I did know I was supposed to be changing this, manpage please? Stuff like this really needs to be in the changelog, at least with a see manpage so-and-so for details line. Well, to be clear, I didn't say you should - I just asked if you did, and the answer was no :) In general, we're hoping that new users will use the new changers, and existing users can migrate or not at their option. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Another nitpick / feature request
On Sat, Sep 18, 2010 at 2:17 PM, Jon LaBadie j...@jgcomp.com wrote: I'll certainly defer to you knowledge of the difficulty. I was thinking that the logic is already there for the split disk feature. The added code would be for saving its state and recreating it on the next flush. Yes, that would be tricky, as would finding a way to record the continuation of the dump in the catalog (the catalog is log-based, so you can't edit an already-recorded dump). I know, Suitable Patches Are Always Welcomed. :) I've been trying not to say that too much, but yeah :) Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sat, Sep 18, 2010 at 3:35 PM, Stephen Corbesero corbes...@ptd.net wrote: (gdb) info threads * 1 Thread 28469480 (LWP 100148) 0x28af6225 in __error () from /lib/libthr.so.3 Can you run the 'bt' command here to see what that thread is up to? (gdb) bt Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: My 3.1.2 server starts ok, but taper never writes anything
On Sat, Sep 18, 2010 at 4:17 PM, Stephen Corbesero corbes...@ptd.net wrote: Also, if it is worth anything, the state of the process is listed as 'umtxn' from top/ps. (This is a FreeBSD 8.1 system.) Huh, so that sounds like a kernel lock: http://old.nabble.com/what-is-umtxn-td20860047.html Can you identify further the last few lines of perl executed? You should use Amanda::Debug::debug instead, so that your logging will be intermixed with the other information from taper_debug, in the taper debug log. You mentioned that you enabled taper_debug, but never sent along the debug log - can you do that now? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: 3.1.2 : Problems with mails?
On Fri, Sep 17, 2010 at 7:51 AM, Stefan G. Weichinger s...@amanda.org wrote: Attached my poor draft ... it pulls sources in already but fails with automake ... I know too less about that to debug that ... any hint? You should probably base it on the 3.1.2 ebuild - robbat made a number of changes. As for the automake errors - if you look at the autogen script http://github.com/zmanda/amanda/blob/master/autogen You'll see it does a number of things before calling aclocal and the other autotools. You'll need to replicate that in the svn ebuild. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: Nitpicks?
On Fri, Sep 17, 2010 at 2:59 PM, Stefan G. Weichinger s...@amanda.org wrote: And as a bonus: the possibility to get these warnings to another mail-address than the usual reports. There have been a number of notification-related nitpicks so far. While they're good ideas, I don't want to re-implement monitoring and notification systems in Amanda. Are there projects out there that you think would work well with Amanda, if we could build some interfaces, to do this sort of notification? So far, I have: - tape-monkey emails (just the next tape message from amreport) - pager-friendly amcheck reports - trending-based warning notifications If we could make this info available to a monitoring framework (or several..), then these sorts of emails would be much easier to produce. I know there is a basic NAGIOS interface. Are there other ideas? Amreport has an --xml output mode, although it's not tested at all so currently YMMV. But that might be a nice way to start to export status data from Amanda into a tool designed for monitoring and notification. Dustin -- Open Source Storage Engineer http://www.zmanda.com
