RE: Amanda vs Homegrown
Hi Mike, Generally, if you can't justify it yourself I'm wondering why you're using Amanda. But anyway, here's something that may be worth mentioning: business continuity. The backups will be used in the event that something catastrophic has happened. Let's say that the building has burned down and the guy that did the backup scripts is away on business/holiday. With amanda, all you have to do is get some machines up and running and restore from your offsite backup (you are taking a backup offsite, aren't you? - quick note: don't completely trust fire safes) using an easily obtainable backup package. After this kind of critical failure amanda needs a single restore operation (assuming a full backup on one media) and everything ends up hunky-dory as fast as the tape drive/network can work. It can't be up and running any faster than this. With the hand-rolled scripts it could easily take someone a few days to figure out where everything should be going, or for the guy to wrote them to get back/talk someone through it over the phone. The company is going to lose money already from the original disaster, but it would lose even more if people can't carry on working as soon as possible. I'm currently using hand-rolled scripts here on my Windows servers, but that's because I don't want a restore from tape to end up broken by some kind of star/linux/smb/ntfs complication. Once ntbackup has backed up to a single bkf file (including NTFS permissions and the like), Amanda is free to back it up to tape. It adds an extra level of complication, but ntbackup stores the original location in the bkf file which makes it simple for someone else to restore. However, the customer is still the customer and if they decide they aren't happy with amanda, or more importantly that they don't trust it, you might find that you have to go against your gut feeling and do what they want. It would still be worth pointing out what a huge security risk the rcp command is, and if they insist on using their scripts at least get them to remove the r* accounts setup stuff and use something like rsync over an encrypted channel (why bother protecting the file on the disk if you're going to potentially transfer it in plain text over the network). One other thing. If you have space on the backup media and enough time to do a complete backup every night, why would you want to do anything other than a full backup? I hope this helps, Mark Lidstone IT and Network Support Administrator BMT SeaTech Ltd Grove House, Meridians Cross, 7 Ocean Way Ocean Village, Southampton. SO14 3TJ. UK Tel: +44 (0)23 8063 5122 Fax: +44 (0)23 8063 5144 E-Mail: mailto:[EMAIL PROTECTED] Website: www.bmtseatech.co.uk == Confidentiality Notice and Disclaimer: The contents of this e-mail and any attachments are intended only for the use of the e-mail addressee(s) shown. If you are not that person, or one of those persons, you are not allowed to take any action based upon it or to copy it, forward, distribute or disclose the contents of it and you should please delete it from your system. BMT SeaTech Limited does not accept liability for any errors or omissions in the context of this e-mail or its attachments which arise as a result of Internet transmission, nor accept liability for statements which are those of the author and not clearly made on behalf of BMT SeaTech Limited. == -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Sent: 20 April 2005 19:05 To: amanda-users@amanda.org Subject: Amanda vs Homegrown Well, I have spent a few days converting a client from a bunch of hand rolled scripts that rcp files all over the place, to amanda. All the while saying that this will be better, this is good, this is how it should work. Of course I couldn't complete it in a day, and there were issues of configuration. So now that I am mostly complete and am ready to put this project away. The client comes in today and says this is taking entirely too long (to get working), and I want a single piece of media with a full backup of everything and my scripts were working just fine.. To which I say, Uhhh, I mean I was dumbfounded, stopped in my tracks by such a ludicrous statement. So. Does anyone have some good business case stuff or other verbage of a paragraph or three that I can use to convince this person that an actual backup program is better in all ways than hand rolled scripts, and that a full backup on a single piece of media may look attractive, but actually is not???
RE: Amanda vs Homegrown
On Thu, 21 Apr 2005, Mark Lidstone wrote: It would still be worth pointing out what a huge security risk the rcp command is, and if they insist on using their scripts at least get them to remove the r* accounts setup stuff and use something like rsync over an encrypted channel (why bother protecting the file on the disk if you're going to potentially transfer it in plain text over the network). So have you modified amanda to encrypt your network transfers? It doesn't do that out of the box you know. -Mitch
RE: Amanda vs Homegrown
Hi Mitch, Good point - I think I got sidetracked while I was writing that bit. The security risk I originally meant to point out is related to the r-commands accounts setup (password-less remote login as another user). You can still use that to argue your point to the customer though. Thanks, Mark Lidstone IT and Network Support Administrator BMT SeaTech Ltd Grove House, Meridians Cross, 7 Ocean Way Ocean Village, Southampton. SO14 3TJ. UK Tel: +44 (0)23 8063 5122 Fax: +44 (0)23 8063 5144 E-Mail: mailto:[EMAIL PROTECTED] Website: www.bmtseatech.co.uk == Confidentiality Notice and Disclaimer: The contents of this e-mail and any attachments are intended only for the use of the e-mail addressee(s) shown. If you are not that person, or one of those persons, you are not allowed to take any action based upon it or to copy it, forward, distribute or disclose the contents of it and you should please delete it from your system. BMT SeaTech Limited does not accept liability for any errors or omissions in the context of this e-mail or its attachments which arise as a result of Internet transmission, nor accept liability for statements which are those of the author and not clearly made on behalf of BMT SeaTech Limited. == -Original Message- From: Mitch Collinsworth [mailto:[EMAIL PROTECTED] Sent: 21 April 2005 13:04 To: Mark Lidstone Cc: amanda-users@amanda.org Subject: RE: Amanda vs Homegrown On Thu, 21 Apr 2005, Mark Lidstone wrote: It would still be worth pointing out what a huge security risk the rcp command is, and if they insist on using their scripts at least get them to remove the r* accounts setup stuff and use something like rsync over an encrypted channel (why bother protecting the file on the disk if you're going to potentially transfer it in plain text over the network). So have you modified amanda to encrypt your network transfers? It doesn't do that out of the box you know. -Mitch
Re: Amanda vs Homegrown
Hi! Mitch Collinsworth wrote: On Thu, 21 Apr 2005, Mark Lidstone wrote: It would still be worth pointing out what a huge security risk the rcp command is, and if they insist on using their scripts at least get them to remove the r* accounts setup stuff and use something like rsync over an encrypted channel (why bother protecting the file on the disk if you're going to potentially transfer it in plain text over the network). So have you modified amanda to encrypt your network transfers? It doesn't do that out of the box you know. Dont know if there is an Amanda-wish-list somewhere, but using ssh, ssl or something similar for the client-server connection would be a nice goddie. Btw. is there any concept how to add desaster recovery - even if its' platform specific and involves usage of non-amanda tools? Any papers or how-tos concerning this? Bye, Peter WOTLmade
very large dumps in holding area
I am having a problem with amanda leaving very large dumps in the
holding area and not writing them to tape. The tapes are 20/40GB tapes,
but a 23GB dump doesn't get written even though there is a blank tape
available. Even if I try to flush this dump, it runs for a long time and
then errors again with No space left on device.
Here is the log that I got.
The dumps were flushed to tapes Progeny01-0043, Progeny01-0044.
*** A TAPE ERROR OCCURRED: [[writing file: No space left on device]].
Some dumps may have been left in the holding disk.
Run amflush again to flush them to tape.
The next 2 tapes Amanda expects to used are: Progeny01-0020, Progeny01-0021.
FAILURE AND STRANGE DUMP SUMMARY:
morimoto /net/morimoto/home lev 5 FAILED [out of tape]
STATISTICS:
Total Full Daily
Estimate Time (hrs:min)0:00
Run Time (hrs:min) 3:39
Dump Time (hrs:min)0:00 0:00 0:00
Output Size (meg) 0.00.00.0
Original Size (meg) 0.00.00.0
Avg Compressed Size (%) -- -- --
Filesystems Dumped0 0 0
Avg Dump Rate (k/s) -- -- --
Tape Time (hrs:min)0:03 0:00 0:03
Tape Size (meg) 361.20.0 361.2
Tape Used (%) 2.20.02.2 (level:#disks ...)
Filesystems Taped14 0 14 (1:13 3:1)
Avg Tp Write Rate (k/s) 2408.5-- 2408.5
USAGE BY TAPE:
LabelTime Size %Nb
Progeny01-0043 0:03 361.22.214
Progeny01-0044 0:00 0.00.0 0
There are two amanda directories in my holding area:
20050420 is 28G
20050421 is 3.2G
How do I get amanda to readjust what gets backed up when to make it more
even over the full tape cycle? Is it possible with amadmin? When I do an
amadmin balance, I get this:
due-date #fsorig KB out KB balance
--
4/21 Thu3 37603890 26119168 +330.0%
4/24 Sun1 14116820 14116820 +132.4%
4/25 Mon1 12436100 11076062+82.3%
4/26 Tue174643407464340+22.9%
4/30 Sat2 12133630 11246536+85.1%
5/01 Sun3 536890 452308-92.6%
5/02 Mon251054604698652-22.7%
5/03 Tue6 128834408596350+41.5%
5/04 Wed4 104575608194571+34.9%
5/07 Sat3 24676810 23458226 +286.2%
--
TOTAL 26 137414940 115423033 6074896
(estimated 19 runs per dumpcycle)
(3 filesystems overdue, the most being overdue 146 days)
which doesn't seem balanced at all.
Vicki
Re: very large dumps in holding area
On Thu, 21 Apr 2005 at 9:24am, Vicki Stanfield wrote I am having a problem with amanda leaving very large dumps in the holding area and not writing them to tape. The tapes are 20/40GB tapes, but a 23GB dump doesn't get written even though there is a blank tape available. Even if I try to flush this dump, it runs for a long time and then errors again with No space left on device. Are you using software or hardware compression? What does your tapetype look like? The dumps were flushed to tapes Progeny01-0043, Progeny01-0044. *** A TAPE ERROR OCCURRED: [[writing file: No space left on device]]. Some dumps may have been left in the holding disk. Run amflush again to flush them to tape. The next 2 tapes Amanda expects to used are: Progeny01-0020, Progeny01-0021. *snip* You clipped the *really* interesting part -- the line in the NOTES section from taper. What's that line look like? -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: very large dumps in holding area
Joshua Baker-LePain wrote: On Thu, 21 Apr 2005 at 9:24am, Vicki Stanfield wrote I am having a problem with amanda leaving very large dumps in the holding area and not writing them to tape. The tapes are 20/40GB tapes, but a 23GB dump doesn't get written even though there is a blank tape available. Even if I try to flush this dump, it runs for a long time and then errors again with No space left on device. Are you using software or hardware compression? What does your tapetype look like? The dumps were flushed to tapes Progeny01-0043, Progeny01-0044. *** A TAPE ERROR OCCURRED: [[writing file: No space left on device]]. Some dumps may have been left in the holding disk. Run amflush again to flush them to tape. The next 2 tapes Amanda expects to used are: Progeny01-0020, Progeny01-0021. *snip* You clipped the *really* interesting part -- the line in the NOTES section from taper. What's that line look like? Okay. The tapetype is: HP-C5683A. I'm guessing that the holding area isn't big enough (which I had been told before I admit, but since it's on its own partition I haven't done anything about it). I remember hearing that you can combine two holding areas as one; is that possible for me to add space from another drive and combine it with my current holding area? Where can I find instructions on how to do that? NOTES: driver: WARNING: /var/spool/amanda: 36505600 KB requested, but only 8085788 KB available. planner: Incremental of pollux:/net/pollux/project/new-ps-archive/60 bumped to level 3. planner: Incremental of morimoto:/net/morimoto/home bumped to level 6. planner: Incremental of morimoto:/var bumped to level 2. planner: morimoto /net/morimoto/home 20050421 0 [dump larger than tape, 39666051 KB, full dump delayed] Thanks, Vicki
Re: very large dumps in holding area
Jon LaBadie wrote: On Thu, Apr 21, 2005 at 10:49:00AM -0400, Vicki Stanfield wrote: If the dump is 28GB, should it not fit on a 20GB tape if it is compressed. Aren't the tapes basically 20/40GB? Not in my mind. They are 20GB tapes. I.e. they can hold 20GB of ones and zero bits. What those ones and zeros represent could be 40GB of data compressed ton 20GB of zeros and ones. I.e. 50% compression. But the level of compression varies greatly with the type of data being compressed and the algorithm being used. In fact, the compressor used by your tape drive can actually expand some types of data rather than compressing it. Using gzip as the software compressor I've tested data and gotten 0, 3, 10, 25, 50, 75, even 90+ percent compression. If the data in your dump compresses 50%, fine, it will fit on a 20GB tape. But if it only compresses 3%, I doubt it. And if it was already compressed by gzip (or has lots of gzipped files), it WILL EXPAND if the tapedrive compressor works on it. Okay, I see that I asked my initial question badly. I know that you very seldom if ever get 2-1 compression. I just meant that I thought that 1.4 to 1 wasn't too much to ask. Perhaps that is incorrect. Anyway, I am perfectly willing to stop using hardware compression altogether but haven't figured out how to make the setting permanent yet using GNU-mt 2.4.2. Vicki
Re: very large dumps in holding area
--On Thursday, April 21, 2005 12:27:20 -0400 Vicki Stanfield [EMAIL PROTECTED] wrote: Jon LaBadie wrote: On Thu, Apr 21, 2005 at 10:49:00AM -0400, Vicki Stanfield wrote: If the dump is 28GB, should it not fit on a 20GB tape if it is compressed. Aren't the tapes basically 20/40GB? Not in my mind. They are 20GB tapes. I.e. they can hold 20GB of ones and zero bits. What those ones and zeros represent could be 40GB of data compressed ton 20GB of zeros and ones. I.e. 50% compression. But the level of compression varies greatly with the type of data being compressed and the algorithm being used. In fact, the compressor used by your tape drive can actually expand some types of data rather than compressing it. Using gzip as the software compressor I've tested data and gotten 0, 3, 10, 25, 50, 75, even 90+ percent compression. If the data in your dump compresses 50%, fine, it will fit on a 20GB tape. But if it only compresses 3%, I doubt it. And if it was already compressed by gzip (or has lots of gzipped files), it WILL EXPAND if the tapedrive compressor works on it. Okay, I see that I asked my initial question badly. I know that you very seldom if ever get 2-1 compression. I just meant that I thought that 1.4 to 1 wasn't too much to ask. Perhaps that is incorrect. Depends on your data. You may want to define both compressing and non-compressing dumptypes and compress DLE's that can benefit from it and not ones that don't (assuming you're using software compression). Anyway, I am perfectly willing to stop using hardware compression altogether but haven't figured out how to make the setting permanent yet using GNU-mt 2.4.2. On some drives you may have to set jumpers or dip switches on the drive itself to disable it. Even then, once a tape has been written in hw compression mode it will sense that from a hidden header and keep using compression. Gene usually jumps in here with his script to fix the hidden header, but I'm sure you can find it in the list archives many times. Frank Vicki -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Amanda vs Homegrown
On Wed, Apr 20, 2005 at 11:04:43AM -0700, Mike wrote: Well, I have spent a few days converting a client from a bunch of hand rolled scripts that rcp files all over the place, to amanda. All the while saying that this will be better, this is good, this is how it should work. Of course I couldn't complete it in a day, and there were issues of configuration. So now that I am mostly complete and am ready to put this project away. The client comes in today and says this is taking entirely too long (to get working), and I want a single piece of media with a full backup of everything and my scripts were working just fine.. To which I say, Uhhh, I mean I was dumbfounded, stopped in my tracks by such a ludicrous statement. So. Does anyone have some good business case stuff or other verbage of a paragraph or three that I can use to convince this person that an actual backup program is better in all ways than hand rolled scripts, and that a full backup on a single piece of media may look attractive, but actually is not??? What I feel would be best is to identify the deficiencies of the current system (perceived or real) and how amanda would address them. Then you can add the icing of what other benefits amanda can provide. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Exclude list and disklist
On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. The second question is about disklist. If I have to change the directory layout of a system. And change the disklist, I can see that in the next backup, the new list is used. But, all the old list are still in the index (I can see it in amrecover). I think it's good if it keep the old structure (disklist) until all the tapes that contain that data are reused. Is this the case, if not, how do I delete the entries? The same If you change the disklist, amanda treats those as new entries. The old entries won't be backed up anymore, but will still be recoverable until the tapes are overwritten. as exclude list, if some users update me that some directories in their home are not supposed to be backup, but it was backup before. Does the next amdump run redo the dump not to include that directory or only only when it detects more changes in that directory? Would it exclude the list, or I should restart some of the amanda processes? Amanda leaves no processes running between dumps. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: Exclude list and disklist
Joshua Baker-LePain wrote: On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. Got it. Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? Thanks, Kuas.
Re: Exclude list and disklist
On Thu, 21 Apr 2005 at 3:58pm, Kuas wrote Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? From 'man amanda': exclude [ list|file ][[optional][ append ][ string ]+] Default: file. There is two exclude list exclude file and exclude list. With exclude file , the string is a gnutar exclude expression. With exclude list , the string is a file name on the client containing gnutar exclude expression. All exclude expression are concatenated in one file and passed to gnutar as a --exclude-from argument. With the append keyword, the string are appended to the current value of the list, without it, the string overwrite the list. =If optional is specified for exclude list, then amcheck will not =complain if the file doesn't exist or is not readable. For exclude list, If the file name is relative, the disk name being backed up is prepended. So if this is entered: exclude list .amanda.excludes the actual file use would be /var/.amanda.excludes for a backup of /var, /usr/local/.amanda.excludes for a backup of /usr/local, and so on. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: Exclude list and disklist
On Thu, Apr 21, 2005 at 03:58:58PM -0400, Kuas enlightened us: Joshua Baker-LePain wrote: On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. Got it. Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? I use exclude list optional .amanda.exclude So if the file doesn't exist, it isn't an error. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 pgpL5TSKYGyuZ.pgp Description: PGP signature
