Re: [Amanda-users] amanda.conf config help
Upendra,
I don't recall how large your holding area is or if you end up
with multiple DLE in holding waiting for tape or not.
I've had good success with taperalgo and selecting the largestfit
option. This seems to fill my (non-spanning physical) tapes pretty
close to 100%.
If you have lots of work area and can delay your dumps until there
are mulipul DLE in the work area so the taper algorithms have more
choices there are paramters for that as well.
I believe (unless they have been removed in the last itteration)
that these relatively new parameters will help you get where you
want to go.
Note this is one of several examples in the amanda.conf file.
There are several examples to look at.
# You want to fill tapes completely even in the case of failed dumps, and
# don't care if some dumps are left on the holding disk after a run:
# flush-threshold-dumped100 # (or more)
# flush-threshold-scheduled 100 # (or more)
# taperflush100
# autoflush yes
On Wed, Dec 15, 2010 at 01:34:31PM -0500, upengan78 wrote:
Hello again,
I think I am starting to realize that something needs to be configured
properly in order to Fill the tapes to their MAX capacity. I'd like some
advise on that. I can create another thread for that however, it may be
something to do with existing configuration so I am continuing the discussion
here.
I am doing below cron jobs currently (no manual amdumps)
#Amcheck
59 19 * * 1-5 su - amanda -c /opt/csw/sbin/amcheck -a weeklyfull
5 20 * * * su - amanda -c /opt/csw/sbin/amcheck -a monthlyfull
#amdump
01 00 * * 1-5 su - amanda -c /opt/csw/sbin/amdump weeklyfull
15 01 * * * su - amanda -c /opt/csw/sbin/amdump monthlyfull
weeklyfull- amanda.conf
inparallel 4
maxdumps 4
dumpcycle 7
runspercycle 5
tapecycle 9
runtapes 3
dumpuser amanda
tpchanger chg-disk# a virtual tape changer
tapedev file:/bk/location/amanda/vtapes/weeklyfull/slots
changerfile /opt/csw/etc/amanda/weeklyfull/changerfile
labelstr WF-.*
#label_new_tapes PLUTO-%%
autolabel WF-%%
tapetype DVD_SIZED_DISK
logdir /opt/csw/etc/amanda/weeklyfull
infofile /opt/csw/etc/amanda/weeklyfull/curinfo
indexdir /opt/csw/etc/amanda/weeklyfull/index
tapelist /opt/csw/etc/amanda/weeklyfull/tapelist
#etimeout 600 # number of seconds per filesystem for estimates.
etimeout 3600 # number of seconds per filesystem for estimates.
#etimeout -600 # total number of seconds for estimates.
# a positive number will be multiplied by the number of filesystems on
# each host; a negative number will be taken as an absolute total time-out.
# The default is 5 minutes per filesystem.
#dtimeout 1800# number of idle seconds before a dump is aborted.
dtimeout 3600# number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
# for each client host
holdingdisk hd1 {
directory /random/amandahold/hold
}
holdingdisk hd2 {
directory /random1/amanda/holdingdisk2
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
# tape_splitsize 1 Gb
tape_splitsize 1024 mbytes
# fallback_splitsize 512 MB
fallback_splitsize 4096 MB
split_diskbuffer /random/buffer
}
define tapetype DVD_SIZED_DISK {
filemark 1 KB
length 10240 MB
}
inparallel 5
maxdumps 5
dumpcycle 30 days
runspercycle 30
tapecycle 20
runtapes 9
dumpuser amanda
tpchanger chg-disk# a virtual tape changer
tapedev file:/bk/location/amanda/vtapes/monthlyfull/slots
changerfile /opt/csw/etc/amanda/monthlyfull/changerfile
labelstr MF-.*
#label_new_tapes PLUTO-%%
autolabel MF-%%
tapetype DVD_SIZED_DISK
logdir /opt/csw/etc/amanda/monthlyfull
infofile /opt/csw/etc/amanda/monthlyfull/curinfo
indexdir /opt/csw/etc/amanda/monthlyfull/index
tapelist /opt/csw/etc/amanda/monthlyfull/tapelist
#etimeout 600 # number of seconds per filesystem for estimates.
etimeout 3600 # number of seconds per filesystem for estimates.
#etimeout -600 # total number of seconds for estimates.
# a positive number will be multiplied by the number of filesystems on
# each host; a negative number will be taken as an absolute total time-out.
# The default is 5 minutes per filesystem.
#dtimeout 1800# number of idle seconds before a dump is aborted.
dtimeout 3600# number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
# for each client host
holdingdisk hd1 {
directory /random/amandahold/hold
}
holdingdisk hd2 {
directory /random1/amanda/holdingdisk2
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
# tape_splitsize 1
Re: [Amanda-users] amanda.conf config help
On Fri, Dec 03, 2010 at 09:55:14AM -0500, upengan78 wrote:
Ok guys, I got busier with other tasks so could not reply. sorry about that.
I did manage upgrading to Amanda 3.x version on solaris amanda server and
solaris amanda client.
I tried amcheck and it worked. So I tried amdump. However it seems to me I
really need to adjust the numbers because I got below email when I ran the
dump. May be using multiple DLEs is my only option rather than one here.
Congratulations on a successful amdump!
Here is the Amanda email I got. I don't understand what it is trying to tell
me to do.
These dumps were to tapes TEST-1, TEST-2.
The next 6 tapes Amanda expects to use are: 6 new tapes.
The next 6 new tapes already labelled are: TEST-3, TEST-4, TEST-5, TEST-6,
TEST-7, TEST-8
OK, you allow amanda to use up to six tapes, but it only needed two.
STRANGE DUMP SUMMARY:
TEST.domain.com /bk/location lev 0 STRANGE (see below)
Lots of strange messages are quite normal, particularly when
dumping an active/live file system.
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min) 0:29
Run Time (hrs:min) 2:07
Dump Time (hrs:min) 1:38 1:38 0:00
Output Size (meg)16131.316131.30.0
Original Size (meg) 31043.731043.70.0
Avg Compressed Size (%) 52.0 52.0--
fairly compressible data.
Filesystems Dumped 1 1 0
Avg Dump Rate (k/s) 2795.5 2795.5--
Tape Time (hrs:min) 1:38 1:38 0:00
Tape Size (meg) 16131.316131.30.0
Tape Used (%) 157.5 157.50.0
Filesystems Taped 1 1 0
Parts Taped 33 33 0
Hmm, a 16.1GB dump written in taped in 33 parts. Sounds
like your splits are about 0.5GB. I wouldn't leave it that
low unless you are really concerned with maximizing each
vtape's filling.
Avg Tp Write Rate (k/s) 2795.9 2795.9--
USAGE BY TAPE:
Label Time Size %NbNc
TEST-1 1:1210485088k 100.0 120
TEST-2 0:27 6556954k 62.5 013
Hmm, 20 parts successfully taped on the first (TEST-1) vtape.
So you must be sizing them at about 10GB.
STRANGE DUMP DETAILS:
/-- TEST.domain.com /bk/location lev 0 STRANGE
sendbackup: start [TEST.domain.com:/bk/location level 0]
sendbackup: info BACKUP=/opt/csw/bin/gtar
sendbackup: info RECOVER_CMD=/opt/csw/bin/gzip -dc |/opt/csw/bin/gtar -xpGf
- ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/filenames.log: File
removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.out: File
removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.out.BAK:
File removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.sim: File
removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.sp: File
removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.sp.cache:
File removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/no_name.st0: File
removed before we read it
? /opt/csw/bin/gtar: ./fall2010/ece429/albert/cpu32/sue/tclIndex: File
removed before we read it
| Total bytes written: 32551710720 (31GiB, 5.3MiB/s)
sendbackup: size 31788780
sendbackup: end
\
These are strange, not failed. I.e., tar noted something but
continued to run. Gtar makes a list of files it will backup up
then copies them. If during the interval between noting a file
to back up and the actual backup, some{one|thing} removes the
file, the above strange messages are produced. I'm not
concerned about missing backups of things that were just removed,
they were probably some type of temporary file.
Strange messages are also generaged about files that change while
being backed up (eg. log files) and the backup of these may be
inaccurate. Gtar does not backup network sockets nor (I think)
device files. But it generates unexpected, thus strange mesgs.
NOTES:
planner: Incremental of TEST.domain.com:/bk/location bumped to level 2.
planner: TEST.domain.com /bk/location 20101202160419 0
/opt/csw/libexec/amanda/runtar exited with status 1: see
/tmp/amanda/client/test/sendsize.20101202160241.debug
taper: Will request retry of failed split part.
taper: tape TEST-1 kb 9961472 fm 20 [OK]
While attempting to put part 21 on the vtape, the end of the
tape was reached. Thus an error. But from the line below,
it successfully was sent to the next vtape.
taper: tape TEST-2 kb 6556954 fm 13 [OK]
big estimate:
Re: [Amanda-users] amanda.conf config help
dispite the multiple DLEs the wall clock time is not shorter
than the dumptime, check use of inparallel and maxdump and
you should be able to get some concurrency, not that running
into the work day is an issue with this config, but its good
to practice and now about it later.
On Fri, Dec 03, 2010 at 04:04:16PM -0500, upengan78 wrote:
#Congratulations on a successful amdump!
Wow, tell you what I am so happy to see those words! Thanks much!
By the time I saw your reply, I had impatiently split my single DLE into 9
DLEs which also includes the last one for excluding all 8 DLEs on the top.
I did see same message again in the amdump summary :
taper: Will request retry of failed split part.
taper: tape TEST-1 kb 9845830 fm 15 [OK]
taper: tape TEST-2 kb 6661060 fm 7 [OK]
But now that I read your reply, I think it just doesn't mean a lot as long as
I see OK below that particular message. Here is full DUMP summary :
These dumps were to tapes TEST-1, TEST-2.
The next 6 tapes Amanda expects to use are: 6 new tapes.
The next 6 new tapes already labelled are: TEST-3, TEST-4, TEST-5, TEST-6,
TEST-7, TEST-8
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min) 0:10
Run Time (hrs:min) 1:51
Dump Time (hrs:min) 1:38 1:38 0:00
Output Size (meg)16120.016120.00.0
Original Size (meg) 30661.630661.60.0
Avg Compressed Size (%) 52.6 52.6--
Filesystems Dumped 9 9 0
Avg Dump Rate (k/s) 2809.3 2809.3--
Tape Time (hrs:min) 0:17 0:17 0:00
Tape Size (meg) 16120.016120.00.0
Tape Used (%) 157.4 157.40.0
Filesystems Taped 9 9 0
Parts Taped 22 22 0
Avg Tp Write Rate (k/s) 16506.916506.9--
USAGE BY TAPE:
Label Time Size %NbNc
TEST-1 0:1010485219k 100.0 815
TEST-2 0:07 6661060k 63.5 2 7
NOTES:
planner: Adding new disk TEST.domain.com:/bk/location/./apple2007.
planner: Adding new disk TEST.domain.com:/bk/location/./apple2008.
planner: Adding new disk TEST.domain.com:/bk/location/./apple2009.
planner: Adding new disk TEST.domain.com:/bk/location/./apple2010.
planner: Adding new disk TEST.domain.com:/bk/location/./pear2008.
planner: Adding new disk TEST.domain.com:/bk/location/./pear2009.
planner: Adding new disk TEST.domain.com:/bk/location/./pear2010.
planner: Adding new disk TEST.domain.com:/bk/location/./other.
planner: Adding new disk TEST.domain.com:/bk/location/./_rest_.
taper: Will request retry of failed split part.
taper: tape TEST-1 kb 9845830 fm 15 [OK]
taper: tape TEST-2 kb 6661060 fm 7 [OK]
big estimate: TEST.domain.com /bk/location/./apple2007 0
est: 244128kout 119306k
big estimate: TEST.domain.com /bk/location/./apple2009 0
est: 834272kout 521751k
big estimate: TEST.domain.com /bk/location/./apple2010 0
est: 3646880kout 2857940k
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISKL ORIG-kB OUT-kB COMP% MMM:SS KB/s MMM:SS
KB/s
-- -
--
TEST.domain.co -u/./_rest_ 0 2981200 2155287 72.37:25 4838.8 2:05
17242.3
TEST.domain.co -./apple2007 0 488140 119306 24.41:40 1188.0 0:06
19884.3
TEST.domain.co -./apple2008 0 6484400 3803120 58.7 28:08 2253.5 4:23
14460.5
TEST.domain.co -./apple2009 0 1668440 521751 31.33:17 2642.9 0:29
17991.4
TEST.domain.co -./apple2010 0 7308970 2857940 39.1 22:30 2117.1 2:46
17216.5
TEST.domain.co -tu/./other 0 5833440 3828839 65.6 18:01 3542.5 3:44
17093.0
TEST.domain.co -pear2008 0 88010 46283 52.60:23 2040.3 0:02
23141.5
TEST.domain.co -pear2009 0 4127930 1932585 46.89:16 3475.2 1:53
17102.5
TEST.domain.co -pear2010 0 2416990 1241776 51.47:15 2853.0 1:12
17246.9
(brought to you by Amanda version 3.1.1)
Now I am thinking about, removing runtapes option completely, and add
splitdisk_buffer option to the holding disk, I am confused about one thing,
why you mentioned that looks like your splits are about 0.5GB. I have
actually set tape_splitsize = 1Gb although I do have fallback_splitsize =
512MB . Do you mean I should grow this value to a higher value or you said
that about tape_splitsize value?
Other than this, your explanation was really informative and thanks again for
that.
+--
|This was sent by upendra.gan...@gmail.com via
Re: [Amanda-users] amanda.conf config help
Hi! amcheck test /opt/csw/etc/amanda/test/amanda.conf, line 43: dumptype parameter expected /opt/csw/etc/amanda/test/amanda.conf, line 43: end of line is expected amcheck: errors processing config file I think you mean the split_diskbuffer parameter: See example amanda.conf: # split_diskbuffer - (optional) When dumping a split dump in PORT-WRITE # mode (usually meaning no holding disk), buffer the split # chunks to a file in the directory specified by this option. # Default: [none] Cheers, Thomas
Re: [Amanda-users] amanda.conf config help
On Fri, Dec 03, 2010 at 04:29:59PM -0500, upengan78 wrote:
Oh, it looks like when I add splitdisk_buffer, it shows me below error on
amcheck,
amcheck test
/opt/csw/etc/amanda/test/amanda.conf, line 43: dumptype parameter expected
/opt/csw/etc/amanda/test/amanda.conf, line 43: end of line is expected
amcheck: errors processing config file
spelling error split_diskbuffer != splitdisk_buffer
Relevant portion from amanda.conf
holdingdisk hd1 {
directory /random/amandahold/test
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
tape_splitsize 1 Gb
#fallback_splitsize 512 MB
fallback_splitsize 2048 MB
splitdisk_buffer /random/amandahold/test
}
define tapetype DVD_SIZED_DISK {
filemark 1 KB
length 10240 MB
}
I should have mentioned, Amanda server has 16GB of memory, you still
recommend splitdisk_buffer?
+--
|This was sent by upendra.gan...@gmail.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
End of included message
--
Jon H. LaBadie j...@jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)
Re: [Amanda-users] amanda.conf config help
On Fri, Dec 03, 2010 at 04:04:16PM -0500, upengan78 wrote: Now I am thinking about, removing runtapes option completely, and add splitdisk_buffer option to the holding disk, I am confused about one thing, why you mentioned that looks like your splits are about 0.5GB. I have actually set tape_splitsize = 1Gb although I do have fallback_splitsize = 512MB . Do you mean I should grow this value to a higher value or you said that about tape_splitsize value? At the time I wrote that I saw you had backed up and taped 16GB of data and they were taped in 33 parts. 16GB / 33 is about 0.5GB/part. Your amdump run used the fallback_splitsize because you had not specified a split_diskbuffer. If one had been specified, data would move from holding disk to the split_diskbuffer creating parts of tape_splitsize. If a part failed, such as at the end of a tape, then the part would be retried on the next tape (if available). Without a split_diskbuffer the parts have to be created in memory. In this situation the parts are sized to the fallback_splitsize rather than tape_splitsize. Typically this is smaller than tape_splitsize to avoid using too much memory. As you had no split_diskbuffer, the fallback_splitsize (0.5GB) was used as the part size. -- Jon H. LaBadie j...@jgcomp.com JG Computing 12027 Creekbend Drive (703) 787-0884 Reston, VA 20194 (703) 787-0922 (fax)
Re: [Amanda-users] amanda.conf config help
On Mon, Nov 29, 2010 at 10:08:04AM -0500, upengan78 wrote: @ Jon, It doesn't show me 3.1.1 version available in CSW which I am using here. See below, /opt/csw/bin/pkgutil -c amanda Checking catalog integrity with gpg. gpg: Signature made Sun Oct 17 21:35:22 2010 CDT using DSA key ID A1999E90 gpg: Good signature from Blastwave Software (Blastwave.org Inc.) softw...@blastwave.org package installed catalog CSWamanda 2.5.2p1,REV=2008.05.21SAME Is openCSW for opensolaris or something, ofcourse I will google it now :) I don't know the details, but there was a disagreement among the owner of the blastwave name and some (all?) of the developers who maintained the packages. This eventually caused a fork of the catalog with the new one being called openCSW. It seemed to me that openCSW is now more complete and better maintained than blastwave. I switched earlier this year. I think packages from the two sites are supposed to be ?interchangeable? (i.e. you should be able to pick and choose from either site). I've not encountered any problems caused by my switching sites, but if I were switching a production system I'd make sure all pkgs were from the same site. jl -- Jon H. LaBadie j...@jgcomp.com JG Computing 12027 Creekbend Drive (703) 787-0884 Reston, VA 20194 (703) 787-0922 (fax)
Re: [Amanda-users] amanda.conf config help
Nothing strickes me as wrong with your new config but...
I'm sorry, I can't actually comment on spanning, never
configured it on any of my amanda servers.
Just for the sake of argument I'm going to include an
extract of my disklist file.
Originally we backed up a 1Tbyte drive, this was impractical,
as much for wall clock time as for any other reason.
Because we are no longer backing up the 'partition' we can't
use dump, the DLE specifies tar. I broke the partition into
two main pieces, with planning and a little luck the directories
that being with letters [A-Q] will contain about the same amount
of data as the directories beginning R to Z.
Because _users_ have access on this file system to create new
directories at will I also included a DLE that would pick up
any directories or files beginning with lower-case letters.
These I allow to backup all at once since there currently
aren't any. ie, avoid missing them by oversight.
trel /Users comp-user-tar
#trel /treluser-tar
trel /trelAQ /trel {
comp-server-user-tar
include ./[A-Q]*
}
trel /trelRZ /trel {
comp-server-user-tar
include ./[R-Z]*
}
trel /trelaz /trel {
user-tar
include ./[a-z]*
}
On Wed, Nov 24, 2010 at 11:37:31AM -0500, upengan78 wrote:
I have made some changed to amanda.conf. First I have changed disk size to
10GB, then number of Vtapes/slots = 29 and also added runtapes = 6
My amanda.conf
dumpcycle 7
runspercycle 7
tapecycle 29
runtapes 6
dumpuser amanda
tpchanger chg-disk# a virtual tape changer
tapedev file:/random/amanda/vtapes/test/slots
changerfile /opt/csw/etc/amanda/test/changerfile
labelstr TEST-.*
#label_new_tapes TEST-%%
#autolabel TEST-%%
tapetype DVD_SIZED_DISK
logdir /opt/csw/etc/amanda/test
infofile /opt/csw/etc/amanda/test/curinfo
indexdir /opt/csw/etc/amanda/test/index
tapelist /opt/csw/etc/amanda/test/tapelist
#etimeout 600 # number of seconds per filesystem for estimates.
etimeout 3600 # number of seconds per filesystem for estimates.
#etimeout -600 # total number of seconds for estimates.
# a positive number will be multiplied by the number of filesystems on
# each host; a negative number will be taken as an absolute total time-out.
# The default is 5 minutes per filesystem.
#dtimeout 1800# number of idle seconds before a dump is aborted.
dtimeout 3600# number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
# for each client host
holdingdisk hd1 {
directory /another drive/amanda/amandahold/test
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
}
define tapetype DVD_SIZED_DISK {
filemark 1 KB
length 10240 MB
}
I am still confused about spanning because
http://wiki.zmanda.com/index.php/How_To:Split_Dumps_Across_Tapes mentions
that spanning is automatically enabled in Vtapes. Now is that for specific
version that means or all versions and is there any command to verify if it
is enabled or not? Copying the relevent portion from wiki below.
Disk Backups (Vtapes)
For vtapes, spanning is automatically enabled, as the VFS device supports
LEOM. You can add a part_size if you'd like to split dumps into smaller
parts; otherwise, Amanda will just fill each vtape with a single part before
moving on to the next vtape.
Thanks for continuing to help guys. Appreciate it and it is wonderful to see
how much deep knowledge people have about backups and sharing it happily.
And, I do understand there has to be offsite storage as I know pipes can
burst and it has happened before. These things I plan to consider once I am
confortable with setup. :)
Can anyone advise on spanning ? how to do it in Vtapes?
Thanks
+--
|This was sent by upendra.gan...@gmail.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
---
Brian R Cuttler brian.cutt...@wadsworth.org
Computer Systems Support(v) 518 486-1697
Wadsworth Center(f) 518 473-6384
NYS Department of HealthHelp Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
Re: [Amanda-users] amanda.conf config help
On Wed, Nov 24, 2010 at 02:22:52PM -0500, upengan78 wrote: Hello Brian, Thanks again. I just ran amdump test and looks like there is some issue which I am not able to understand why it's happening. Here is ther email from Amanda Admin. These dumps were to tape TEST-1. The next 6 tapes Amanda expects to use are: 6 new tapes. The next 6 new tapes already labelled are: TEST-2, TEST-3, TEST-4, TEST-5, TEST-6, TEST-7. FAILURE AND STRANGE DUMP SUMMARY: TEST.domain.com bk/location lev 0 FAILED [dump larger than available tape space, 30927955 KB, incremental dump also larger than tape] planner: FATAL cannot fit anything on tape, bailing out When you increased the size of your vtapes did you also alter the length of the tapes in your tapedev definition ? amanda seems to think that vtape_length * 6 estimated_dump_size IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation.
Re: [Amanda-users] amanda.conf config help
On Wed, Nov 24, 2010 at 11:37:31AM -0500, upengan78 wrote:
I have made some changed to amanda.conf. First I have changed disk size to
10GB, then number of Vtapes/slots = 29 and also added runtapes = 6
My amanda.conf
dumpcycle 7
runspercycle 7
tapecycle 29
runtapes 6
dumpuser amanda
tpchanger chg-disk# a virtual tape changer
tapedev file:/random/amanda/vtapes/test/slots
changerfile /opt/csw/etc/amanda/test/changerfile
labelstr TEST-.*
#label_new_tapes TEST-%%
#autolabel TEST-%%
tapetype DVD_SIZED_DISK
logdir /opt/csw/etc/amanda/test
infofile /opt/csw/etc/amanda/test/curinfo
indexdir /opt/csw/etc/amanda/test/index
tapelist /opt/csw/etc/amanda/test/tapelist
#etimeout 600 # number of seconds per filesystem for estimates.
etimeout 3600 # number of seconds per filesystem for estimates.
#etimeout -600 # total number of seconds for estimates.
# a positive number will be multiplied by the number of filesystems on
# each host; a negative number will be taken as an absolute total time-out.
# The default is 5 minutes per filesystem.
#dtimeout 1800# number of idle seconds before a dump is aborted.
dtimeout 3600# number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
# for each client host
holdingdisk hd1 {
directory /another drive/amanda/amandahold/test
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
}
define tapetype DVD_SIZED_DISK {
filemark 1 KB
length 10240 MB
}
I am still confused about spanning because
http://wiki.zmanda.com/index.php/How_To:Split_Dumps_Across_Tapes mentions
that spanning is automatically enabled in Vtapes. Now is that for specific
version that means or all versions and is there any command to verify if it
is enabled or not? Copying the relevent portion from wiki below.
Disk Backups (Vtapes)
For vtapes, spanning is automatically enabled, as the VFS device supports
LEOM. You can add a part_size if you'd like to split dumps into smaller
parts; otherwise, Amanda will just fill each vtape with a single part before
moving on to the next vtape.
From the same document:
In Amanda earlier than Amanda-3.2, this was done with some dumptype
parameters.
For these versions, see How To:Split Dumps Across Tapes (Amanda-3.1 and
earlier).
You said you are using amanda version 2.5.x. Clearly that is earlier than 3.2.
--
Jon H. LaBadie j...@jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)
Re: [Amanda-users] amanda.conf config help
On Tue, Nov 23, 2010 at 10:27:15AM -0500, upengan78 wrote: Hi, Using Amanda on Solaris 10 2.5.2p1,REV=2008.05.21 and amanda client on a solaris 8 which I beleive has same version of Amanda. OpenCSW shows amanda version 3.1.1. Why are you using such an old version? Jon -- Jon H. LaBadie j...@jgcomp.com JG Computing 12027 Creekbend Drive (703) 787-0884 Reston, VA 20194 (703) 787-0922 (fax)
Re: [Amanda-users] amanda.conf config help
Upengan,
You have 50 gig under backup/amanda control, but how much data
are you currently backing up per week ?
With current settings, tapecycle and dumpcycle you will have
between 2 and 3 full dumps per partition on tape at any one
time, 2.x X 50 Gig 80 Gig so you are already short on (vtape) space.
I believe that vtape size is an upper bound, unused space on
the vtape is actually free space on the disk, so you can actually
oversubscribe the 300 gig pool by some percentage (depending on
numerous factors that we currently do NOT have a handle on).
With 4 Gig tapes... how large are your DLEs ? 4 Gig is relatively
small, are you using tape spanning or are all of the DLE's small ?
With 4-gig tapes and 50 gig under backup you must use 12 tapes
for a complete dump of the system, with a pool of 20 tapes you
can not have two complete dump cycles...
It seems that either I'm badly misunderstanding your config or
you are badly underconfiguring your tape pool.
There is probably more stuff we could address in your email
but I want to make sure I've understood this much before digging
elsewhere.
Brian
On Tue, Nov 23, 2010 at 10:27:15AM -0500, upengan78 wrote:
Hi,
Using Amanda on Solaris 10 2.5.2p1,REV=2008.05.21 and amanda client on a
solaris 8 which I beleive has same version of Amanda.
I am not sure what settings for tapecycle dumpcycle should be used for
efficient usage of Vtapes. I'd like advise in setting those parameters in
amanda.conf
Here is the situation,
Data to be backed up is around 50G and grows by about 100-200MB each day.
I have 20 Vtapes and each tape is 4G size on the server. I do have total
300G available on server disk but I just decided to start with 80G to start
with.
Current settings in Amanda.conf are,
dumpcycle 7
runspercycle 7
tapecycle 20
dumpuser amanda
tpchanger chg-disk# a virtual tape changer
tapedev file:/export/amanda/vtapes/test/slots
changerfile /opt/csw/etc/amanda/test/changerfile
labelstr TEST-.*
#label_new_tapes TEST-%%
#autolabel TEST-%%
tapetype DVD_SIZED_DISK
logdir /opt/csw/etc/amanda/test
infofile /opt/csw/etc/amanda/test/curinfo
indexdir /opt/csw/etc/amanda/test/index
tapelist /opt/csw/etc/amanda/test/tapelist
#etimeout 600 # number of seconds per filesystem for estimates.
etimeout 3600 # number of seconds per filesystem for estimates.
#etimeout -600 # total number of seconds for estimates.
# a positive number will be multiplied by the number of filesystems on
# each host; a negative number will be taken as an absolute total time-out.
# The default is 5 minutes per filesystem.
#dtimeout 1800# number of idle seconds before a dump is aborted.
dtimeout 3600# number of idle seconds before a dump is aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
# for each client host
holdingdisk hd1 {
directory /random/amanda/amandahold/test
}
define dumptype comp-tar {
program GNUTAR
compress fast
index yes
record yes # Important! avoid interfering with production runs
}
define tapetype DVD_SIZED_DISK {
filemark 4 KB
length 4482 MB
}
I'd like to retain backup for a month if that's possible by doing some
modifications in amanda.conf and adding more tapes of modifying tape size or
so..
Please advise..
Thanks
+--
|This was sent by upendra.gan...@gmail.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
---
Brian R Cuttler brian.cutt...@wadsworth.org
Computer Systems Support(v) 518 486-1697
Wadsworth Center(f) 518 473-6384
NYS Department of HealthHelp Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
Re: [Amanda-users] amanda.conf config help
On Tue, 23 Nov 2010 12:35:10 -0500 upengan78 amanda-fo...@backupcentral.com wrote: Firstly, thanks for the response! 1. You have 50 gig under backup/amanda control, but how much data are you currently backing up per week ? Currently not doing any backups through Amanda. I am trying to understand this whole theory of numbers involved in configuration. I want to start backing up a 50G partition with a Full backup on Monday and incrementals on each day till next week's Monday. So I think for 7 days I will have to backup 50G + 200M x 6 days = 50.6G . Let's say 55G a week. If you have a dump cycle of 7, and expect 50 G per, then you need 50 x 3 G, or 150 G, plus some spare. My original configuration is with little idea. dumpcycle 7 runspercycle 7 tapecycle 20 I though above would yield me 7 Days of backup including 1 Full and 6 incrementals in 1 dumpcycle and all 20 tapes will be used. I don't know if I am right. You need 3 x dump cycle tapes plus some spares, so I'd bump that up to 25, a nice integer factor of 150. The reason you need three times dump cycle is that Amanda spreads out level 0 (full) backups across the dump cycle, and this ensures that you always have at least the most recent level 0 backup for any DLE. 2. With current settings, tapecycle and dumpcycle you will have between 2 and 3 full dumps per partition on tape at any one time, 2.x X 50 Gig 80 Gig so you are already short on (vtape) space. Correct I think I also think I am short of Vtapes. I wonder if I should add more 4.xx G size tapes or redo Vtapes of 40G size each to utilize whole 300GB of disk space. I use tapes of 10 G and find that a convenient size. I also set runtapes to 2, but be warned that that introduces a new and potentially messy factor into your tape requirements calculation. I also set tape_splitsize to 900 mb, which splits up large dumps into smaller chunks. 11of them will barely fit onto a 10 G vtape, so I get maximum use of each one. 3. I believe that vtape size is an upper bound, unused space on the vtape is actually free space on the disk, so you can actually oversubscribe the 300 gig pool by some percentage (depending on numerous factors that we currently do NOT have a handle on). Correct, and I have not looked into the calculation. I am not sure I understand what oversubscribe actually is but I wonder if I really need it. could you tell me what it will actually do in my case? Let's say you have 25 4 G tapes, for a theoretical supply of 100 G, so you give Amanda a partition of 100 G. But you notice that the runs tend to under-use each tape, by an average of 10%. So you are actually using 90 G. You could (in theory) add two more tapes to that partition. I recommend against it; I prefer to play with the tape size and tape_splitsize to make things better emulate tapes. 4. With 4 Gig tapes... how large are your DLEs ? 4 Gig is relatively small, are you using tape spanning or are all of the DLE's small ? I don't believe Tape spanning is configured. I just used Vtapes document from Amanda wiki and so used 4GB dvd size as it is mentioned there. Are you planning to back up vtapes to DVD? If not, I'd say, don't worry about the 4 G size and pick something that suits your data set. -- Charles Curley /\ASCII Ribbon Campaign Looking for fine software \ /Respect for open standards and/or writing? X No HTML/RTF in email http://www.charlescurley.com/ \No M$ Word docs in email Key fingerprint = CE5C 6645 A45A 64E4 94C0 809C FFF6 4C48 4ECD DFDB
Re: [Amanda-users] amanda.conf config help
On Tue, Nov 23, 2010 at 12:35:10PM -0500, upengan78 wrote: Firstly, thanks for the response! 1. You have 50 gig under backup/amanda control, but how much data are you currently backing up per week ? Currently not doing any backups through Amanda. I am trying to understand this whole theory of numbers involved in configuration. I want to start backing up a 50G partition with a Full backup on Monday and incrementals on each day till next week's Monday. So I think for 7 days I will have to backup 50G + 200M x 6 days = 50.6G . Let's say 55G a week. When you first run amanda it will see that you have no dumps of any of the DLEs (DiskList Entries) and will want to dump level 0 for all of them. Not knowing the physical config - local drives, networked amanda clients, bus architecture, you might consider starting with only a few DLE in your disklist file and adding a couple each day. Once you have a 'recent' dump, that is a level 0 without one dumpcycle, you will (tend to) perform non-0 dumps for other days. You talk in terms of 'new data' each day, do you touch existing files ? Its not a block level dump, its a file level dump so if you are appending to existing files or modifying existing files other than those you are creating with new data you will be backing them up also, so you may backup more data than you expect. How many DLEs, disklist entries, will you have, what are their relative sizes ? My original configuration is with little idea. dumpcycle 7 runspercycle 7 tapecycle 20 I though above would yield me 7 Days of backup including 1 Full and 6 incrementals in 1 dumpcycle and all 20 tapes will be used. I don't know if I am right. Correct in principle, in practice its a little more complicated. Once amanda gets a handle on the DLE sizes it will try to level the amanda of data across each day of dumps in the dumpcycle. This will take a while and you will find that if you have 'lots' of free space in your tape pool amanda may promote level 0 dumps and perform them for any given DLE in less than dumpcycle days. 2. With current settings, tapecycle and dumpcycle you will have between 2 and 3 full dumps per partition on tape at any one time, 2.x X 50 Gig 80 Gig so you are already short on (vtape) space. Correct I think I also think I am short of Vtapes. I wonder if I should add more 4.xx G size tapes or redo Vtapes of 40G size each to utilize whole 300GB of disk space. Did you plan on using the VTape pool for anything else ? How large are your DLEs ? Are they actually all small enough to expect level 0 dumps to fit on 4 gig volumes ? 3. I believe that vtape size is an upper bound, unused space on the vtape is actually free space on the disk, so you can actually oversubscribe the 300 gig pool by some percentage (depending on numerous factors that we currently do NOT have a handle on). I am not sure I understand what oversubscribe actually is but I wonder if I really need it. could you tell me what it will actually do in my case? Oversubscribe the vtape pool drive... I think if you, for instance, use vtapes of 10 gig and on some day write 6 gig that you have 4 gig of real space free. I don't think the vtapes are fixed size containers. This would leave 4 gig free that day and ultimately enough free space to have, 300/10 = 30 vtapes, perhaps 35 vtapes in the tape pool. Actual numbers depend on how full each tape will be, compression ration if you use it, that sort of thing. 4. With 4 Gig tapes... how large are your DLEs ? 4 Gig is relatively small, are you using tape spanning or are all of the DLE's small ? I don't believe Tape spanning is configured. I just used Vtapes document from Amanda wiki and so used 4GB dvd size as it is mentioned there. that's why I am here now to tweak it. DLEs in my disklist are as below, amandaclient.domain.com /backup/location comp-tar Ah - how many DVD do you typically write per day now ? Is that all level 0 or a mix of 0 and non-0 dumps ? Are you manually intermixing the levels now to even out dump times across your week ? If so, you can achieve the same result with amanda, but the scheduling becomes automatic. You don't normally want to force the levels but let amanda choose them for you. 5. With 4-gig tapes and 50 gig under backup you must use 12 tapes for a complete dump of the system, with a pool of 20 tapes you can not have two complete dump cycles... Okay. that makes sense. 6. It seems that either I'm badly misunderstanding your config or you are badly underconfiguring your tape pool. Yes, I can provide you whatever is required. I know even 300G may not very large but then I'd like to have best possible that I can make use of it as of now. Thanks for helping me out! 300 Gig is more than large enough to manage backups on 50 Gig of data. What is the structure of your vtape pool ? One large spindle ? Raid 5, raid 6 ? a pair of raid 1 drives
Re: [Amanda-users] amanda.conf config help
On 11/23/10 2:34 PM, upengan78 wrote: snip One thing I am confused about and want to confirm do I really need to use spanning in Virtual tapes ? snip *How many DLEs, disklist entries, will you have, what are their relative sizes ? Only 1 entry for a amanda-network client server. Size is about 50G and grows by 100M-200M a day or data size sometimes stays as it is for 2-3 days. Just a comment that this isn't making good use of Amanda. You will get a full backup of 50G, and then incrementals of 100M-200M on all the other days of the dump cycle. Making your vtapes all one size then wastes a lot of space, not to mention that the load is virtually all on one day of the cycle when the full backup is done. You should break up the 50G into multiple pieces. Suppose there are, say, 10 directories in there, and that stays the same. Make each of those directories a disklist entry. Then Amanda's planner will decide when to do a full of each directory in such a way as to smooth out the load and the usage of space in the vtapes over the span of the dump cycle. If it works out well, and you have a 7 day dump cycle, then you should end up getting about 7G+/- every day after it gets started. Since each dle has to have a full backup before it can do incrementals, you could add one dle each day to your disklist to get a smoother start. Otherwise, the first time you run it, it will want to do a full of everything anyway. Since you are not going to be using the space for anything else, make your vtapes about 10G. Give yourself, say, 21 of those. You will have some space left. If you set up a holding disk, then Amanda can run multiple dle's in parallel. Ideally, your holding disk should be somewhere else, but you could use the remaining 90G or so if you don't have anything else. I like to have a separate drive for a holding disk so that I'm not in competition for i/o bandwidth with anything that I might be backing up, or in your case with the vtapes. That's just an example of how to go about it. It would give you a typecycle with 3 weeks of backup with 3 full dump cycles of 7 days. -- --- Chris Hoogendyk - O__ Systems Administrator c/ /'_ --- Biology Geology Departments (*) \(*) -- 140 Morrill Science Center ~~ - University of Massachusetts, Amherst hoogen...@bio.umass.edu --- Erdös 4
Re: [Amanda-users] amanda.conf config help
Upendra, On Tue, Nov 23, 2010 at 02:34:17PM -0500, upengan78 wrote: Thanks for the advise. I understand you have asked me some questions and that is actually making me think of more points to be considered while sizing and make correct decisions. Some of the points that you have mentioned I did not really think of them when I started. Thanks for pointing them out. Pretty much that is what I do. I ask or re-ask refined questions and often lead conversations in that mode. One thing I am confused about and want to confirm do I really need to use spanning in Virtual tapes ? My assumption was that with 50 Gig of data you'd be backing up DLEs that where more than 4 gig. Since they clearly fit on DVD at that size you either don't use, will not need, spanning or you are allowing your dump/tar mechanism to roll your dumps onto secondary volumes. Er, vocabulary. DLEs traditionally had to fit on a tape volume. DLE-size = backup_media_size This causes problems when your backup segments are larger than your media, tape spanning was invented to handle this, it allowed the larger dumps to span across your output media. I guess in the earliest sense of it, dumps where unix partitions and they where small relative to physical tape. With increases in disk size and the advent of raid the partitions are allowed to become 'large' relative to the tape media. I don't know how you do your backups now, you may have included that information in your details but I have not absorbed it. If you have many small partitions to backup you can treat them each as a DLE and DUMP them to vtape. If you have larger partitions and have been TAR'ing segments then you can tell amanda that you are backing up a segment of the partition with TAR, actually gtar in most cases and continue to dump these small DLEs to your vtapes. You may, with the vtapes want to restructure your sementation of the data so that you have fewer but larger DLEs and increase the size of the vtapes accordingly. This document here mentions that one doesn't need to configure 'runtapes' variable with vtapes but looking at amdump o/p in previous email I think I must mention 'runtape' With jukeboxes, vtapes (where you can think of your vtapes as being in a virtual-jukebox) or attended backups where you can have an operator swap media for you, you can have more than one run-tape volumes written per amdump run. You must have some mechanism now to dump your data to multiple DVD each night, so in effect your backup schema utilizes a runtapes value greater than 1. Now back to the questions. @Charles * Are you planning to back up vtapes to DVD? If not, I'd say, don't worry about the 4 G size and pick something that suits your data set. No I am not planning on that. I think I better make tapesize larger like 40G or so @Brian *You talk in terms of 'new data' each day, do you touch existing files ? It will be modification of exsiting files as well as addition of directories and more files and may involve removal of directories/files. Traditional unix dump is the default for amdump, non-level 0 dumps utilize the well known level 1, 2... dumps. A level one will select files that have been modified since the most recent level 0, so any files that are either new or modified since the most recent lower level number will be selected. You have to expect in this case to backup more data than you are adding to your system, but also remember that its segregated by the DLE which in effect partitions your data. *How many DLEs, disklist entries, will you have, what are their relative sizes ? Only 1 entry for a amanda-network client server. Size is about 50G and grows by 100M-200M a day or data size sometimes stays as it is for 2-3 days. So, what mechanism do you use to back it up ? You are using either Tar or dump and allowing it to span output volumes ? *Did you plan on using the VTape pool for anything else ? No but I think I did not mention that actual filesystem size of RAID5 storage mounted is 300G and I only created 20 tapes of 4.x G size on it. I don't plan to use the remaining space for anything else. If you are backing up 50 Gig at one time you need to either have a 50 Gig vtape or you need to tell amanda it can span the output data across multiple vtape volumes. You need to enable tape spanning as well as set runtapes accordingly. If its practical I'd think in terms of dividing your data into different DLE, different backup partitions, perhaps by selecting a set of top level directories on the raid partition and treating them as separate partitions and telling amanda to dump with gtar rather than dump (which only dumps in partition level segments). There are many examples of this in the amanda wiki and on the internet in general, I can send you my own working examples. *How large are your DLEs ? Are they actually all small enough to expect level 0 dumps to fit on 4 gig volumes ? I have only 1 DLE
Re: [Amanda-users] amanda.conf config help
On Tue, 23 Nov 2010 14:34:17 -0500 upengan78 amanda-fo...@backupcentral.com wrote: One thing I am confused about and want to confirm do I really need to use spanning in Virtual tapes ? This document here mentions that one doesn't need to configure 'runtapes' variable with vtapes but looking at amdump o/p in previous email I think I must mention 'runtape' You don't have to use it, but you will find you get more efficient use of your storage space if you do. If all your DLEs are the same size, you won't need it because Amanda will even out the level 0 backups across the dump cycle, and then all the dumps will be of similar size. But that's a rare situation. More often there are one or two huge DLEs and the rest are small. Then you have to accommodate the occasional huge dump without wasting much space the rest of the time. Tape spanning lets you accommodate the huge DLEs by spreading them across multiple tapes. The rest of the time you use fewer tapes. -- Charles Curley /\ASCII Ribbon Campaign Looking for fine software \ /Respect for open standards and/or writing? X No HTML/RTF in email http://www.charlescurley.com/ \No M$ Word docs in email Key fingerprint = CE5C 6645 A45A 64E4 94C0 809C FFF6 4C48 4ECD DFDB
Re: [Amanda-users] amanda.conf config help
On Tue, Nov 23, 2010 at 02:34:17PM -0500, upengan78 wrote: Thanks for the advise. While Amanda can be used for your needs, it may not be an optimal choice. Amanda is a backup manager, she makes choices about what and how to backup your DLEs. But you are trying to do most or all of Amanda's job for her. OTOH, your environment is simple enough (one host, one DLE, 50GB data) that self-management is not unreasonable. Amanda's design addressed a problem that can be seen even in your scheme; a monster dump once a week, tiny incrementals the other days. Amanda tries to balance the amount of data dumped each day by spreading the level 0 dumps out over the entire dumpcycle. So each day some DLEs get incrementals and a few others get full dumps. Obviously with only one DLE, it is difficult to spread the fulls out. Sizing your vtapes is a concern. Make them large enough to hold a full dump and you waste lots of space on incremental days. To deal with this, choose a relatively small size and tell Amanda that she can use multiple tapes if needed (i.e. set runtapes to an appropriate number). You will also have to allow tape spanning, otherwise the media has to be big enough to hold the entire DLE. Spanning is not default. You can create multiple DLEs from your current single DLE. This is done using include and exclude directives. For example, you could have /var, /home, /usr/share, /usr ex- cluding /usr/share, /opt, etc. Then a final DLE that starts at / and is a catch-all which excludes the directories in the other DLEs. You asked what someone meant by oversubscribing your vtapes. It means your vtapes are sized to use more than one per dump and recognizing that, on average, the last one each day is only half full. My setup has 2300GB of space for vtapes and I've created 100 x 25GB tapes, i.e. 2500GB if all are full. But because of those partially full tapes I actually have 270GB of free space, only about 85% full. I could push it and add another 5 or 10 vtapes :)). Jon -- Jon H. LaBadie j...@jgcomp.com JG Computing 12027 Creekbend Drive (703) 787-0884 Reston, VA 20194 (703) 787-0922 (fax)
