Re: Howto utilise DDS-4 20/40GB tape with Amanda
Hi PB My other issue is that concerns me If I also increase the number of tapes I currently use from 10 to possibly 20 will this enable me in the future to backup a server that is 80GB using incremental dumps and what is best for my monthly 20GB tapes that I wanted to pull out to archive as I wanted to do a full backup to 1 tape if this is now not possible could I still to the same as with my daily config and increase the runspercycle to 5 or more. I am using in my Daily amanda.conf dumpcycle 4 weeks # the number of days in the normal dump cycle # Setting dumpcycle to 0 - Setting this to zero tries to do a full backup each run. runspercycle 1 # the number of amdump runs in dumpcycle days # (4 weeks * 5 amdump runs per week -- just weekdays) tapecycle 10 tapes Would it be better to use your full suggestion as below: dumcycle 1 week runspercycle 5 # each working day tapecycle 20 tapes My Monthly is currently as below: dumpcycle 0 weeks # the number of days in the normal dump cycle runspercycle 1 # the number of amdump runs in dumpcycle days # (4 weeks * 5 amdump runs per week -- just weekdays) tapecycle 12 tapes But would these mean that I will have to change the tape frequently during the day. I am contemplating cannibalising a workstation and installing a DLT drive as a possible solution but cost rather a lot. Cheers On Mon, 2005-09-12 at 09:58 +0200, Paul Bijnens wrote: > Chuck Amadi Systems Administrator wrote: > > Hi relating to possible > > > > Here's my current amanda.conf > > > > dumpcycle 0 weeks > > runspercycle 1 > > tapecycle 10 tapes > > > > Thus increase runspercycle to any figure below tapecycle of 10 tapes. > > > > So for example runspercycle 1 would be sufficient. > > oho. You're doing a full dump each run. > But your tapecapacity is not sufficient to do that. > So that leaves you to do "runtapes 2" or more (use more > tapes for each run. > Or increase the dumpcycle (to avoid a full dump each time), and > then run multiple times during such a dumpcycle. > > Why not: > > dumcycle 1 week > runspercycle 5# each working day > tapecycle 10 tapes > > And if you need more history increase the number of tapes. > > > > > > Cheers > > > > > > > > > >>Chuck Amadi Systems Administrator wrote: > >> > >>>Hi I have managed to config the hard disk partitions in my disklist but > >>>I am short of tape space to include the remainder of the hard disk > >>>partitions. > >>> > >>>I had initially run the following commands is it possible to get > >>>compression functioning to enable to use 40GB as opposed to native 20GB. > >>> > >>>* - amtapetype -f /dev/nst0 # determine tape type takes along time > >> > >>When specifying also a realistic estimate "-e 20 g" it is usually much > >>faster (2-4 hours) > >> > >> > >> > >>>* - turn off datacompression run mt command that controls magnetic tape > >>>drive operation. * - mt -f /dev/nst0 status and mt -f /dev/nst0 > >>>datcompression 0 Compression off. > >>> > >>>As I don't want to buy bigger tapes if possible. > >> > >> > >>If you already use software compression, than enabling hardware > >>compression will _reduce_ your effective tapecapacity by 15-35%, because > >>the compression algorithms in those tapedrives behave bad on already > >>compressed data. > >> > >>When not too many files change, you can run less fulls and more > >>incrementals on each run by increasing the dumpcycle (runspercycle > >>actually). Or increase "runtapes" (and flush each morning the > >>remainders if you have only one drive). > >> > >> > > -- Unix/ Linux Systems Administrator Chuck Amadi The Surgical Material Testing Laboratory (SMTL), Princess of Wales Hospital Coity Road Bridgend, United Kingdom, CF31 1RQ. Email chuck.smtl.co.uk Tel: +44 1656 752820 Fax: +44 1656 752830
Re: Howto utilise DDS-4 20/40GB tape with Amanda
On Fri, Sep 09, 2005 at 05:26:39PM +0200, Paul Bijnens wrote: > Jon LaBadie wrote: > >I don't think it is actually a 'flag', but merely the presence of HW > >compressed data at the front of the tape. The compression setting > >is only relevant when writing tapes. Reading switches automatically > >based on the type of data (HW compressed or not) written before. > >And that "auto switch" is what causes the problem. You set the > >tape to compression off, read a bit of compressed data (like the > >amanda label at the start of the tape previously written with > >compression on) and the drive "auto switches" to compression on. > > > >To clear this state for amanda you have to write a bunch of data > >at the beginning of the tape, after setting compression off, and > >be sure to not read anything between the two. > > > The good news is, that when I try to reproduce the problem, I cannot > (anymore). > > A few years ago, I struggled with the compression settings of my > tape drive (a DDS-2 drive), and I used sequences like below > to test the hardware compression status of the drive. > It was then that I added the hwcompr test heuristics to amtapetype, > at that time still called "tapetype". > I'm not 100% sure, but I believe I used a Linux kernel 2.4.18. > > I try to reproduce the problem now, but my current config does NOT > autoswitch after reading a tape: > > $ mt -f /dev/st2 compression 1 > $ amtapetype -oc -f /dev/st2 >... yes hw compression enabled > $ mt -f /dev/st2 compression 0 > $ amtapetype -oc -f /dev/st2 >... no warning about hw compression!!! > > Amtapetype does read the tape before writing! I even added a > "dd if=/dev/st2 of=/dev/null bs=32k" inbetween, and still got > same result. > > That either means that phase of the moon is correct for handling of > tapes, or that current versions of Linux do not have the problem > anymore. It tried it both on a 2.4.20 kernel and a 2.6.9 kernel, > and both do not autoswitch for writing (they do for reading, just as > expected). > > Can anyone still reproduce the problem? And under what circumstances? Not being a linux tape user I'm not sure, but wasn't there some change in the way scsi tape control information could be set a while back? Maybe there was a new driver introduced that stores the settings and reapplies them on open for write. I've always wondered why we never heard the opposite problem. I.e. people who had used software compression and now are trying hardware compression but it doesn't seem to be working. Amanda would read the uncompressed data and the drive would switch off compression before anything was written. But I don't recall any such reports. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Howto utilise DDS-4 20/40GB tape with Amanda
Hi Cheers for the guidance. Thanks On Fri, 2005-09-09 at 09:37 -0400, Jon LaBadie wrote: > On Fri, Sep 09, 2005 at 09:39:31AM +0100, Chuck Amadi Systems Administrator > wrote: > > Hi I have managed to config the hard disk partitions in my disklist but > > I am short of tape space to include the remainder of the hard disk > > partitions. > > > > I had initially run the following commands is it possible to get > > compression functioning to enable to use 40GB as opposed to native 20GB. > > > > * - amtapetype -f /dev/nst0 # determine tape type takes along time > > * - turn off datacompression run mt command that controls magnetic tape > > drive operation. * - mt -f /dev/nst0 status and mt -f /dev/nst0 > > datcompression 0 Compression off. > > > > As I don't want to buy bigger tapes if possible. > > Your tape's capacity is 20GB! Not 40GB, never will be, no way, no how. > > But compression (hardware or software) might be able to shrink a lot of > data down to 20GB. > > The rub is that compression algorithms, whether applied by your computer > or by the computer in your tape drive, are only effective the first time > they are applied. And as PB pointed out, when fed already compressed > data, the dumb algorithm used by the tape drive is likely to expand, > rather than shrink, the data! > > An analogy, your new 250GB hard disk drive. If you use it to store a > bunch of zip'ed files, does it become a 500GB hard disk drive? No, > it is still a 250GB drive that happens to be storing compressed data. > > Similarly, your 20GB tape stores 20GB of data. Those data may be > 20GB of plain old uncompressed information, or a lot more information > that is compressed by your computer before taping or by the drive > during taping -- but not both!. > -- Unix/ Linux Systems Administrator Chuck Amadi The Surgical Material Testing Laboratory (SMTL), Princess of Wales Hospital Coity Road Bridgend, United Kingdom, CF31 1RQ. Email chuck.smtl.co.uk Tel: +44 1656 752820 Fax: +44 1656 752830
Re: Howto utilise DDS-4 20/40GB tape with Amanda
Jon LaBadie wrote: I don't think it is actually a 'flag', but merely the presence of HW compressed data at the front of the tape. The compression setting is only relevant when writing tapes. Reading switches automatically based on the type of data (HW compressed or not) written before. And that "auto switch" is what causes the problem. You set the tape to compression off, read a bit of compressed data (like the amanda label at the start of the tape previously written with compression on) and the drive "auto switches" to compression on. To clear this state for amanda you have to write a bunch of data at the beginning of the tape, after setting compression off, and be sure to not read anything between the two. The good news is, that when I try to reproduce the problem, I cannot (anymore). A few years ago, I struggled with the compression settings of my tape drive (a DDS-2 drive), and I used sequences like below to test the hardware compression status of the drive. It was then that I added the hwcompr test heuristics to amtapetype, at that time still called "tapetype". I'm not 100% sure, but I believe I used a Linux kernel 2.4.18. I try to reproduce the problem now, but my current config does NOT autoswitch after reading a tape: $ mt -f /dev/st2 compression 1 $ amtapetype -oc -f /dev/st2 ... yes hw compression enabled $ mt -f /dev/st2 compression 0 $ amtapetype -oc -f /dev/st2 ... no warning about hw compression!!! Amtapetype does read the tape before writing! I even added a "dd if=/dev/st2 of=/dev/null bs=32k" inbetween, and still got same result. That either means that phase of the moon is correct for handling of tapes, or that current versions of Linux do not have the problem anymore. It tried it both on a 2.4.20 kernel and a 2.6.9 kernel, and both do not autoswitch for writing (they do for reading, just as expected). Can anyone still reproduce the problem? And under what circumstances? -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
Re: Howto utilise DDS-4 20/40GB tape with Amanda
On Fri, Sep 09, 2005 at 09:39:31AM +0100, Chuck Amadi Systems Administrator wrote: > Hi I have managed to config the hard disk partitions in my disklist but > I am short of tape space to include the remainder of the hard disk > partitions. > > I had initially run the following commands is it possible to get > compression functioning to enable to use 40GB as opposed to native 20GB. > > * - amtapetype -f /dev/nst0 # determine tape type takes along time > * - turn off datacompression run mt command that controls magnetic tape > drive operation. * - mt -f /dev/nst0 status and mt -f /dev/nst0 > datcompression 0 Compression off. > > As I don't want to buy bigger tapes if possible. Your tape's capacity is 20GB! Not 40GB, never will be, no way, no how. But compression (hardware or software) might be able to shrink a lot of data down to 20GB. The rub is that compression algorithms, whether applied by your computer or by the computer in your tape drive, are only effective the first time they are applied. And as PB pointed out, when fed already compressed data, the dumb algorithm used by the tape drive is likely to expand, rather than shrink, the data! An analogy, your new 250GB hard disk drive. If you use it to store a bunch of zip'ed files, does it become a 500GB hard disk drive? No, it is still a 250GB drive that happens to be storing compressed data. Similarly, your 20GB tape stores 20GB of data. Those data may be 20GB of plain old uncompressed information, or a lot more information that is compressed by your computer before taping or by the drive during taping -- but not both!. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Howto utilise DDS-4 20/40GB tape with Amanda
On Fri, Sep 09, 2005 at 11:07:16AM +0100, Rodrigo Ventura wrote: > > Hum, I'm a bit confused here... I read somewhere that it was not > trivial to disable hardware compression. I have an autoloader with a > DAT 72 drive, connected to a linux server. The command mt > datcompression 0 Compression off does not seem to work. A relevant question is how are you detecting the compression state of the compression drive before and after the mt command? > > I read that the hardware compression flag was written on the tape, and > to disable it I had to "clear" that bit in all tapes. Is this true? > How can I do that? See regular postings by Gene Heskett where he details his procedure. I don't think it is actually a 'flag', but merely the presence of HW compressed data at the front of the tape. The compression setting is only relevant when writing tapes. Reading switches automatically based on the type of data (HW compressed or not) written before. And that "auto switch" is what causes the problem. You set the tape to compression off, read a bit of compressed data (like the amanda label at the start of the tape previously written with compression on) and the drive "auto switches" to compression on. To clear this state for amanda you have to write a bunch of data at the beginning of the tape, after setting compression off, and be sure to not read anything between the two. > > I also read that there was non-documentated dip switch settings on the > drive that could disable hardware compression. I have a HP DAT > 72x6. The manuals just states the correct dip switch settings for each > operating system. Any way I could get the info on what exactly each > dip switch does? Oh, it is well documented. One thing HP does pretty good is document. Go to their support website for storage, tape drives. Locate your model (may have to get to some legacy section, maybe not). There will be a listing for user manuals or docs or somesuch. Download not just the user manuals, but the technical docs too. I have a dds2 drive and a dds3 autoloader. Recently I glanced at the docs for dds4 & 5 drives. Switch is still there from what I saw. On my dds2 drive I have to take the cover off to access the switch from the bottom. On the bottom of my dds3 6x24 autoloader there is a 3 cm cave about 5 cm deep exposing the dip switch. Two of the dip switches are relevant. I don't know their numbers. One says, when powered up, what state is compression set to, on or off. The second says can this state be changed by software commands, yes or no. I.e. whether you can change the power up state with your mt command. For my two drives the manufacturer's defaults were on and yes. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Howto utilise DDS-4 20/40GB tape with Amanda
Hum, I'm a bit confused here... I read somewhere that it was not trivial to disable hardware compression. I have an autoloader with a DAT 72 drive, connected to a linux server. The command mt datcompression 0 Compression off does not seem to work. I read that the hardware compression flag was written on the tape, and to disable it I had to "clear" that bit in all tapes. Is this true? How can I do that? I also read that there was non-documentated dip switch settings on the drive that could disable hardware compression. I have a HP DAT 72x6. The manuals just states the correct dip switch settings for each operating system. Any way I could get the info on what exactly each dip switch does? Cheers, Rodrigo -- *** Rodrigo Martins de Matos Ventura <[EMAIL PROTECTED]> *** Web page: http://www.isr.ist.utl.pt/~yoda *** Teaching Assistant and PhD Student at ISR: ***Instituto de Sistemas e Robotica, Polo de Lisboa *** Instituto Superior Tecnico, Lisboa, PORTUGAL *** PGP fingerprint = 0119 AD13 9EEE 264A 3F10 31D3 89B3 C6C4 60C6 4585
Re: Howto utilise DDS-4 20/40GB tape with Amanda
Chuck Amadi Systems Administrator wrote: Hi I have managed to config the hard disk partitions in my disklist but I am short of tape space to include the remainder of the hard disk partitions. I had initially run the following commands is it possible to get compression functioning to enable to use 40GB as opposed to native 20GB. * - amtapetype -f /dev/nst0 # determine tape type takes along time When specifying also a realistic estimate "-e 20 g" it is usually much faster (2-4 hours) * - turn off datacompression run mt command that controls magnetic tape drive operation. * - mt -f /dev/nst0 status and mt -f /dev/nst0 datcompression 0 Compression off. As I don't want to buy bigger tapes if possible. If you already use software compression, than enabling hardware compression will _reduce_ your effective tapecapacity by 15-35%, because the compression algorithms in those tapedrives behave bad on already compressed data. When not too many files change, you can run less fulls and more incrementals on each run by increasing the dumpcycle (runspercycle actually). Or increase "runtapes" (and flush each morning the remainders if you have only one drive). -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
