Re: Tapetype when utilizing hardware compression
On Thursday 17 January 2002 09:10 am, Don Potter wrote: I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0n define tapetype unknown-tapetype { comment just produced by tapetype program length 17587 mbytes filemark 13 kbytes speed 1011 kps } Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment just produced by tapetype program length 19565 mbytes filemark 4 kbytes speed 1101 kps } Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Any suggestions would be appreciated. Don Potter First Don, be aware that tapetype uses /dev/urandom as a data source, and /dev/urandom prides itself on being as truely random as it can be. It takes repeatable, predictable data to be able to compress it by any great amount.. The output of urandom wil typically drive a hardware compressor to make a file bigger, not smaller. What it boils down to is that the values you get from tapetype will be truely the absolute worst case values. Typical hardware compression will gain 2/1 on text and such sparse files, while a really good software algorythm can easily double that again. However, the hardware compression can be easily defeated by preceeding it with a good software compressor so that the copy on the tape might be 10 or more percent larger on tape than the actual compressed file is. If you have the cpu horspower, always use software only, with the hardware compression in the drive disabled. -- Cheers, gene
Tapetype when utilizing hardware compression
I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0ndefine tapetype unknown-tapetype { comment "just produced by tapetype program" length 17587 mbytes filemark 13 kbytes speed 1011 kps }Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB:Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment "just produced by tapetype program" length 19565 mbytes filemark 4 kbytes speed 1101 kps }Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased.Any suggestions would be appreciated.Don Potter
Re: Tapetype when utilizing hardware compression
On Thu, 17 Jan 2002 at 9:10am, Don Potter wrote I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: tapetype writes random data, which compresses horribly. So horribly, in fact, that it expands. Thus your results. Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Err, I don't think that does what you think it does. That's the compressing tape device, which should do the same thing as toggling the switch on the front panel. It's still hardware compression. Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Well, double is *always* optimistic. But, as I said above, random data doesn't compress. To use amanda with hardware compression, do the following: 1) Run tapetype on your tapes using *no* hardware compression. 2) Multiply that number by an estimate of how well your data compresses. 3) In your tapetype in amanda.conf, use the bigger number. 4) As amanda runs, you may need to tweak your tape length in amanda.conf. As an example, I've got a Sony AIT1 (35GB native) that I use with hardware compression. Depending on the data, I get between 45-55GB on there. I've set the length in my tapetype to 5MB. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
RE: Tapetype when utilizing hardware compression
If you read the instructions for tapetype, it says to run it without compression. It will typically report close to your native capacity which it looks like it is doing. -Original Message-From: Don Potter [mailto:[EMAIL PROTECTED]]Sent: Thursday, January 17, 2002 8:11 AMTo: amanda-usersSubject: Tapetype when utilizing hardware compressionI ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0ndefine tapetype unknown-tapetype { comment "just produced by tapetype program" length 17587 mbytes filemark 13 kbytes speed 1011 kps }Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB:Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment "just produced by tapetype program" length 19565 mbytes filemark 4 kbytes speed 1101 kps }Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased.Any suggestions would be appreciated.Don Potter
Re: Tapetype when utilizing hardware compression
IIRC, the tapetype test uses random data, so hardware compress may (?) actually increase the amount of the data. -Kevin Zembower - E. Kevin Zembower Unix Administrator Johns Hopkins University/Center for Communications Programs 111 Market Place, Suite 310 Baltimore, MD 21202 410-659-6139 Don Potter [EMAIL PROTECTED] 01/17/02 09:10AM I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0n define tapetype unknown-tapetype { comment just produced by tapetype program length 17587 mbytes filemark 13 kbytes speed 1011 kps } Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment just produced by tapetype program length 19565 mbytes filemark 4 kbytes speed 1101 kps } Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Any suggestions would be appreciated. Don Potter
Re: Tapetype when utilizing hardware compression
On Thursday 17 January 2002 09:10 am, Don Potter wrote: I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0n define tapetype unknown-tapetype { comment just produced by tapetype program length 17587 mbytes filemark 13 kbytes speed 1011 kps } Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment just produced by tapetype program length 19565 mbytes filemark 4 kbytes speed 1101 kps } Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Any suggestions would be appreciated. Don Potter First Don, be aware that tapetype uses /dev/urandom as a data source, and /dev/urandom prides itself on being as truely random as it can be. It takes repeatable, predictable data to be able to compress it by any great amount.. The output of urandom wil typically drive a hardware compressor to make a file bigger, not smaller. What it boils down to is that the values you get from tapetype will be truely the absolute worst case values. Typical hardware compression will gain 2/1 on text and such sparse files, while a really good software algorythm can easily double that again. However, the hardware compression can be easily defeated by preceeding it with a good software compressor so that the copy on the tape might be 10 or more percent larger on tape than the actual compressed file is. If you have the cpu horspower, always use software only, with the hardware compression in the drive disabled. -- Cheers, gene
Re: Tapetype when utilizing hardware compression
Thank You all for all your input...you have enlightened me immensely .became so used to using commercial applications that due the thinking for you. Don Gene Heskett wrote: On Thursday 17 January 2002 09:10 am, Don Potter wrote: I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: Command: tapetype -d /dev/rmt/0n define tapetype unknown-tapetype { comment just produced by tapetype program length 17587 mbytes filemark 13 kbytes speed 1011 kps } Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Command: tapetype -d /dev/rmt/0cn define tapetype unknown-tapetype { comment just produced by tapetype program length 19565 mbytes filemark 4 kbytes speed 1101 kps } Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Any suggestions would be appreciated. Don Potter First Don, be aware that tapetype uses /dev/urandom as a data source, and /dev/urandom prides itself on being as truely random as it can be. It takes repeatable, predictable data to be able to compress it by any great amount.. The output of urandom wil typically drive a hardware compressor to make a file bigger, not smaller. What it boils down to is that the values you get from tapetype will be truely the absolute worst case values. Typical hardware compression will gain 2/1 on text and such sparse files, while a really good software algorythm can easily double that again. However, the hardware compression can be easily defeated by preceeding it with a good software compressor so that the copy on the tape might be 10 or more percent larger on tape than the actual compressed file is. If you have the cpu horspower, always use software only, with the hardware compression in the drive disabled.