Re: Tapetype when utilizing hardware compression
On Thursday 17 January 2002 09:10 am, Don Potter wrote:
I ran the tapetype test to our tapedrive (ADIC DS9400D) using
DLTTAPE IV. I frontpaneled the compression so I expected at
least 40 GB when the tapetype was completed. But I only got
about 17GB:
Command: tapetype -d /dev/rmt/0n
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}
Then I ran it with software compression (/dev/rmt/0cn) and I
only got 20 GB:
Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}
Both ways I would of expected close to double the native writes.
Any ideas why the compression would not of increased.
Any suggestions would be appreciated.
Don Potter
First Don, be aware that tapetype uses /dev/urandom as a data
source, and /dev/urandom prides itself on being as truely random
as it can be. It takes repeatable, predictable data to be able
to compress it by any great amount.. The output of urandom wil
typically drive a hardware compressor to make a file bigger, not
smaller.
What it boils down to is that the values you get from tapetype
will be truely the absolute worst case values. Typical hardware
compression will gain 2/1 on text and such sparse files, while a
really good software algorythm can easily double that again.
However, the hardware compression can be easily defeated by
preceeding it with a good software compressor so that the copy on
the tape might be 10 or more percent larger on tape than the
actual compressed file is.
If you have the cpu horspower, always use software only, with the
hardware compression in the drive disabled.
--
Cheers, gene
Tapetype when utilizing hardware compression
I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV.
I frontpaneled the compression so I expected at least 40 GB when the tapetype
was completed. But I only got about 17GB:
Command: tapetype -d /dev/rmt/0ndefine tapetype unknown-tapetype {
comment "just produced by tapetype program"
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB:Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment "just produced by tapetype program"
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased.Any suggestions would be appreciated.Don Potter
Re: Tapetype when utilizing hardware compression
On Thu, 17 Jan 2002 at 9:10am, Don Potter wrote I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the compression so I expected at least 40 GB when the tapetype was completed. But I only got about 17GB: tapetype writes random data, which compresses horribly. So horribly, in fact, that it expands. Thus your results. Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB: Err, I don't think that does what you think it does. That's the compressing tape device, which should do the same thing as toggling the switch on the front panel. It's still hardware compression. Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased. Well, double is *always* optimistic. But, as I said above, random data doesn't compress. To use amanda with hardware compression, do the following: 1) Run tapetype on your tapes using *no* hardware compression. 2) Multiply that number by an estimate of how well your data compresses. 3) In your tapetype in amanda.conf, use the bigger number. 4) As amanda runs, you may need to tweak your tape length in amanda.conf. As an example, I've got a Sony AIT1 (35GB native) that I use with hardware compression. Depending on the data, I get between 45-55GB on there. I've set the length in my tapetype to 5MB. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
RE: Tapetype when utilizing hardware compression
If you
read the instructions for tapetype, it says to run it without compression.
It will typically report close to your native capacity which it looks like it is
doing.
-Original Message-From: Don Potter
[mailto:[EMAIL PROTECTED]]Sent: Thursday, January 17, 2002
8:11 AMTo: amanda-usersSubject: Tapetype when utilizing
hardware compressionI ran the tapetype test to our
tapedrive (ADIC DS9400D) using DLTTAPE IV. I frontpaneled the
compression so I expected at least 40 GB when the tapetype was completed.
But I only got about 17GB:
Command: tapetype -d /dev/rmt/0ndefine tapetype unknown-tapetype {
comment "just produced by tapetype program"
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}Then I ran it with software compression (/dev/rmt/0cn) and I only got 20 GB:Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment "just produced by tapetype program"
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}Both ways I would of expected close to double the native writes. Any ideas why the compression would not of increased.Any suggestions would be appreciated.Don Potter
Re: Tapetype when utilizing hardware compression
IIRC, the tapetype test uses random data, so hardware compress may (?)
actually increase the amount of the data.
-Kevin Zembower
-
E. Kevin Zembower
Unix Administrator
Johns Hopkins University/Center for Communications Programs
111 Market Place, Suite 310
Baltimore, MD 21202
410-659-6139
Don Potter [EMAIL PROTECTED] 01/17/02 09:10AM
I ran the tapetype test to our tapedrive (ADIC DS9400D) using DLTTAPE
IV. I frontpaneled the compression so I expected at least 40 GB when
the tapetype was completed. But I only got about 17GB:
Command: tapetype -d /dev/rmt/0n
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}
Then I ran it with software compression (/dev/rmt/0cn) and I only got
20 GB:
Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}
Both ways I would of expected close to double the native writes. Any
ideas why the compression would not of increased.
Any suggestions would be appreciated.
Don Potter
Re: Tapetype when utilizing hardware compression
On Thursday 17 January 2002 09:10 am, Don Potter wrote:
I ran the tapetype test to our tapedrive (ADIC DS9400D) using
DLTTAPE IV. I frontpaneled the compression so I expected at
least 40 GB when the tapetype was completed. But I only got
about 17GB:
Command: tapetype -d /dev/rmt/0n
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}
Then I ran it with software compression (/dev/rmt/0cn) and I
only got 20 GB:
Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}
Both ways I would of expected close to double the native writes.
Any ideas why the compression would not of increased.
Any suggestions would be appreciated.
Don Potter
First Don, be aware that tapetype uses /dev/urandom as a data
source, and /dev/urandom prides itself on being as truely random
as it can be. It takes repeatable, predictable data to be able
to compress it by any great amount.. The output of urandom wil
typically drive a hardware compressor to make a file bigger, not
smaller.
What it boils down to is that the values you get from tapetype
will be truely the absolute worst case values. Typical hardware
compression will gain 2/1 on text and such sparse files, while a
really good software algorythm can easily double that again.
However, the hardware compression can be easily defeated by
preceeding it with a good software compressor so that the copy on
the tape might be 10 or more percent larger on tape than the
actual compressed file is.
If you have the cpu horspower, always use software only, with the
hardware compression in the drive disabled.
--
Cheers, gene
Re: Tapetype when utilizing hardware compression
Thank You all for all your input...you have enlightened me immensely
.became so used to using commercial applications that due the
thinking for you.
Don
Gene Heskett wrote:
On Thursday 17 January 2002 09:10 am, Don Potter wrote:
I ran the tapetype test to our tapedrive (ADIC DS9400D) using
DLTTAPE IV. I frontpaneled the compression so I expected at
least 40 GB when the tapetype was completed. But I only got
about 17GB:
Command: tapetype -d /dev/rmt/0n
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 17587 mbytes
filemark 13 kbytes
speed 1011 kps
}
Then I ran it with software compression (/dev/rmt/0cn) and I
only got 20 GB:
Command: tapetype -d /dev/rmt/0cn
define tapetype unknown-tapetype {
comment just produced by tapetype program
length 19565 mbytes
filemark 4 kbytes
speed 1101 kps
}
Both ways I would of expected close to double the native writes.
Any ideas why the compression would not of increased.
Any suggestions would be appreciated.
Don Potter
First Don, be aware that tapetype uses /dev/urandom as a data
source, and /dev/urandom prides itself on being as truely random
as it can be. It takes repeatable, predictable data to be able
to compress it by any great amount.. The output of urandom wil
typically drive a hardware compressor to make a file bigger, not
smaller.
What it boils down to is that the values you get from tapetype
will be truely the absolute worst case values. Typical hardware
compression will gain 2/1 on text and such sparse files, while a
really good software algorythm can easily double that again.
However, the hardware compression can be easily defeated by
preceeding it with a good software compressor so that the copy on
the tape might be 10 or more percent larger on tape than the
actual compressed file is.
If you have the cpu horspower, always use software only, with the
hardware compression in the drive disabled.
