Hickey, Larry uttered:
I have a blob structure which is primarily doubles. Is there anyone with
some experience with doing data compression to make the blobs smaller?
No experience with compressing blobs...
Tests I have
run so far indicate that compression is too slow on blobs of a few meg to
be practical.
I get now at least 20 to 40 inserts per second but if a single compression
takes over a
second, it's clearly not worth the trouble. Does anybody have experience
with a compression scheme with blobs that consist of mostly arrays of
doubles?
If you have lots of common values, such as 0.0 or 1.0, then you may get
very good compression. However, if the doubles are the result of arbitrary
calculations, that may hit the binary representation of your data, so that
two values close to 1.0 may actually be 0.9999999999 and 0.999999999996.
The data may compress better if stored as a textual representation, and be
more portable to boot. But if the format is out of your hands, that may
not be an option.
Some schemes ( ibsen) offer lightening speed decompression so if the
database was primarily used to read, this would be good choice but very
expensive to do
the compression required to make it.
You don't indicate what compression algorithms you've tried (other than
ibsen).
You may find LZO an alternative, as it is designed for high performance
(run time) compression. It is GPL, but commercial licenses are available.
You might also want to try zlib, which may or may not perform better than
ibsen, but has liberal licensing.
If your blobs are big, and your compression is fast, you may have higher
performance due to the reduction in IO. But something like bzip2 will
likely reduce your performance too much.
Christian
--
/"\
\ / ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
X - AGAINST MS ATTACHMENTS
/ \
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
