Hash: SHA1

Adam GROSZER wrote:
> Hello Jim,
> Tuesday, May 11, 2010, 4:46:46 PM, you wrote:
> JF> On Tue, May 11, 2010 at 7:47 AM, Adam GROSZER <agros...@gmail.com> wrote:
>>> Hello Jim,
>>> Tuesday, May 11, 2010, 1:37:19 PM, you wrote:
>>> JF> On Tue, May 11, 2010 at 7:13 AM, Adam GROSZER <agros...@gmail.com> 
>>> wrote:
>>>>> Hello Jim,
>>>>> Tuesday, May 11, 2010, 12:33:04 PM, you wrote:
>>>>> JF> On Tue, May 11, 2010 at 3:16 AM, Adam GROSZER <agros...@gmail.com> 
>>>>> wrote:
>>>>>>> Hello Jim,
>>>>>>> Monday, May 10, 2010, 1:27:00 PM, you wrote:
>>>>>>> JF> On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink <r...@fourdigits.nl> 
>>>>>>> wrote:
>>>>>>>>> That's really interesting! Did you notice any issues performance 
>>>>>>>>> wise, or
>>>>>>>>> didn't you check that yet?
>>>>>>> JF> I didn't check performance. I just iterated over a file storage 
>>>>>>> file,
>>>>>>> JF> checking compressed and uncompressed pickle sizes.
>>>>>>> I'd say some checksum is then also needed to detect bit failures that
>>>>>>> mess up the compressed data.
>>>>> JF> Why?
>>>>> I think the gzip algo compresses to a bit-stream, where even one bit
>>>>> has an error the rest of the uncompressed data might be a total mess.
>>>>> If that one bit is relatively early in the stream it's fatal.
>>>>> Salvaging the data is not a joy either.
>>>>> I know at this level we should expect that the OS and any underlying
>>>>> infrastructure should provide error-free data or fail.
>>>>> Tho I've seen some magic situations where the file copied without
>>>>> error through a network, but at the end CRC check failed on it :-O
>>> JF> How would a checksum help?  All it would do is tell you your hosed.
>>> JF> It wouldn't make you any less hosed.
>>> Yes, but I would know why it's hosed.
> JF> How so?  How would you know why it is hosed.
> Because of data corruption in the compressed stream.
> JF> Note BTW that the zlib format already includes a checksum.
> JF>   http://www.faqs.org/rfcs/rfc1950.html
> I missed that. Case closed then ;-) Sorry for the noise.

A zipped file is not different from other (binary) data stored within
the ZODB. Data corruption can occur always - zipped or not.

Side note: we implemented compression support as part of our CMS on the
application layer where we store large binary files as linked PData
chains. However we do not compress in any case - only for certain
content-types (it does not make sense to compress zip or jar files).
We also store a md5 hash for each object (and never had a corruption
issue so far).

Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/


<<attachment: lists.vcf>>

For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to