[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith g...@krypto.org added the comment: fix for J. David's issue submitted to trunk r73565 and py3k r73566 just in time for the 3.1 release. release30-maint r73567 release26-maint r73568 -- status: open - closed versions: +Python 3.0, Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gaëtan de Menten g...@openhex.org added the comment: Regarding the issue J. David Ibáñez has, I have a few comments to add: - It's also present on 32bit. - AFAICT: * it's present in both 2.6 branch trunk (as of 68886), * it's a problem with line 1110 (in 2.6 branch), or line 1122 in trunk, which should read LLL instead of lLL, * it is an omission from changeset 61591: http://svn.python.org/view/python/trunk/Lib/zipfile.py?view=diffr1=61590r2=61591 which changed line 964 (at the time) http://svn.python.org/view/python/trunk/Lib/zipfile.py?annotate=61591#l964 but not the corresponding one in writestr. I'm not sure whether or not a separate bug report should be opened for this issue. -- nosy: +gdementen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
J. David Ibáñez jda...@itaapy.com added the comment: There it is. -- Added file: http://bugs.python.org/file13630/Document.odt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith g...@krypto.org added the comment: J. David Ibáñez - do you still happen to have that Document.odt laying around? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith g...@krypto.org added the comment: seems there are bugs with it not staying signed as it should on some 64bit platforms. i'll be looking into this shortly. its a good candidate bug for 2.6.x and 3.0.x releases. -- keywords: -easy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith g...@krypto.org added the comment: err not 3.0.x, 3.0 is always unsigned like anyone sane would want. :) -- versions: -Python 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
J. David Ibáñez jda...@itaapy.com added the comment:
I believe I have hit this bug. With Python 2.6.1 in a Gentoo Linux
64 bits.
This code:
from zipfile import ZipFile
inzip = ZipFile('Document.odt')
outzip = ZipFile('out.zip', 'w')
for f in inzip.infolist():
if f.filename != 'content.xml':
print f.filename, '(CRC: %s)' % f.CRC
outzip.writestr(f, inzip.read(f.filename))
outzip.close()
Produces this output:
...
styles.xml (CRC: 3930581528)
test_odt.py:10: DeprecationWarning: 'l' format requires -2147483648 =
number = 2147483647
outzip.writestr(f, inzip.read(f.filename))
...
The CRC is not a 32bits signed, and then the module struct complains,
here:
zipfile.py:1098
self.fp.write(struct.pack(lLL, zinfo.CRC, zinfo.compress_size,
Apparently 'struct.pack' expects 'zinfo.CRC' to be a 32 signed it,
which is not.
I can attach the 'Document.odt' file if you want.
--
nosy: +jda...@itaapy.com
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1202
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Facundo Batista [EMAIL PROTECTED] added the comment: Let me reopen this, I think we have an issue with this fix. The conclusion of this discussion so far is that in 3.0 the crc32 will behave like the standard, which is a good thing (tm), but in 2.6 it will not: it should return a signed integer. I agree with this outcome! The documentation for 2.6, the commit message for the fix and what it's said here states that: 2.6 always returns signed, 2**31...2**31-1 come back as negative integers. This is *not* actually happening: s = **10 print zlib.crc32(s) # 2.6, 32 bits -872059092 print zlib.crc32(s) # 2.6, 64 bits 3422908204 The problem in the code is, IMHO, that the 32b rounding is being forced by assigning the result to an int (Modules/zlibmodule.c, line 929), but this rounding does not actually work for 64b (because the int has 64 bit, and even as it's signed, the number stays big and positive). Thank you! -- nosy: +facundobatista resolution: fixed - status: closed - open ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith [EMAIL PROTECTED] added the comment: An int is 32-bits on all popular platforms. Anyways i'll double check things. What platforms did you run your test on? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Michael Becker [EMAIL PROTECTED] added the comment: In case it isn't obvious the work around for pre 3.0 to get the right sum is something like: x=zlib.adler32(str) if x 0: x=(long(x) + 4294967296L) # 2^32, long may or may not be needed here -- nosy: +mbecker __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith [EMAIL PROTECTED] added the comment: The workaround I prefer to use is: x = zlib.adler32(mystr) 0xL __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith [EMAIL PROTECTED] added the comment: working on this now. foo = 'abcdefghijklmnop' 2.x 32-bit long: zlib.crc32(foo) returns -1808088941 2.x 64-bit long: zlib.crc32(foo) returns 2486878355 This is because PyInt_FromLong() happily fits the value into a signed long internally to the integer object on 64-bit platforms. They are both the same number if considered with 0x. I'm doing as guido suggests and leaving this slightly odd behavior for 2.x so that crc32 and adler32 always return an integer object. in 3.0 they'll always return an unsigned value. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith [EMAIL PROTECTED] added the comment: question: should I also make 64-bit 2.x return a signed value as well to be consistent with 32bit python 2.x? Consistency in case someone ever pickles the value and sends it to another python instance of a different word length would be good... __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Guido van Rossum [EMAIL PROTECTED] added the comment: Sure. (Though sending pickles to 3.0 would still cause problems. Consumers of pickled checksums would do wise to *always* take the CRC mod 2**32 before doing comparisons.) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Jesús Cea Avión [EMAIL PROTECTED] added the comment: I store CRC in reed-solomon schema of mine. I compare with equality, so, I think we should enforce CRC(python 32 bits) == CRC(python 64 bits). I will need to touch my code in python 3.0, but that will be inevitable anyway... __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Gregory P. Smith [EMAIL PROTECTED] added the comment: fixed. 3.0 always returns unsigned. 2.6 always returns signed, 2**31...2**31-1 come back as negative integers. trunk r61449 branches/py3k r61459 -- resolution: - fixed status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Changes by Gregory P. Smith: -- assignee: - gregory.p.smith keywords: +64bit, easy nosy: +gregory.p.smith priority: - normal versions: +Python 2.6, Python 3.0 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Changes by Jesús Cea Avión: -- nosy: +jcea __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Armin Rigo added the comment: The C reference code in rfc1950 for Adler-32 and in rfc1952 for CRC-32 compute with and return unsigned long values. From this point of view, returning negative values on 32-bit machines from CPython's zlib module can be considered a bug. That only leaves open the question of backward compatibility. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Armin Rigo added the comment: Obscure but reasonable. (I suspect you meant to say that py3k should return the *unsigned* value for better compliance with the standard.) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Guido van Rossum added the comment: Obscure but reasonable. (I suspect you meant to say that py3k should return the *unsigned* value for better compliance with the standard.) Yes. :) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Tim Lesher added the comment: Both CRC-32 and ADLER32 are standards (described in ISO 3309 and RFC 1950 respectively); whatever fix implemented should make sure that the output complies. ISO 3309 isn't available online as far as I can see, but CRC-32 reference code is published in RFC 1952; RFC 1950 contains reference code for ADLER32. -- nosy: +tlesher __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
Guido van Rossum added the comment: Since it's basically a magic cookie, not a meaningful numeric value, I'd propose sticking with backwards compatibility and fixing the 64-bit version to return a signed version. return x - ((x 0x8000) 1) anyone? -- nosy: +gvanrossum __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1202] zlib.crc32() and adler32() return value
New submission from Armin Rigo: The functions zlib.crc32() and zlib.adler32() return a signed value in the range(-2147483648, 2147483648) on 32-bit platforms, but an unsigned value in the range(0, 4294967296) on 64-bit platforms. This means that half the possible answers are numerically different on these two kinds of platforms. Ideally, this should be fixed by having them always return unsigned numbers (their C return type is unsigned too). It's unclear if we can do this without breaking existing code, though :-( -- components: Extension Modules messages: 56130 nosy: arigo severity: normal status: open title: zlib.crc32() and adler32() return value type: behavior __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1202 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
