[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Roundup Robot added the comment: New changeset bd71352e950f by Alexandre Vassalotti in branch 'default': Issue #6784: Strings from Python 2 can now be unpickled as bytes objects. http://hg.python.org/cpython/rev/bd71352e950f -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Alexandre Vassalotti added the comment: I fixed up the last few review comments and submitted the patch. Thank you for the help! -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24640/BytestrPickler_c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24688/test_pickle.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24719/pickle_bytestr.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24906/pickle_bytes_code.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24907/pickle_bytes_tests.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24568/pickle.py.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen added the comment: Hi Alexandre, Attached is a diff based on r87793:0c508d87f80b. Merlijn -- Added file: http://bugs.python.org/file33011/bytestrpickle.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen added the comment: I have fixed most of the nits in this patch, except for: 1) the intermediate bytes object being created; inlining is an option, as storchaka suggested, but I'd rather have you decide what it should become before implementing it; 2) make clinic gives me ./python -E ./Tools/clinic/clinic.py --make Error in file ./Modules/_pickle.c on line 6611: Checksum mismatch! Expected: bed0d8bbe1c647960ccc6f997b33bf33935fa56f Computed: 58dcccb705487695fec30980f566027bc68d9c69 make: *** [clinic] Error 255 and I have no clue how to fix that -- the clinic docs are sparse, to say the least; 3) The tests are still in their own test case; please decide between the two of you what is the best solution; 4) I have grouped the test cases: test_load_python2_str_as_bytes (which checks protocols 0, 1, and 2), test_load_python2_unicode_as_str and test_load_long_python2_str_as_bytes; 5) I have moved the commands to create the shown pickled versions from docstrings to comments. If you think they are not useful, I'll remove them, but I found them pretty useful while shortening the strings. -- Added file: http://bugs.python.org/file33016/bytestrpickle.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Alexandre Vassalotti added the comment: I cleaned up the patch. I will submit it tonight if there is no major objections. -- Added file: http://bugs.python.org/file33019/pickle_python2_str_as_bytes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Antoine Pitrou added the comment: How about updating the documentation as well? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Serhiy Storchaka added the comment: And what about an issue mentioned in msg153659? -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Alexandre Vassalotti added the comment: Could you provide a single patch with the implementation and the tests together? I will try to find some time this week to review this. -- assignee: docs@python - alexandre.vassalotti priority: normal - high stage: - patch review versions: +Python 3.4 -Python 2.7, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Mikhail Korobov kmik...@gmail.com: -- nosy: +kmike ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: Based on the discussion on python-dev [1], this is an updated implementation that uses encoding='bytes' to signal str-bytes behaviour. http://mail.python.org/pipermail/python-dev/2012-March/117536.html -- Added file: http://bugs.python.org/file24906/pickle_bytes_code.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: ...and the tests to go with that. -- Added file: http://bugs.python.org/file24907/pickle_bytes_tests.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24567/test_bytestrpickle.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment:
OK, and now a version that's not broken... I forgot to initialize self-bytestr
for PicklerObject/UnpicklerObject. *puts on the you-broke-the-build-hat*
Except for test_packaging.test_caches, this version passes all tests --
test_packaging.test_caches, which seems to fail because I make install'd python
and installed {distribute,pip,setuptools,virtualenv}.
--
Added file: http://bugs.python.org/file24719/pickle_bytestr.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6784
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Removed file: http://bugs.python.org/file24714/pickle_bytestr.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: And a complete patch that implements the tests, the python implementation and the C implementation. I'm not completely happy with the code duplication in read_string/read_binstring/read_short_binstring C implementation, so that might be an improvement (however, there is already a lot of code duplication there at the moment). Again: comments would be very welcome... -- Added file: http://bugs.python.org/file24714/pickle_bytestr.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: Added tests in Lib/test format. After applying pickle.py.patch and BytestrPickler_c.diff, ./python -m test -v -m PyPicklerBytestrTests test_pickle returns 12 tests, no errors, while ./python -m test -v -m CPicklerBytestrTests test_pickle only passes test_dump_bytes_protocol_0 (test.test_pickle.CPicklerBytestrTests) ... ok test_dump_bytes_protocol_1 (test.test_pickle.CPicklerBytestrTests) ... ok test_dump_bytes_protocol_2 (test.test_pickle.CPicklerBytestrTests) ... ok test_dump_bytes_protocol_3 (test.test_pickle.CPicklerBytestrTests) ... ok and has 8 errors (as expected). -- Added file: http://bugs.python.org/file24688/test_pickle.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: Ok, this is my first attempt at the Pickler part of the C implementation. I'll have to adapt the python implementation to match this one. All BytestrPicklerTests in test_bytestrpickle.py pass, and ./python -m test -G -v test_pickle passes. Comments on style etc. are very welcome. -- Added file: http://bugs.python.org/file24640/BytestrPickler_c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Éric Araujo mer...@netwok.org: -- versions: +Python 3.3 -Python 2.6, Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Ronny Pfannschmidt ronny.pfannschm...@gmail.com added the comment: im unlikely to find the time to try and fix pickle/cpickle myself in the next few months -- nosy: +Ronny.Pfannschmidt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: Last night, I hacked together a wrapper to do what loewis suggested [1]. It pickles bytes to str (for protocol = 2), and unpickles str to bytes. If I (ever) get the build system and tests of python itself to work, I'll try and see if I can implement a nicer solution - at least for pickle.py. [1] https://github.com/valhallasw/py2/blob/master/bytestrpickle.py -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Antoine Pitrou pit...@free.fr added the comment: If I (ever) get the build system and tests of python itself to work, If you have any problems with that, don't hesitate to ask on python-dev (or see http://mail.python.org/mailman/listinfo/core-mentorship ) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: Added file: http://bugs.python.org/file24567/test_bytestrpickle.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: OK, this is the pickle.py patch. A new parameter 'bytestr' has been added to both _Pickler and _Unpickler to toggle the pickle.string=bytes behaviour: _Pickler: IF protocol = 2 AND bytestr=True THEN bytes are stored as STRING/SHORT_BINSTRING/BINSTRING ELSE (the old behaviour; obj for protocol =2, else BINARY) _Unpickler: IF bytestr=True THEN STRING/SHORT_BINSTRING/BINSTRING are read as bytes ELSE they are read as str (old behaviour) I also extracted the decoding stuff from the three string reading functions to a single one. -- keywords: +patch Added file: http://bugs.python.org/file24568/pickle.py.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: P.S. (sorry for forgetting this in the original post ;-)) Both ./python -m test -G -v test_pickle and ./python test_bytestrpickle.py pass, but I have not run the entire test suite, as that takes ~90 minutes on my laptop The test script should of course be merged with test_pickle.py at some time -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Merlijn van Deen valhall...@gmail.com added the comment: Any news on this? Just as a note, pickletools.py also does not reflect the current behaviour; pickle types STRING, BINSTRING and SHORT_BINSTRING are all defined with stack_after=[pystring]: [1, line 992] I(name='STRING', code='S', arg=stringnl, stack_before=[], stack_after=[pystring], proto=0, doc=(...) ) although the doc=... does describe it will be decoded, the object type of pystring is still defined as bytes: [1, line 747] pystring = StackObject( name='string', obtype=bytes, doc=A Python (8-bit) string object.) [1] http://hg.python.org/cpython/file/98df29d51e12/Lib/pickletools.py -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Merlijn van Deen valhall...@gmail.com: -- nosy: +valhallasw ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +jdharper ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
RonnyPfannschmidt ronny.pfannschm...@gmx.de added the comment: i'll try to add some tests now hopefully i can get rid of the implicit badness like trying to coerce bytes to unicode in unpickle and storing bytes as list in pickle for protocol 3 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Gabriel Genellina gagsl-...@yahoo.com.ar added the comment: Note that this is also a documentation issue: The pickle serialization format is guaranteed to be backwards compatible across Python releases. -- assignee: - georg.brandl components: +Documentation nosy: +gagenellina, georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6784] byte/unicode pickle incompatibilities between python2 and python3
Changes by RonnyPfannschmidt ronny.pfannschm...@gmx.de: -- title: byte/unicode pickle incompatibilities between python2 andand python3 - byte/unicode pickle incompatibilities between python2 and python3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6784 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
