[issue3657] pickle can pickle the wrong function
Alexandre Vassalotti added the comment: This was fixed in 3.4 with the introduction of method pickling. I don't think it would be appropriate to backport this to 2.7. Thus, I am closing this as a won't fix for 2.x. -- assignee: - alexandre.vassalotti resolution: - wont fix stage: needs patch - committed/rejected status: open - closed versions: -Python 3.1, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
R. David Murray rdmur...@bitdance.com added the comment: The randomly failing tests seem to have been the high priority issue. The remaining, eponymous issue seems to be of rather lower priority, so I'm setting it to normal. Although Tim wanted a separate issue for the pickling problem, I think there's too much useful info about the underlying problem in this issue for it to make sense to open a new one. -- assignee: nnorwitz - nosy: +r.david.murray priority: high - normal stage: - needs patch type: - behavior versions: +Python 3.1, Python 3.2 -Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Alexander Belopolsky belopol...@users.sourceforge.net added the comment: Revision 68903 was merged in py3k in r68908. It looks like a similar issue shows up in test_random: == ERROR: test_pickling (test.test_random.MersenneTwister_TestBasicOps) -- Traceback (most recent call last): File /home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_random.py, line 107, in test_pickling state = pickle.dumps(self.gen) File /home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/pickle.py, line 1358, in dumps Pickler(f, protocol, fix_imports=fix_imports).dump(obj) _pickle.PicklingError: Can't pickle class 'random.Random': it's not the same object as random.Random See http://www.python.org/dev/buildbot/all/builders/amd64%20gentoo%203.1/builds/819/steps/test/logs/stdio -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Antoine Pitrou pit...@free.fr added the comment: Revision 68903 was merged in py3k in r68908. It looks like a similar issue shows up in test_random: == ERROR: test_pickling (test.test_random.MersenneTwister_TestBasicOps) -- Traceback (most recent call last): File /home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_random.py, line 107, in test_pickling state = pickle.dumps(self.gen) File /home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/pickle.py, line 1358, in dumps Pickler(f, protocol, fix_imports=fix_imports).dump(obj) _pickle.PicklingError: Can't pickle class 'random.Random': it's not the same object as random.Random Actually, this might have to do with the fix I committed to test_threaded_import in r82885. In order for test_threaded_import to work, we have to unload the Guinea pig module before importing it from several threads at once. For whatever reason, test_threaded_import uses random as its Guinea pig module, which means random gets unloaded and reimported again. But at this point, I must admit I don't even understand the failure message. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Antoine Pitrou pit...@free.fr added the comment: It should be noted that, contrary to Amaury's suggestion, pickling random.seed fails under 3.x: pickle.dumps(random.seed) Traceback (most recent call last): File stdin, line 1, in module File /home/antoine/py3k/__svn__/Lib/pickle.py, line 1314, in dumps Pickler(f, protocol, fix_imports=fix_imports).dump(obj) _pickle.PicklingError: Can't pickle class 'method': attribute lookup builtins.method failed Furthermore, the original problem can also be reproduced under 3.x, using Amaury's trick: pickle.dumps(random.random) b'\x80\x03crandom\nrandom\nq\x00.' list(sys.modules.values())[0].random = random.random pickle.dumps(random.random) b'\x80\x03cheapq\nrandom\nq\x00.' I think a possible heuristic in whichmodule() would be, if __module__ is not found or None, to look for a __module__ attribute on __self__: random.random.__module__ random.random.__self__.__module__ 'random' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Mark Dickinson dicki...@gmail.com added the comment: Antoine's fix in r82919 / r82920 fixes the test_random failure for me. (Before the fix, ./python.exe ./Lib/test/regrtest.py test___all__ test_threaded_import test_random was enough to produce the failure.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: -tjreedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Terry J. Reedy tjre...@udel.edu added the comment: 'works for me' contradicts 'open', so I unset that -- nosy: +tjreedy resolution: works for me - versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Changes by Mark Dickinson dicki...@gmail.com: -- nosy: +marketdickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Mark Dickinson dicki...@gmail.com added the comment: I just committed Tim's suggested change in r68906. This seemed a no- brainer, regardless of what should be done about pickle.whichmodule. One fewer sporadic buildbot failure sounds like a good thing to me. (I hadn't noticed the pickletools failures until just after I committed a pickle module change, so I was rather thrown until Antoine directed me here...) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Antoine Pitrou [EMAIL PROTECTED] added the comment: I expect this is why nobody has ever complained about it: unless you're looking at the strings embedded in the pickle GLOBAL opcode, it's unlikely to have a visible consequence. Well, it may have a consequence if pickle picks the random function from a third-party module named foobar, and you give the pickle to a friend and expect it to work for him while he hasn't installed module foobar. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment: I'm not sure that pickling random.random is a good idea; did you try to pickle the random.seed function? Their definition look very similar (at the end of random.py: _inst = Random() seed = _inst.seed random = _inst.random ) but Random.seed is a python method, whereas Random.random is inherited from _randommodule.c. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment: The explanation is actually simple, do not blame bsddb :-) random.random is a built-in method, so its __module__ attribute is not set. To find it, pickle.whichmodule loops through all sys.modules, and pick the first one that contains the function. I reproduced the problem with a simple file: # someModule.py from random import random then, start python and: import someModule, random, pickle pickle.dumps(random.random, 0) 'csomeModule\nrandom\np0\n.' You may have to change the name of someModule, to be sure that it appears before random in sys.modules. Tested on Windows and x86_64 Linux. To correct this, one direction would be to search only built-in or extension modules; for bound methods (random.random.__self__ is not None), try to dump separately the instance and the method name (but getattr(random._inst, 'random') is not random.random). Or simply change the test... -- nosy: +amaury.forgeotdarc ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Tim Peters [EMAIL PROTECTED] added the comment: No thought went into picking random.random in the test -- it was just a random ;-) choice. Amaury's analysis of the source of non-determinism is on target, and the easiest fix is to pick a coded-in-Python function to pickle instead. I suggest, e.g., changing the sporadically failing doctest to: import pickletools dis(pickle.dumps(pickletools.dis, 0)) 0: cGLOBAL 'pickletools dis' 17: pPUT0 20: .STOP highest protocol among opcodes = 0 -- nosy: +tim_one ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Changes by Alexandre Vassalotti [EMAIL PROTECTED]: -- nosy: +alexandre.vassalotti ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Antoine Pitrou [EMAIL PROTECTED] added the comment: But it still means pickling a function/method defined in a builtin extension module can give wrong results, doesn't it deserve being fixed? -- nosy: +pitrou ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Tim Peters [EMAIL PROTECTED] added the comment: Amaury, yes, it would be nice if pickle were more reliable here. But that should be the topic of a different tracker issue. Despite the Title, this issue is really about sporadic pickletools.py failures, and the pickletools tests are trying to test pickletools, not pickle.py (or cPickle). Changing the test as suggested makes it reliably test what it's trying to test (namely that pickletools.dis() produces sensible output for pickle's GLOBAL opcode). Whether pickle/cPickle should do a better job of building GLOBAL opcodes in the first place is a distinct issue. In any case, since pickle/cPickle have worked this way forever, and the only known bad consequence to date is accidental sporadic test failures in pickletools.py, the underlying pickle/cPickle issue shouldn't be a release blocker regardless. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Changes by Barry A. Warsaw [EMAIL PROTECTED]: -- priority: release blocker - high ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Tim Peters [EMAIL PROTECTED] added the comment: BTW, note that the Title of this issue is misleading: pickle.whichmodule() uses object identity (is: if ... getattr(module, funcname, None) is func: ) to determine whether the given function object is supplied by a module, so it's /not/ the case that a wrong function can be pickled. The worst that can happen is that the correct function is pickled but obtained from a possibly surprising module. For example, random.random can't be confused with any other function named random. I expect this is why nobody has ever complained about it: unless you're looking at the strings embedded in the pickle GLOBAL opcode, it's unlikely to have a visible consequence. It would still be nice if pickle could identify the most natural module for a given function, but hard to make a case that doing so would be much more than /just/ nice. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Benjamin Peterson [EMAIL PROTECTED] added the comment: I can reproduce this on 2.6. -- nosy: +benjamin.peterson ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Changes by Barry A. Warsaw [EMAIL PROTECTED]: -- priority: deferred blocker - release blocker ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Barry A. Warsaw [EMAIL PROTECTED] added the comment: Neal, can you verify that this is still a problem now that bsddb has been removed? The tests, run in the order of the last comment, succeed for me in both 2.6 and 3.0, debug build or not, on both my single processor Ubuntu box and dual core Mac. -- nosy: +barry priority: release blocker - deferred blocker resolution: - works for me ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Neal Norwitz [EMAIL PROTECTED] added the comment: It seems that if the tests are run in this order: ./python -E -tt ./Lib/test/regrtest.py -u all test_xmlrpc test_ctypes test_json test_bsddb3 test_pickletools The error will trigger consistently. That is in 2.6 with a debug build on a dual cpu box. A debug build of 3.0 on the same machine did not fail though I don't know if 3.0 has this problem. I was unable to prune the list further. The 3 tests (xmlrpc, ctypes and json) can be run in any order prior to bsdb3 and then pickletools. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
New submission from Neal Norwitz [EMAIL PROTECTED]: test_pickletools fails sporadically on at least two platforms I've seen. http://www.python.org/dev/buildbot/all/x86%20gentoo%20trunk/builds/4120/step-test/0 http://www.python.org/dev/buildbot/all/ppc%20Debian%20unstable%20trunk/builds/1908/step-test/0 File /home/buildslave/python-trunk/trunk.norwitz-x86/build/Lib/pickletools.py, line ?, in pickletools.__test__.disassembler_test Failed example: dis(pickle.dumps(random.random, 0)) Expected: 0: cGLOBAL 'random random' 15: pPUT0 18: .STOP highest protocol among opcodes = 0 Got: 0: cGLOBAL 'bsddb.test.test_thread random' 31: pPUT0 34: .STOP highest protocol among opcodes = 0 ** 1 items had failures: 1 of 25 in pickletools.__test__.disassembler_test -- components: Interpreter Core messages: 71830 nosy: nnorwitz priority: release blocker severity: normal status: open title: pickle can pickle the wrong function versions: Python 2.6 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Neal Norwitz [EMAIL PROTECTED] added the comment: The valgrind errors below are possibly related. Conditional jump or move depends on uninitialised value(s) PyUnicodeUCS2_EncodeUTF8 (unicodeobject.c:2216) _PyUnicode_AsString (unicodeobject.c:1417) save (_pickle.c:930) Pickler_dump (_pickle.c:2292) Conditional jump or move depends on uninitialised value(s) PyUnicodeUCS2_EncodeUTF8 (unicodeobject.c:2220) _PyUnicode_AsString (unicodeobject.c:1417) save (_pickle.c:930) Pickler_dump (_pickle.c:2292) Conditional jump or move depends on uninitialised value(s) PyUnicodeUCS2_EncodeUTF8 (unicodeobject.c:2227) _PyUnicode_AsString (unicodeobject.c:1417) save (_pickle.c:930) Pickler_dump (_pickle.c:2292) Conditional jump or move depends on uninitialised value(s) PyUnicodeUCS2_EncodeUTF8 (unicodeobject.c:2229) _PyUnicode_AsString (unicodeobject.c:1417) save (_pickle.c:930) Pickler_dump (_pickle.c:2292) Conditional jump or move depends on uninitialised value(s) PyUnicodeUCS2_EncodeUTF8 (unicodeobject.c:2233) _PyUnicode_AsString (unicodeobject.c:1417) save (_pickle.c:930) Pickler_dump (_pickle.c:2292) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3657] pickle can pickle the wrong function
Neal Norwitz [EMAIL PROTECTED] added the comment: Indeed. The problem was an incorrect conversion of str - unicode, instead of converting to bytes. On getting the buffer from unicode, it tried to read data which was uninitialized. Hmmm, this fix is for 3.0 only, but the problem is happening in 2.6. Leaving open. Committed revision 66021. -- assignee: - nnorwitz ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com