[issue19272] Can't pickle lambda (while named functions are ok)
Roundup Robot added the comment: New changeset d103ba56710e by Ethan Furman in branch 'default': Issue #19272: slight clarification of pickle docs with regard to lambda. http://hg.python.org/cpython/rev/d103ba56710e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: Added some clarification to the docs to make it clearer that lambda functions cannot be pickled. Facundo [1], if you want to pursue being able to pickle lambda functions please open an enhancement issue. Some of the questions that come to mind: 1) for a random name, where will the name be stored (I'm thinking a __pickle__ dict on the module)? 2) for pickling the code object itself, what are the ramifications (this is directly contrary to how pickle was designed) These questions should be discussed on PyDev, and will probably require a PEP. [1] or anyone else :) -- assignee: - docs@python nosy: +docs@python resolution: - invalid stage: - committed/rejected status: open - closed versions: -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Changes by Ethan Furman et...@stoneleaf.us: -- keywords: +patch Added file: http://bugs.python.org/file32155/issue19272.stoneleaf.01.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
New submission from Facundo Batista: This is ok: Python 3.4.0a3+ (default:86af5991c809, Oct 13 2013, 16:42:52) ... import pickle def f(): ... pass ... pickle.dumps(f) b'\x80\x03c__main__\nf\nq\x00.' However, when trying to pickle a lambda, it fails: pickle.dumps(lambda: None) Traceback (most recent call last): File stdin, line 1, in module _pickle.PicklingError: Can't pickle class 'function': attribute lookup builtins.function failed (Found this because I'm getting the same problem but when trying to use concurrent.futures.ProcessExecutor) -- components: Library (Lib) messages: 200062 nosy: facundobatista priority: normal severity: normal status: open title: Can't pickle lambda (while named functions are ok) type: behavior versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Amaury Forgeot d'Arc added the comment: Functions are pickled by name, not by code. Unpickling will only work if a function with the same name is present in in the same module (__main__ in your example) This is why pickling a lambda won't work: they have no individual names. -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Jesús Cea Avión added the comment: Would be interesting to be able to pickle function.__code__. Although, thinking about this, it would be not portable between Python releases, something that pickle guarantee. Thoughs? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: According to the docs[1]: 12.1.4. What can be pickled and unpickled? The following types can be pickled: - None, True, and False - integers, floating point numbers, complex numbers - strings, bytes, bytearrays - tuples, lists, sets, and dictionaries containing only picklable objects - functions defined at the top level of a module - built-in functions defined at the top level of a module - classes that are defined at the top level of a module - instances of such classes whose __dict__ or the result of calling __getstate__() is picklable Notice that lambda is not in that list. The docs for concurrent.futures.ProcessPoolExecutor[2] state: 17.4.3. ProcessPoolExecutor The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned. [1] http://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled [2] http://docs.python.org/dev/libraryconcurrent.futures.html?highlight=concurrent#processpoolexecutor -- nosy: +ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Facundo Batista added the comment: Ethan, lambda functions are included in functions defined at the top level of a module. Probably we should note there something like except lambdas, because function pickling is by name, not by code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Facundo Batista added the comment: Jesús, Amaury: What if pickle would assign a random unique name to the lambda (like, an UUID) so it can be pickled and unpickled? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: Yeah, that one line should say, named functions defined at the top level of a module. The following three paragraphs do, however, mention named several times. Sounds like a doc issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: The problem with a randam unique name is making sure you get the same random unique name across different runs, different pythons, and different orders of execution. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
R. David Murray added the comment: So don't make it random, use a hash of the code object :) (I'm not sure I'm serious, the thought just popped into my head...) -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Jesús Cea Avión added the comment: Lambdas are anonymous functions, by definition. And, usually, they are not top level functions, but defined inside others. If at your top level (module) you do: a = lambda x: 2*x You don't have an anonymous function, but a function called a. You can argue why def a() : return 2*x can be picked, but a = lambda x: 2*x can not. They look similar. What the poster actually wanted (tell me if I am wrong), I guess, is to be able to serialize the function code and send it to other process to be executed there. Something like mobile code. As is, pickle doesn't allow it (that is the reason I was brainstorming about being able to pickle function.__code__ objects), and it is good because pickle guarantees compatibilities between Python versions, but __code__ objects are particular to a certain Python virtual machine and certain version of it. That said, projects like PYRO provide mobile code. I think that a recipe for that would be nice in the documentation, with a big warning saying be sure you have the same implementation in both sides, and explicit versioning in the client and version checking in the server, in the recipe (to make it clear that same version is something you need). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: From the pickle docs: = Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. Similarly, classes are pickled by named reference, ... = There is no bug here; there /may/ be a doc clarification here. However, if we're talking about enhancing pickle we can take 3.3 off the table, and unless somebody gets busy quick 3.4 as well. Personally, I don't see the need. Just define your functions at the top level (with def, not lambda -- lambda functions are anonymous and don't have a useful __name__ attribute). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19272] Can't pickle lambda (while named functions are ok)
Ethan Furman added the comment: Jesús Cea Avión added the comment: If at your top level (module) you do: a = lambda x: 2*x You don't have an anonymous function, but a function called a. Actually, you do have an anonymous function, which happens to be bound to the name a. Compare: -- def a(): ... pass -- a.__name__ 'a' and -- a = lambda: None -- a.__name__ 'lambda' -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19272 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
