[issue21986] Pickleability of code objects is inconsistent

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

2. It is normal. The third argument of copyreg.pickle() is not used now.

The patch LGTM.

--
title: Idle: disable pickleability of user code objects -> Pickleability of 
code objects is inconsistent

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Terry J. Reedy]

Terry J. Reedy added the comment:

Here are the issues for me.

1. Except for your interest, this is a lower priority for me that most all of 
the other 125 Idle issues.

2. I am not familiar with pickling and wonder about the following in the patch. 
It deletes a statememt
copyreg.pickle(types.CodeType, pickle_code, unpickle_code)
that registers two functions.  But the new code only registers pickle_code.  
Will
 message = pickle.loads(packet)
then work?  Maybe, since pickle_code has this.
 return unpickle_code, (ms,)
Is this normal?  Is registring unpickle_code not needed?  Would it make any 
sense to wrap code objects in a CodePickler class with __getstate__ and 
__setstate__ methods?

3. The interprocess communication implemented in rpc.py is central to Idle's 
execution of user code.  So I would want to be confident that a patch does not 
break it.  What I would like is a test script (executed by hand is ok) that 
works now, fails with the deletion of the copyreg statement, and works again 
with the rest of the patch.

4. I have no idea if or from where a code object is sent.  One way to search 
would be to grep idlelib for rpc calls.  Another would be to add a print 
statement to pickle_code, start Idle from the console with 'python -m idlelib', 
and run through all the menu options that involve the user process while 
watching the console for messages.

So you could try one of the searches methods to find a test.  Or you can wait 
for someone who currently understands pickle well enough to review and apply 
the patches without a test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Claudiu Popa added the comment:

Terry, can I do something to move this issue forward?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Changes by Claudiu Popa :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Claudiu Popa added the comment:

TypeError is raised only in Python 2, in Python 3 it's PicklingError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Terry J. Reedy]

Terry J. Reedy added the comment:

ppperry: Component Windows (or Macintosh) means Windows (or Mac) specific.  The 
rpc code is general to all systems.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Terry J. Reedy]

Terry J. Reedy added the comment:

There is no unittest module for rpc yet.

Should the pickle test include a test that pickling a code object fails, and 
with the proper exception?  Is ppperry correct about PicklingError?

Chainmap is new in 3.3 so 2.7 would need the copy version of the patch.

--
components: +IDLE -Library (Lib), Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Claudiu Popa added the comment:

Thanks, Serhiy.

--
Added file: http://bugs.python.org/file35969/issue21986.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Instead of copying dispatch_table, use ChainMap.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Claudiu Popa added the comment:

Maybe something like the attached patch. It doesn't have tests, though, I 
didn't find any tests for the idlelib.rpc anyway.

--
keywords: +patch
Added file: http://bugs.python.org/file35968/idlelib.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[ppperry]

Changes by ppperry :


--
components: +Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Martin v . Löwis]

Martin v. Löwis added the comment:

I agree with Claudiu. IDLE should pickle with a private dispatch_table.

--
nosy: +loewis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Claudiu Popa]

Claudiu Popa added the comment:

It works in IDLE because it registers a custom pickling for code objects, in 
idlelib.rpc:

  copyreg.pickle(types.CodeType, pickle_code, unpickle_code)


where pickle_code / unpickle_code calls marshal.dumps/loads.

Although, I admit that this is weird. If idlelib.rpc is using this for 
transferring data between RPC instances, that's okay, but leaking the behaviour 
in the IDLE's interactive interpreter is not that okay, because leads to 
different results and expectancies between IDLE and Python's interactive 
interpreter.

--
nosy: +Claudiu.Popa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Terry J. Reedy]

Terry J. Reedy added the comment:

Code is really a code object, so the compile does not seem to be the problem. 
In 2.7, on Win7, I get exactly the same output after removing the possibly 
spurious space in the string posted.  (ppperry), what system (OS) are you 
using. (In the future, please report both system and Python version).

3.4 gives a mild variation
b'\x80\x03cidlelib.rpc\nunpickle_code\nq\x00CZ\xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00@\x00\x00\x00s\x08\x00\x00\x00e\x00\x00\x01d\x00\x00S)\x01N)\x01Z\ndummy_code\xa9\x00r\x01\x00\x00\x00r\x01\x00\x00\x00z\x06\xda\x08\x01\x00\x00\x00s\x00\x00\x00\x00q\x01\x85q\x02Rq\x03.'

I ran both test_pickle and test_pickletools from both the command line 
>python -m test -v test_pickletools
and from an Idle editor with F5, with no errors and apparently identical output.

import pickle
code = compile("dummy_code", "this is a file name", "exec")
print(type(code))
print('pickle: ', pickle.dumps(code))
>>> produces

pickle:  b'\x80\x03cidlelib.rpc ... \x13this is a file name\...'

which shows that the bytes come from picke, not from a garbled traceback.

Since the bytes look like an actual pickle, I tried unpickling and it works:

import pickle
code = compile("'a string'", "", "eval")
pick = pickle.dumps(code)
code2 = pickle.loads(pick)
print(eval(code), eval(code2))
>>>
a string a string

import pickle
code1 = compile("print('a string')", "", "exec")
pick = pickle.dumps(code1)
code2 = pickle.loads(pick)
exec(code1)
exec(code2)
>>> 
a string
a string

I do not see any Idle bug here, so this might be closed unless you want to make 
this a pickle enhancement issue to work on code objects in the standard 
interpreter. The 3.4 interpreter traceback gives more info on why the pickle 
fails there.

>>> code0 = compile("'abc'", '', 'eval')
>>> pick = pickle.dumps(code0)
Traceback (most recent call last):
  File "", line 1, in 
_pickle.PicklingError: Can't pickle : attribute lookup code on 
builtins failed

In Idle, __builtins__.code fails, just as in the interpreter, so pickle is 
doing something different, and more successful, in the slightly altered Idle 
user-process execution environment.

--
components: +Library (Lib) -IDLE
nosy: +pitrou
stage:  -> needs patch
versions: +Python 2.7, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +kbk, roger.serwy, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[ppperry]

Changes by ppperry :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21986] Pickleability of code objects is inconsistent

[ppperry]

New submission from ppperry:

In IDLE:

>>> code = compile("dummy_code", "", "exec")
>>> pickle.dumps(code)
"cidlelib.rpc\nunpickle_code\np0\n(S'c\\x00\\x00\\x00\\x00\\x00\\x00 
\\x00\\x00\\x01\\x00\\x00\\x00@\\x00\\x00\\x00s\\x08\\x00\\x00\\x00e\\x00\\x00\\x01d\\x00\\x00S(\\x01\\x00\\x00\\x00N(\\x01\\x00\\x00\\x00t\\n\\x00\\x00\\x00dummy_code(\\x00\\x00\\x00\\x00(\\x00\\x00\\x00\\x00(\\x00\\x00\\x00\\x00s\\x06\\x00\\x00\\x00t\\x08\\x00\\x00\\x00\\x01\\x00\\x00\\x00s\\x00\\x00\\x00\\x00'\np1\ntp2\nRp3\n."


Outside of IDLE:
>>> code = compile("dummy_code", "", "exec")
>>> pickle.dumps(code)
Traceback (most recent call last):
  File "", line 1, in 
  File "C:\Python27\lib\pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
  File "C:\Python27\lib\pickle.py", line 306, in save
rv = reduce(self.proto)
  File "C:\Python27\lib\copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle code objects



Also, the error probably should be a pickle.PicklingError, not a TypeError.

--
components: IDLE
messages: 223161
nosy: ppperry
priority: normal
severity: normal
status: open
title: Pickleability of code objects is inconsistent

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com