[issue9935] Faster pickling of instances

2011-03-11 Thread Roundup Robot 

Roundup Robot devnull@devnull added the comment:

New changeset ff0220c9d213 by Antoine Pitrou in branch 'default':
Issue #9935: Speed up pickling of instances of user-defined classes.
http://hg.python.org/cpython/rev/ff0220c9d213

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2011-03-11 Thread Antoine Pitrou 

Changes by Antoine Pitrou pit...@free.fr:


--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-25 Thread Jesse Noller 

Jesse Noller jnol...@gmail.com added the comment:

I doubt I, or Ask will have the time to rewrite the entire multiprocessing test 
suite right now to work around the change Antoine.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-25 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

 I doubt I, or Ask will have the time to rewrite the entire
 multiprocessing test suite right now to work around the change
 Antoine.

Well, I'm not asking anyone to rewrite the entire multiprocessing test suite; 
and, besides, I've provided a patch myself to improve it in that respect ;) (in 
issue10173)

Of course, it also means the present pickle patch is imperfect, though the 
result of __reduce__ in this case looks more like a side-effect of an 
implementation detail than documented behaviour (since the result isn't usable 
anyway). I may try to come up with a better patch before the 3.2 beta but it's 
not sure I will find enough time/motivation.

--
resolution: fixed - 
stage: committed/rejected - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-25 Thread Jesse Noller 

Jesse Noller jnol...@gmail.com added the comment:

On Mon, Oct 25, 2010 at 10:17 AM, Antoine Pitrou rep...@bugs.python.org wrote:

 Well, I'm not asking anyone to rewrite the entire multiprocessing test suite; 
 and, besides, I've provided a patch myself to improve it in that respect ;) 
 (in issue10173)

I just saw that one - I'll poke at that next

 Of course, it also means the present pickle patch is imperfect, though the 
 result of __reduce__ in this case looks more like a side-effect of an 
 implementation detail than documented behaviour (since the result isn't 
 usable anyway). I may try to come up with a better patch before the 3.2 beta 
 but it's not sure I will find enough time/motivation.

Okie doke.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-22 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

Committed in r85797.

--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-22 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

The commit broke the Windows buildbots because (un)pickling a TextIOWrapper now 
raises an exception:

 f = open(LICENSE)
 pickle.dumps(f)
b'\x80\x03c_io\nTextIOWrapper\nq\x00)\x81q\x01}q\x02X\x04\x00\x00\x00modeq\x03X\x01\x00\x00\x00rq\x04sb.'
 g = pickle.loads(pickle.dumps(f))
Traceback (most recent call last):
  File stdin, line 1, in module
AttributeError: '_io.TextIOWrapper' object has no attribute '__dict__'


It should be noted that it didn't work before, but no exception was raised. The 
result was just nonsensical:

 f = open(LICENSE)
 pickle.dumps(f)
b'\x80\x03c_io\nTextIOWrapper\nq\x00)\x81q\x01.'
 g = pickle.loads(pickle.dumps(f))
 g
Traceback (most recent call last):
  File stdin, line 1, in module
ValueError: I/O operation on uninitialized object


The very fact that test_multiprocessing tries to pickle a file object is 
unfortunate, and is probably a bug in itself. test_multiprocessing is known for 
pickling lots of things, since it generally transfers a whole TestCase 
instance...

--
assignee:  - pitrou
nosy: +jnoller
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-22 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

The difference has to do with the result of __reduce__:

With the patch:

 open(LICENSE).__reduce_ex__(3)
(function __newobj__ at 0x7fa392a0ff30, (class '_io.TextIOWrapper',), 
{'mode': 'r'}, None, None)

Without:

 open(LICENSE).__reduce_ex__(3)
(function __newobj__ at 0x7f2cf2361a70, (class '_io.TextIOWrapper',), None, 
None, None)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-04 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

Alexandre, do you have opinion on this?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-10-04 Thread Alexandre Vassalotti 

Alexandre Vassalotti alexan...@peadrop.com added the comment:

Sorry Antoine, I have been busy with school work lately.

I like the general idea and I will try to look at your patch ASAP.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-09-28 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

My patch breaks pickling of transparent proxies such as weakref.proxy().
(since these have a different __class__ than Py_TYPE(self), through tp_getattr 
hackery). I will need to remove a couple of optimizations.

(unfortunately, there don't seem to be any tests for such case; my initial 
patch breaks neither test_pickle nor test_pickletools)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-09-28 Thread Antoine Pitrou 

Antoine Pitrou pit...@free.fr added the comment:

Corrected patch, including new tests for pickling of weak proxies.

--
Added file: http://bugs.python.org/file19044/pickleinst2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-09-24 Thread Antoine Pitrou 

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9935] Faster pickling of instances

2010-09-23 Thread Antoine Pitrou 

New submission from Antoine Pitrou pit...@free.fr:

This is a bunch of assorted optimizations which make pickling of user-defined 
classes quite a bit faster.

Example on a minimal instance:

$ python -m timeit -s import pickle; import collections, __main__; 
__main__.X=type('X', (), {}); x=X() pickle.dumps(x)
- before: 10 loops, best of 3: 8.11 usec per loop
- after: 10 loops, best of 3: 2.95 usec per loop

Example on a namedtuple:

$ python -m timeit -s import pickle; import collections, __main__; 
__main__.X=collections.namedtuple('X', 'a'); x=X(5) pickle.dumps(x)
- before: 10 loops, best of 3: 9.52 usec per loop
- after: 10 loops, best of 3: 3.78 usec per loop

Unladen Swallow's pickling benchmark:

### pickle ###
Min: 0.792903 - 0.704288: 1.1258x faster
Avg: 0.796241 - 0.706073: 1.1277x faster
Significant (t=39.374217)
Stddev: 0.00410 - 0.00307: 1.3342x smaller
Timeline: http://tinyurl.com/38elzvv

--
components: Library (Lib)
files: pickleinst.patch
keywords: patch
messages: 117253
nosy: alexandre.vassalotti, belopolsky, pitrou
priority: normal
severity: normal
stage: patch review
status: open
title: Faster pickling of instances
type: performance
versions: Python 3.2
Added file: http://bugs.python.org/file18986/pickleinst.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9935
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com