[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This is new feature in 3.5.

--
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Jason Curtis]

Jason Curtis added the comment:

This is still an issue with operator.attrgetter in 3.4.3, even after clearing 
sys.modules['_operator']:

$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['_operator'] = None
>>> import operator
>>> import pickle
>>> pickle.loads(pickle.dumps(operator.attrgetter("foo")))
Traceback (most recent call last):
  File "", line 1, in 
_pickle.PicklingError: Can't pickle .func 
at 0x7f25728d5bf8>: attribute lookup func on operator failed

--
nosy: +Jason Curtis
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
resolution:  - fixed
stage: patch review - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2688655e431a by Serhiy Storchaka in branch 'default':
Issue #22955: Fixed reference leak in attrgetter.repr().
https://hg.python.org/cpython/rev/2688655e431a

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Roundup Robot]

Roundup Robot added the comment:

New changeset 435bc22f39e3 by Serhiy Storchaka in branch 'default':
Issue #22955: attrgetter, itemgetter and methodcaller objects in the operator
https://hg.python.org/cpython/rev/435bc22f39e3

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Roundup Robot]

Roundup Robot added the comment:

New changeset c93e5ba1cc20 by Serhiy Storchaka in branch 'default':
Issue #22955: Fixed test_operator. It left Python implementation in
https://hg.python.org/cpython/rev/c93e5ba1cc20

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

methodcaller with keyword arguments pickled with 
pickle_getter_and_caller3.patch needs Python 3.5 to unpickle. Following patch 
pickles it in backward compatible way.

--
Added file: http://bugs.python.org/file39395/pickle_getter_and_caller4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is revised Josh's patch. Added tests for consistency between both 
implementations, fixed inconsistencies and bugs.

I still hesitate about pickling format of methodcaller. First, there is 
asymmetry between positional and keyword arguments. Second, for now 
methodcaller is not inheritable, but if it will be in future (as 
functools.partial is), it would be harder to extend pickling format to support 
instance attributes.

--
stage: needs patch - patch review
Added file: http://bugs.python.org/file37448/pickle_getter_and_caller3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

 functools.partial is a somewhat less than ideal comparison.  The pure-Python 
 version is not picklable, the Python and C versions return different things 
 (the Python version is a function returning a function, the C version is a 
 regular class and returns an instance).

Looks as Python version of functools.partial() needs a fix.

Reimplementations of the pure-Python itemgetter and attrgetter to automatically 
pickleable Python classes have a disadvantage. It makes the pickling 
incompatible between Python and C versions. This means that itemgetter pickled 
in CPython will be not unpickleable on Python implementation which don't use C 
accelerator and vice versa.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Zachary Ware added the comment:

Serhiy Storchaka added the comment:
 Reimplementations of the pure-Python itemgetter and attrgetter to
 automatically pickleable Python classes have a disadvantage. It makes
 the pickling incompatible between Python and C versions. This means
 that itemgetter pickled in CPython will be not unpickleable on Python
 implementation which don't use C accelerator and vice versa.

That's a very good point that I hadn't thought about.  Consider my
patch withdrawn.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Zachary Ware added the comment:

Serhiy: functools.partial is a somewhat less than ideal comparison.  The 
pure-Python version is not picklable, the Python and C versions return 
different things (the Python version is a function returning a function, the C 
version is a regular class and returns an instance).  Also, both versions make 
their necessary attributes public anyway, unlike methodcaller.

Raymond: Not necessarily the usual approach, no.  However, I think my 
reimplementations of the pure-Python itemgetter and attrgetter have a few 
benefits, namely:
- they're somewhat less complex and thus a bit easier to understand
- they're slightly faster
- they don't require extra pickling methods, which to me just seem like clutter 
when it's so simple to not need them

Note that I have no intention of reimplementing the C versions: those are much 
more mature than the Python versions, and would likely require pickling methods 
anyway.

All that said, I'm not going to fight about it; if I'm overruled, I'm overruled.

Josh: Serhiy's points about needing more tests stand; would you like to add 
them?  You can use your patch or mine as a base, depending on how you feel 
about reimplementing the pure-Python (item|attr}getter.  If you use yours, 
please remember to look through my comments on it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Raymond Hettinger]

Raymond Hettinger added the comment:

 I'd prefer to just reimplement itemgetter and attrgetter to make 
 them picklable rather than adding pickling methods to them;
 see attached patch.

That isn't the usual approach.  The pickling methods are there for a reason.  I 
prefer to leave the existing code in a stable state and avoid unnecessary code 
churn or risk introducing bugs into code that is working correctly and as 
designed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Raymond Hettinger]

Raymond Hettinger added the comment:

Please remember that a potential new pickling feature is the least import part 
of the design of methodcaller, itemgetter, and attrgetter.  Pickle support 
should be driven by the design rather become a predominant consideration.

One other note:  the OP's original concern has very little to do with these 
particular objects.  Instead, it is the picking and unpickling tools themselves 
that tend to have crummy error messages when presented with objects that 
weren't specially designed with pickle support.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

 Instead, it is the picking and unpickling tools themselves that tend to have 
 crummy error messages when presented with objects that weren't specially 
 designed with pickle support.

See issue22995 about this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Zachary Ware added the comment:

I'd prefer to just reimplement itemgetter and attrgetter to make them picklable 
rather than adding pickling methods to them; see attached patch.

I also posted a few comments, but I just went ahead and addressed them myself 
in this patch.  I'm not qualified to give the _operator.c changes a proper 
review, but they look good enough to me if others agree that __reduce__ is the 
best approach in C.

--
Added file: http://bugs.python.org/file37319/issue22955.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Changes by Zachary Ware zachary.w...@gmail.com:


Removed file: http://bugs.python.org/file37319/issue22955.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Changes by Zachary Ware zachary.w...@gmail.com:


Added file: http://bugs.python.org/file37320/issue22955.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

operator.methodcaller is similar to functools.partial which is pickleable and 
can be used as a sample.

In C implementation some code can be shared between __repr__ and __reduce__ 
methods.

As for tests, different protocols should be tested. Also should be tested 
compatibility between C and Python implementations, instances pickled with one 
implementation should be unpickleable with other implementation. Move pickle 
tests into new test class.

If add __repr__ methods, they need tests. The restriction of method name type 
should be tested too.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Zachary Ware]

Zachary Ware added the comment:

Note that pickling of the pure Python version of methodcaller works as expected:

Python 3.4.2 (default, Nov 20 2014, 12:40:10) 
[GCC 4.8.3] on linux
Type help, copyright, credits or license for more information.
 import sys
 sys.modules['_operator'] = None
 import operator
 import pickle
 pickle.loads(pickle.dumps(operator.methodcaller('foo')))
operator.methodcaller object at 0x7ff869945898

The pure Python attrgetter and itemgetter don't work due to using functions 
defined in __init__().

2.7 already raises TypeError on attempts to pickle any of the three.

--
nosy: +zach.ware
title: Pickling of methodcaller and attrgetter - Pickling of methodcaller, 
attrgetter, and itemgetter
versions: +Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Raymond Hettinger]

Raymond Hettinger added the comment:

+1 for adding pickling support to Python 3.5.

I don't see much of a need for any revision to 3.4.

--
nosy: +rhettinger
versions:  -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Josh Rosenberg added the comment:

I've made a patch that I believe should cover all three cases, including tests.

In addition to the pickling behavior, I've made two other changes:

1. methodcaller verifies during construction that the name is a string 
(PyUnicode), and interns it; attrgetter did this already, and I tweaked 
methodcaller to match for correctness and performance reasons
2. I added proper repr functionality to all three objects. Partially this is 
just to make it look nicer, but it was also a decent way to spot verify that 
the pickle/unpickle sequence behaved correctly

Anyone care to review?

--
keywords: +patch
nosy: +josh.r
versions: +Python 3.4
Added file: http://bugs.python.org/file37313/pickle_getter_and_caller.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Changes by Josh Rosenberg shadowranger+pyt...@gmail.com:


--
versions:  -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Josh Rosenberg added the comment:

Don't bother reviewing just yet. There is an issue with attrgetter's pickling 
(which the unit tests caught), and I need to update the pure Python modules to 
match.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Josh Rosenberg added the comment:

Okay, this one passes the tests for the built-in module. I'm not sure what's 
going wrong with the pure Python module. I'm getting the error:

_pickle.PicklingError: Can't pickle class 'operator.attrgetter': it's not 
the same object as operator.attrgetter

once for each of the three objects. Anyone recognize this? Is this some weird 
artifact of the multiple imports required to test both pure Python and C 
versions of the module that I need to work around, or did I make a mistake 
somewhere else?

--
Added file: http://bugs.python.org/file37314/pickle_getter_and_caller.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Changes by Josh Rosenberg shadowranger+pyt...@gmail.com:


Removed file: http://bugs.python.org/file37313/pickle_getter_and_caller.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22955] Pickling of methodcaller, attrgetter, and itemgetter

[Josh Rosenberg]

Josh Rosenberg added the comment:

Ah, solved it (I think). The bootstrapper used to import the Python and C 
versions of the module leaves sys.modules unpopulated (Does pickle itself may 
populate it when it finds no module of that name?). I added a setUp method to 
the unittest class for operator that explicitly sets sys.modules['operator'] to 
whichever version is being tested at the time so pickle's lookup works as 
expected. Is that the right solution? New patch uploaded with that change.

--
Added file: http://bugs.python.org/file37315/pickle_getter_and_caller2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22955
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com