[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

On Tue, Jan 8, 2013 at 2:51 PM, Daniel Shahaf wrote:

>
> Daniel Shahaf added the comment:
>
> Eli Bendersky wrote on Tue, Jan 08, 2013 at 15:00:42 +:
> >
> > Eli Bendersky added the comment:
> >
> > P = import_fresh_module('xml.etree.ElementTree',
> blocked=['_elementtree'])
> > tb = P.TreeBuilder(element_factory=lambda a, b: [a, b])
> > print(pickle.dumps(tb))
> >
> > Gives: _pickle.PicklingError: Can't pickle : attribute
> lookup builtins.function failed
>
> Is that with or without the patch?
>

Without. Pickle can't handle functions that are not top-level, so bound
methods, internal functions, lambdas are all out. So pickling probably
wasn't a first priority for the Python version of TreeBuilder as well...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Eli Bendersky wrote on Tue, Jan 08, 2013 at 15:00:42 +:
> 
> Eli Bendersky added the comment:
> 
> P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
> tb = P.TreeBuilder(element_factory=lambda a, b: [a, b])
> print(pickle.dumps(tb))
> 
> Gives: _pickle.PicklingError: Can't pickle : attribute 
> lookup builtins.function failed

Is that with or without the patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
tb = P.TreeBuilder(element_factory=lambda a, b: [a, b])
print(pickle.dumps(tb))

Gives: _pickle.PicklingError: Can't pickle : attribute lookup 
builtins.function failed

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Daniel, thanks for this patch, it looks very good. I had some comments in the 
code review.

As for TreeBuilder, let's split it into a separate issue - I think it's much 
less critical. We can discuss it later once we're done with Element. If 
anything, it's more important to handle ElementTree itself first - it should be 
very easy (much simpler than Element).

With the factory method I'm not sure how the Py pickling happens. I need to dig 
a bit in pickling for that, because I don't know how method reference pickling 
works.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

New iteration.

Open issues:

- Share code with the init method.  The issue with sharing code with either 
element_init() or create_new_element() would be malloc+realloc: unlike either 
of these methods, we know both the attributes and the number of children at 
allocation time, so we can allocate directly the right number of children.

- C<->Py Interchangeable pickling of TreeBuilder (per above msg).

Differences to previous version:

- Skip C<->Py interchangeability testing of TreeBuilder --- because that one 
started failing with:
  _pickle.UnpicklingError: state is not a dictionary
  It can probably be fixed, but I'd like to address the above question about 
pickling the factory first.

- Use __getstate__ rather than __reduce__ for Element.

- Make Element pickling interchangeable between c/py Element
  (set tp_name to "xml.etree.ElementTree.Element" and match the
  pickled format).

--
Added file: http://bugs.python.org/file28627/i16076-v7.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

In the meantime, unrelated question: should TreeBuilder be pickleable too, and 
interchangeably (pickle|unpickle)able between the Python and C implementations? 
 Asking because the Python TreeBuilder has a _factory member which it 
initializes to 'Element', and I'm not quite sure how cross-pickling should 
handle that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

I think that changing the _elementtree's Element's name to match the Python 
version, and then making sure the same serialized object is returned - is a 
worthy option to try.

Then it will hopefully "just work". When pickle deserializes a user-defined 
object that says it's a xml.etree.ElementTree.Element, it will try to import 
Element from xml.etree.ElementTree and should get the accelerated class. On 
machines without acceleration it will get the Python class.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Eli Bendersky wrote on Thu, Jan 03, 2013 at 14:44:02 +:
> If this change is required (even if we choose to name it
> "xml.etree.ElementTree.Element" for Py compatibility to fix the pickling
> regression, we may find ourselves in a need to change it between 3.3 and
> 3.3.1 and I'm not sure if that's valid. I hope my question on pydev will be
> resolved conclusively.
> 
> Danial, could you investigate if such a change is absolutely required to
> make pickling/unickling of Element work?

There are a few options:

- Change c-Element's tp_name to "xml.etree.ElementTree.Element".

- Register a reduce function for c-Element's that serialises them by
  constructing an equivalent py-Element and returning the latter's
  .__dict__ via the third return value:
  http://docs.python.org/3/library/copyreg#copyreg.pickle

- Add an entry mapping "builtins.Element" to
  "xml.etree.ElementTree.Element" to _compat_pickle.IMPORT_MAPPING
  (which is used by Lib/pickle.py:_Pickler.find_class()).

I haven't tried to implement either of these approaches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

On Jan 4, 2013 2:09 AM, "Stefan Krah"  wrote:
>
>
> Stefan Krah added the comment:
>
> Daniel Shahaf  wrote:
> > Is there a requirement that it loads a particular module?  Would etree
> > users notice the difference if pickle.load() returns an instance of the
> > "other" Element implementation than the one they pickled?
>
> Yes: If you send an "_elementtree.Element" pickle to another machine that
> doesn't have a working C version, it can't be unpickled. As far as I know
> the only way around that is to pickle as "xml.etree.ElementTree.Element".
>
> --
>

Yes this is a good point. Another thing to consider is that if both report
the same name then it will be possible to unpickle on a machine running 3.2

> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Stefan Krah]

Stefan Krah added the comment:

Daniel Shahaf  wrote:
> Is there a requirement that it loads a particular module?  Would etree
> users notice the difference if pickle.load() returns an instance of the
> "other" Element implementation than the one they pickled?

Yes: If you send an "_elementtree.Element" pickle to another machine that
doesn't have a working C version, it can't be unpickled. As far as I know
the only way around that is to pickle as "xml.etree.ElementTree.Element".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Eli Bendersky wrote on Thu, Jan 03, 2013 at 14:44:02 +:
> On Tue, Jan 1, 2013 at 2:56 PM, Daniel Shahaf wrote:
> > I added the "_elementtree" to the tp_name in order to bypass the above
> > error.  Module-qualified names were in use elsewhere (including by
> > _elementtree._element_iterator) so it seemed reasonable.  I'll defer to
> > you about compatibility implications of this change.
> 
> I asked on pydev, but this is a key point to resolve. Can
> pickling/unpickling be made to work correctly without such (or similar)
> change at all? How will the unpickler know which module to load when it
> sees a pickled object of Element type?

Is there a requirement that it loads a particular module?  Would etree
users notice the difference if pickle.load() returns an instance of the
"other" Element implementation than the one they pickled?

> Danial, could you investigate if such a change is absolutely required to
> make pickling/unickling of Element work?

Yes, I'll look into that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

On Tue, Jan 1, 2013 at 2:56 PM, Daniel Shahaf wrote:

>
> Daniel Shahaf added the comment:
>
> Eli Bendersky wrote on Tue, Jan 01, 2013 at 15:54:00 +:
> > Why did you change the class name, by the way, I don't think it's
> > a valid change at least for 3.3 in terms of backwards compatibility.
> >
>
> With unmodified tip of 3.4:
>
> >>> import pickle, xml.etree.ElementTree as ET
> >>> pickle.dumps(ET.Element('foo'))
> Traceback (most recent call last):
>   File "", line 1, in 
> _pickle.PicklingError: Can't pickle : attribute
> lookup builtins.Element failed
>
> I added the "_elementtree" to the tp_name in order to bypass the above
> error.  Module-qualified names were in use elsewhere (including by
> _elementtree._element_iterator) so it seemed reasonable.  I'll defer to
> you about compatibility implications of this change.

I asked on pydev, but this is a key point to resolve. Can
pickling/unpickling be made to work correctly without such (or similar)
change at all? How will the unpickler know which module to load when it
sees a pickled object of Element type?

If this change is required (even if we choose to name it
"xml.etree.ElementTree.Element" for Py compatibility to fix the pickling
regression, we may find ourselves in a need to change it between 3.3 and
3.3.1 and I'm not sure if that's valid. I hope my question on pydev will be
resolved conclusively.

Danial, could you investigate if such a change is absolutely required to
make pickling/unickling of Element work?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Also, the class inheritance in the tests should be amended to follow #16835.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Eli Bendersky wrote on Tue, Jan 01, 2013 at 00:32:51 +:
> 1. Why did you choose to implement __reduce__ and not __getstate__?

Maybe I was simply imitating what some other extension module was doing ;)

By using __reduce__ with the type as first return value, the
__setstate__ code becomes simpler because it doesn't need to address
'tag' and 'attrib'.  This consideration is somewhat moot now that the
unpickle-Python-pickled-Element's code path has to think about all
instance attributes, including those settable by the constructor.

> 2. A lot of code appears to be shared between
> element_setstate_from_attributes and the init method implementation,
> can it be refactored?

It seems there might be room for code reuse --- both functions set
tag,text,tail,attrib (but the unpickler sets the children too).  I'll
look into that in the next iteration (once the class name and pickle
output format issues are settled).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Eli Bendersky wrote on Tue, Jan 01, 2013 at 15:54:00 +:
> Why did you change the class name, by the way, I don't think it's
> a valid change at least for 3.3 in terms of backwards compatibility.
> 

With unmodified tip of 3.4:

>>> import pickle, xml.etree.ElementTree as ET
>>> pickle.dumps(ET.Element('foo'))
Traceback (most recent call last):
  File "", line 1, in 
_pickle.PicklingError: Can't pickle : attribute lookup 
builtins.Element failed

I added the "_elementtree" to the tp_name in order to bypass the above
error.  Module-qualified names were in use elsewhere (including by
_elementtree._element_iterator) so it seemed reasonable.  I'll defer to
you about compatibility implications of this change.

> Regarding that compatibility, and even easier idea would be for the
> C pickle to return the same __dict__ implicitly gathered from the
> Python version, and then only one version of the unpickle is required.

That makes sense.  But going forward it might be even better to define
an explicit __reduce__/__getstate__ for the Python version too, so if
the instance dict grows new members, they won't get serialised
unintentionally.  (This consideration is also why C code in the latest
patch makes some effort to notice unknown args in the dict.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Thank you for the input Stefan. I was actually glancing at _decimal as an 
example of implementing pickling and inter-compatibility between the C and Py 
versions of pickles.

You've chosen compatibility by having the same class name and __reduce__ 
returning the exact same tuple for both implementation. This is a pretty good 
idea.

Unfortunately, in the case of Element it's more difficult, because the existing 
Py implementation does not have __reduce__, so pickle does its thing by looking 
at __dict__. Changing the Py version to have a __reduce__ seems risky as it may 
break compatibility between the different Py versions of Element (say, from 
3.2) and this is bad.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Stefan Krah]

Stefan Krah added the comment:

Yes, currently the C version is also used for unpickling. Actually
this problem was one of the reasons why _decimal sets its name to
"decimal".
 


from test.support import import_fresh_module
import pickle, sys

C = import_fresh_module('xml.etree.ElementTree', fresh=['_elementtree'])
P = import_fresh_module('xml.etree.ElementTree', blocked=['_elementtree'])
e = C.Element('foo', bar=42)
e.text = "text goes here"
e.tail = "opposite of head"
C.SubElement(e, 'child').append(C.Element('grandchild'))
e.append(C.Element('child'))
e.findall('.//grandchild')[0].set('attr', 'other value')
sys.modules['xml.etree.ElementTree'] = C 
s = pickle.dumps(e)
s

b'\x80\x03c_elementtree\nElement\nq\x00X\x03\x00\x00\x00fooq\x01}q\x02X\x03\x00\x00\x00barq\x03K*s\x86q\x04Rq\x05X
\x0e\x00\x00\x00text goes hereq\x06X\x10\x00\x00\x00opposite of 
headq\x07h\x00X\x05\x00\x00\x00childq\x08\x85q\tRq
\nNNh\x00X\n\x00\x00\x00grandchildq\x0b}q\x0cX\x04\x00\x00\x00attrq\rX\x0b\x00\x00\x00other
 valueq\x0es\x86q\x0fRq
\x10NNN\x87q\x11b\x85q\x12\x87q\x13bh\x00h\x08\x85q\x14Rq\x15NNN\x87q\x16b\x86q\x17\x87q\x18b.'

sys.modules['xml.etree.ElementTree'] = P
x = pickle.loads(s)
type(x)


--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Other thoughts.

I'm not sure why you're surprised the C->Python pickle/unpickle works. You've 
changed the type name from Element to _elementtree.Element, so I would guess 
Python always uses the C version to unpickle as well. Can you debug to verify 
what actually goes on under the hood? Why did you change the class name, by the 
way, I don't think it's a valid change at least for 3.3 in terms of backwards 
compatibility.

Regarding that compatibility, and even easier idea would be for the C pickle to 
return the same __dict__ implicitly gathered from the Python version, and then 
only one version of the unpickle is required.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

A couple of high-level questions about the C code (this isn't a detailed review 
yet):

1. Why did you choose to implement __reduce__ and not __getstate__?
2. A lot of code appears to be shared between element_setstate_from_attributes 
and the init method implementation, can it be refactored?
3. I've been thinking about the C<->Python pickle compatibility, and it may be 
too much trouble at this point for no very strong benefit. What could make more 
sense at some point is implementing a __getstate__/__setstate__ in both the 
Python and C versions that are compatible (i.e. produce the same things). But 
this may mean changing the way the Python Element is pickled in 3.3, which may 
not be acceptable. So we may prefer to forego the compatibility in 3.3 and then 
for 3.4 make the change. I'm still not sure what is best here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

BTW, maybe I'm just confused, but I'm a little surprised that C pickling -> 
Python unpickling works: the C __reduce__ returns a 3-element tuple, which 
according to pickle docs[1] must either be a dictionary or be passed to 
__setstate__; it's not a dictionary and the Python implementation doesn't have 
a __setstate__.

http://docs.python.org/3/library/pickle#pickle.object.__reduce__

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Attached.  Differences to previous version:

- Avoid the test___all__ issue (#16817) by manipulating sys.modules
  directly

- Support pickling interoperability between the C and Python
  implementations

--
Added file: http://bugs.python.org/file28502/i16076-v6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

I've added some (currently pyET specific) pickling tests. Daniel, could you 
re-generate the patch? Note that for the C version of pickling you can now 
enable the pickle test. Point to consider - can elements pickled in C be 
unpickled in Python and vice versa? It should probably work (see test_decimal's 
pickling tests).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Roundup Robot]

Roundup Robot added the comment:

New changeset 71508fc738bb by Eli Bendersky in branch '3.3':
For Issue #16076: make sure that pickling of Element objects is tested, and do
http://hg.python.org/cpython/rev/71508fc738bb

New changeset 5a38f4d7833c by Eli Bendersky in branch 'default':
For issue #16076: merge 3.3
http://hg.python.org/cpython/rev/5a38f4d7833c

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Florent Xicluna]

Changes by Florent Xicluna :


--
nosy: +flox

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

I think I understand what's going on there. Pickle, to make sure an object can 
be picked, looks at the module it comes from and tries to import its class, to 
make sure they're the same.

What happens since 3.3 is that for the Python Element class, it imports 
ElementTree. The harness in test_xml_etree carefully sets up to ignore 
_elementtree so the correct class is pulled. However, if test___all__ runs 
before it it just does a brute "from ElementTree import *" which places the C 
version in sys.modules, and this is what pickle finds.

So in the meantime, until issue 15083 is resolved I think it's safe to put ET 
in the ignore list of test___all__.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Any attempt to pickle an Element object in test_xml_etree.py causes that test 
to fail if test___all__ had been run before it; see attached transcript.  It's 
against 3.4, with no changes other than the testsuite changes shown within.

I don't know what the root cause is.  As noted before, adding 'xml.etree' to 
test___all__.AllTest.test_all.blacklist is a workaround, so I assume the root 
cause lies within the test framework --- a bad interaction between the import 
magics in test___all__ and test_xml_etree.

--
Added file: http://bugs.python.org/file28484/transcript-test___all__.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Also, could you explain what makes test___all__ start failing with this patch? 
What are you adding that makes that happen?

P.S. I suspect the root reason is the bad way etree tests are structured in 
general. See issue 15083

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

I wrote the patch against default (3.4), but it applies cleanly to 3.3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Daniel, is your patch made vs. the 3.3 branch? I'll need to apply there first, 
and then merge up to default (3.4).

[Also, removing the 3.2 tag here. 3.2 won't be fixed to make _elementtree 
pickleable - it never was].

--
versions:  -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Re the problem from msg 177134 ('./python -mtest test___all__ test_xml_etree' 
failing), it is a a preexisting problem, in the sense that applying just the 
Lib/test/ part of this patch suffices to trigger it.  Adding 'xml.etree' to the 
blacklist in test___all__.py:50 suffices to address the problem.  At a guess 
that is because when test___all__ imports the 'xml.etree' module, it doesn't 
filter out the '_elementtree' symbol like test_xml_etree.py:test_main() does.

New patch attached; differences to previous version:

- Removed debug prints.
- Skip test_pickling when testing the Python xml.etree.
  (The skips could be removed in the future, if/when the test___all__
  interaction problem is resolved.)

--
Added file: http://bugs.python.org/file28363/i16076-v5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Haven't had a chance to look at the test___all__ -related failures yet.  Still 
planning to do so once I have some time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

@Eli, thanks.  In the meantime, attaching a patch addressing Ezio's review; the 
caveats from msg 177134 still apply.

--
Added file: http://bugs.python.org/file28261/i16076-v4-combined.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Eli Bendersky]

Eli Bendersky added the comment:

Thanks for working on it, Daniel. Unfortunately I won't have time to look at it 
in the near future, but I will definitely look at the patch once I get some 
free time to hack on Python again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +georg.brandl, larry
priority: high -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Changes by Daniel Shahaf :


Removed file: http://bugs.python.org/file28251/i16076-v2-combined.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

Reattaching without the unrelated Python-ast.c change.

--
Added file: http://bugs.python.org/file28252/i16076-v3-combined.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16076] xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no longer pickleable

[Daniel Shahaf]

Daniel Shahaf added the comment:

TreeBuilder was also pickleable in 3.2 but now isn't so; updating summary 
accordingly.

Attaching a checkpoint patch.  It addresses both Element and TreeBuilder, and 
adds tests.  The added tests fail if test___all__ is included in the test run, 
so the patch should not be applied as-is.

Thanks to Ezio Melotti for preliminary review and suggestions on IRC.

--
title: xml.etree.ElementTree.Element is no longer pickleable -> 
xml.etree.ElementTree.Element and xml.etree.ElementTree.TreeBuilder are no 
longer pickleable
Added file: http://bugs.python.org/file28251/i16076-v2-combined.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com