[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:


New changeset 63673916464bace8e2147357395fdf3497967ecb by Stefan Behnel (Miss 
Islington (bot)) in branch '3.8':
[3.8] bpo-34160: explain how to deal with attribute order in ElementTree 
(GH-14867) (GH-14935)
https://github.com/python/cpython/commit/63673916464bace8e2147357395fdf3497967ecb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[miss-islington]

Change by miss-islington :


--
pull_requests: +14708
pull_request: https://github.com/python/cpython/pull/14935

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:


New changeset a3697db0102b9b6747fe36009e42f9b08f0c1ea8 by Stefan Behnel in 
branch 'master':
bpo-34160: explain how to deal with attribute order in ElementTree (GH-14867)
https://github.com/python/cpython/commit/a3697db0102b9b6747fe36009e42f9b08f0c1ea8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

I created a PR that adds a couple of paragraphs to the documentation. Comments 
welcome.

--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Change by Stefan Behnel :


--
pull_requests: +14654
pull_request: https://github.com/python/cpython/pull/14867

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Tal Einat]

Change by Tal Einat :


--
nosy:  -taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Victor, you mean place again this code? 
https://github.com/python/cpython/pull/10163/files#diff-d5a064acb6ae44dcb7e01fee148c733dR926

And the recipe proposed in https://bugs.python.org/issue34160#msg338102 place 
it as a method of ElementTree or just as a helper function?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

Other examples of projects broken by this change:
* python-glyphsLib: https://bugzilla.redhat.com/show_bug.cgi?id=1705391
* pcs: https://bugzilla.redhat.com/show_bug.cgi?id=1705475

What's New In Python 3.8 only briefly mention this change:
"The writexml(), toxml() and toprettyxml() methods of the xml.dom.minidom 
module, and xml.etree now preserve the attribute order specified by the user. 
(Contributed by Diego Rojas and Raymond Hettinger in bpo-34160.)"
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-python-api

Would it be possible to suggest solutions for this backward incompatible 
change? Like https://bugs.python.org/issue34160#msg338102 recipe. Maybe we 
should even put it in the XML documentation, somewhere.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

I implemented (most of) C14N 2.0 in issue 13611. Please give it a try if you 
are interested in the canonical serialisation feature. I would like to include 
it in Py3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

I rejected the (now conflicting) PR that adds a sorting option.
I also sent Victor a tentative (and trivial) patch for the pungi package.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

PR 10452 is still open. Should it be closed if you consider that we must not 
add an optional sort_attrs=False attribute?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

FYI pungi project is also broken by this change and it blocks Fedora Rawhide to 
upgrade to Python 3.8:
https://bugzilla.redhat.com/show_bug.cgi?id=1698514

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

This is done now. Thanks everyone who helped in discussing and implementing 
this change.

I will leave Serhiy's last PR (adding the "sort_attrs" flag option) open for a 
while until I'm sure we have a better solution for comparing XML in 3.8, at 
which point I would reject it.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Stefan, as the module maintainer, you get to make the final decision on this.  
What you've proposed makes sense in light of the prior discussion and the 
python-dev discussion.  Unless you think something new will come along to 
change your mind, just mark as closed/fixed, then continue work on c14n in 
another tracker issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

Thanks to the discussion that rhettinger triggered on python-dev, it seems that 
there is a majority accordance for considering the previous ordering "undefined 
but deterministic" rather than "sorted", and for making the change from 
"arbitrarily sorted" to "user defined" in Py3.8. That matches the current 
status after the merge of the four pull requests, for both ElementTree and 
MiniDOM.

There also seems to be a large fraction of participants that would like to see 
a stdlib way to safely/correctly/easily compare XML output. IMHO, that is 
covered by issue13611, adding a C14N module to ElementTree. There is also a 
(Py2) C14N implementation for minidom, from the now unmaintained PyXML 
package¹, which, according to the license, is already distributed under Python 
copyright. I will try to bring at least the ET module in shape for integration 
before 3.8 beta1.

During the discussions, several ways were shown to achieve deterministic 
behaviour, also across Python releases. My proposal is therefore to *not* add a 
new "sort_attrs" option to the API, with the reasoning that it is a) not 
required for any of the use cases, b) not maintaining compatibility without 
code changes (neither backwards nor forward), and c) not what most people 
actually want who strive for deterministic output. Instead, I think we should 
concentrate on providing and documenting better ways to compare XML output.

¹ 
https://github.com/actmd/PyXML/blob/97f144152878a67cd95dd229fe72e86f19bce706/xml/dom/ext/c14n.py

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

I set the priority to release blocker to make sure that the issue got enough 
attention. Raymond started a thread on python-dev, so I reset the priority:
https://mail.python.org/pipermail/python-dev/2019-March/156709.html

Moreover, the change is now documented in "What's New in Python 3.8?" 
documentation ("Changes in the Python API" section):
https://github.com/python/cpython/commit/06e1e688228013f411aca6309562ef80df6ce5d3

--
priority: release blocker -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> how that's acceptable?

For docutils, we'll most likely propose some variant of Stéphane Wirtel's 
script to test semantic equivalence for docutils.  For other cases, Serhiy is 
working on a C14N canonicalization tool which is specifically designed for the 
task of creating reproducible output, in a cross-language standards compliant 
way.

As Stefan Behnel clearly articulated, there are multiple reasons why Python 
should not guarantee byte-for-byte serialization across point releases.  That 
said, we'll likely make the guarantee across micro-releases.  That will make it 
possible a third mitigation strategy of generating new baseline files for a new 
point releases and adding a version check to decide which baseline to test 
against.

FWIW, we had a similar discussion regarding hash randomization.  While there 
are a number of significant differences, the outcome is relevantL  User tests 
that depended on non-guaranteed implementation details had to be fixed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Julien Palard]

Julien Palard  added the comment:

> Actually, it is really easy to fix

By answering this, it looks like you're currently going this way, and obviously 
you'll succeed fixing docutils, this still mean voluntarily leaving some (or 
many?) other code broken (open source and closed source), how that's acceptable?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> one looks tricky to fix (docutils)

Actually, it is really easy to fix just by updating the target comparison files 
(that is the procedure described in the comments for the test).

What is taking more thought is coming up with a better test that verifies 
intended semantic content rather than the exact serialization.  I'm working 
with Stéphane to figure out how to do this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Julien Palard]

Julien Palard  added the comment:

We found two occurrences of external code breaking due to this change, and one 
looks tricky to fix (docutils), this indicates there are other library that 
will break.

> My (somewhat educated) gut feeling is that most users simply won't care or 
> won't even notice the change.

I concur with this. Why changing the default behavior? The costs of changing 
this looks high, and the benefits looks very low (people not noticing).

As already proposed, a keyword-only `sorted` parameter defaulting to True (the 
current behavior) would allow one to give False and get control of ordering, 
and won't break any existing library.

On the plus side `sorted=True` makes the current behavior (sorted by default) 
more discoverable.

--
nosy: +mdk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

@raymond

Here is a basic comparison between two xml streams.

Maybe we could propose this solution when we want to compare an XML with 
xml.dom.minidom instead of the byte-to-byte comparison.

--
Added file: https://bugs.python.org/file48217/test_xml_compare.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I'll make a post to python-dev so that we can escalate the discussion and get 
broader participation.  Personally, I'm not wedded to any one particular 
outcome and just want to do what is best for the users in the long run.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

> Victor, as much as I appreciate backwards compatibility, I really don't think 
> it's a big deal in this case.

In short, the XML serialization is no longer deterministic. It depends in which 
order attributes are defined. Example:
---
from xml.etree import ElementTree as ET

document = ET.Element('document')
document.attrib['a'] = '1'
document.attrib['b'] = '2'
ET.dump(document)

document = ET.Element('document')
document.attrib['b'] = '2'
document.attrib['a'] = '1'
ET.dump(document)
---

Python 3.7 output:
---


---

Python 3.8 output:
---


---

On this example, it's obvious that the attributes are defined in a different 
order, but the code which generates the XML document can be way more complex 
and use unordered data structures like set().

Why does it matter? Most programs compare XML using basic string comparison, 
they don't implement smart XML comparison which ignore attributes order.

Concrete example:

* A program which rewrites the XML on disk if attribute order changes (useless 
disk write).
* Version Control System (Git, hg, svn, whatever) sees the file as modified and 
produces a change which can be unexpected and annoy users (use more disk space, 
more network bandwidth, etc.)
* https://reproducible-builds.org/
* It breaks docutils unit tests which uses straightfoward assertEqual(). 
docutils is just one example: it's not hard to imagine many other applications 
using XML and which use a similar check.
* etc.

It's not just a matter of parsers.


> My (somewhat educated) gut feeling is that most users simply won't care or 
> won't even notice the change.

... Really? Why do you think that so many people are involved in this issue if 
nobody cares of XML attribute order? :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

Victor, as much as I appreciate backwards compatibility, I really don't think 
it's a big deal in this case. In fact, it might not even apply.

My (somewhat educated) gut feeling is that most users simply won't care or 
won't even notice the change. Out of those who do (or have to) care, many are 
better off by fixing their code to not rely on an entirely arbitrary (sorted by 
name) attribute order than by getting the old behaviour back. And for those few 
who really need attributes to be sorted by name, there's the recipe I posted 
which works on all ElementTree implementations out there with all alive CPython 
versions.

> This recipe does modify the document and so changes the behaviour of the 
> application when it iterates on attributes later

This is actually a very rare thing. If I were to make up numbers, I'd guess 
that some 99% of the applications do XML serialisation as the last thing and 
then throw away the tree afterwards, without touching it (or its attributes) 
again. And the remaining cases are most probably covered by the "don't need to 
care" type of users. I don't think we should optimise for 0.05% of our user 
base by providing a new API option for them. Especially in ElementTree, which 
decidedly aims to be simple.

The example that Ned gave refers to a very specific and narrow case: comparing 
serialised XML, at the byte level, in tests. He was very lucky that ElementTree 
was so stable over the last 10 Python releases that the output did not change 
at all. That is not something that an XML library needs to guarantee. There is 
some ambiguity in XML for everything that's outside of the XML Information set, 
and there is a good reason why the W3C has tackled this ambiguity with an 
explicit and *separate* specification: C14N. So, when you write:

> Many XML parsers rely on the order of attributes

It's certainly not many parsers, and could even be close to none. The order of 
attributes is explicitly excluded from the XML Information set:

https://www.w3.org/TR/xml-infoset/#omitted

Despite this, cases where the order of the attributes matters to the 
*application* are not unheard of. But for them, attributes sorted by their name 
are most likely the problem and not a solution. Raymond mentioned one such 
example. Sorting attributes by their name really only fulfils a single purpose: 
to get reproducible output in cases where the order does *not* matter. For all 
the (few) cases where the order *does* matter, it gets in the way.

But by removing the sorting, as this change does, we still get predictable 
output due to dict ordering. So this use case is still covered. It's just not 
necessarily the same output as before, because now the ordering is entirely in 
the hands of the users. Meaning, those users who *do* care can now actually 
influence the ordering, which was very difficult and hackish to achieve before. 
We are allowing users to remove these hacks, not forcing them to add new ones.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

Please don't break the backward compatibility without an easy way to retrieve 
Python 3.7 behavior.

I set the priority to release blocker until a consensus can be found, and I add 
Lukasz (Python 3.8 release manager) in the loop.


Ned Batchelder: "I'm a bit mystified why people are still opposed to providing 
sorting control, even after people are resorting to "hack work-arounds."  The 
(two) pull requests that provide the control are simple, easy to understand, 
and easy to test."

I concur. I'm fine with sorting by default, but it breaks the backward 
compatibility on purpose without providing an option to opt-in for the old 
behavior :-(

Many XML parsers rely on the order of attributes. It's part of the XML 
"semantics". Well, it shouldn't, but I cannot fix all applications around the 
world to explain them that Python is right, and they are all wrong :-)

It's not straighforward to fix an application to get Python 3.7 behavior. I 
would prefer to not have to copy-paste Stefan Behnel's recipe in every project 
using XML who wants to sort attributes:

"""
Something like this:

def sort_attributes(root):
for el in root.iter():
attrib = el.attrib
if len(attrib) > 1:
attribs = sorted(attrib.items())
attrib.clear()
attrib.update(attribs)
"""

This recipe does modify the document and so changes the behavior of the 
application when it iterates on attributes later, whereas in Python 3.7 
attributes are only sorted while writing the XML into a file.

--
nosy: +lukasz.langa
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

in fact, there is no easy way for the fix for python-docutils.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 06e1e688228013f411aca6309562ef80df6ce5d3 by Raymond Hettinger 
(Diego Rojas) in branch 'master':
bpo-34160: Update news entry for XML order attributes (#12335)
https://github.com/python/cpython/commit/06e1e688228013f411aca6309562ef80df6ce5d3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> Raymond, it sounds like your end-user didn't agree that
> "sorting is no longer necessary or desirable."

Sorry for being unclear.  The user needed to *overcome* the sorting behavior so 
that they could produce an ordering of their own choosing.  The problem then 
became that we had to work out a way to defeat the sorting to allow the 
requested attribute ordering to be preserved.

Roughly:

   casefile = Element('casefile', clearance="confidential",
  access="internal", valid_through="31 Mar 2022")

The desired result is that the attribute order be preserved.  By removing the 
sort, a user can specify an order they want and have that specification honored.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

:) The coolest thing about it is: it's not a hack at all. It's simply making 
use of the new feature in a suitable way.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

Thanks for the suggestion.  My problem has already been fixed with my own hack 
workaround.  Your idea is a good one to leave here so that other frustrated 
users will have code to copy for their hack workaround.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

Something like this:

def sort_attributes(root):
for el in root.iter():
attrib = el.attrib
if len(attrib) > 1:
attribs = sorted(attrib.items())
attrib.clear()
attrib.update(attribs)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

Ned, would it solve your problem to write a helper function that walks the tree 
and sorts the attrib dicts? That would also work in all existing CPython 
versions (because they already sort attributes anyway), for both ElementTree 
and lxml, and you wouldn't have to pass a new option into the serialiser 
conditionally based on the running Python version.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

I'm a bit mystified why people are still opposed to providing sorting control, 
even after people are resorting to "hack work-arounds."  The (two) pull 
requests that provide the control are simple, easy to understand, and easy to 
test.

Raymond, it sounds like your end-user didn't agree that "sorting is no longer 
necessary or desirable."

How many people have to comment here on the difficulties they are having before 
we do the simple thing that will help them?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

FWIW, this issue arose from an end-user problem. She had a hard requirement to 
show a security clearance level as the first attribute.  We did find a work 
around but it was hack.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Is not PR 12354 a duplicate of PR 10452?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

@scoder

ok, we can close my PR and I will submit a patch to docutils.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

FWIW, I second Raymond's opposition against a new option. It's really not 
something that users should care about. Instead of us providing a new "sort or 
not" option, and people adding it (conditionally!) to their code to "fix" 
previous or future behaviour of the tool, they should better invest the time 
into fixing their code to not rely on behaviour that was never guaranteed.

Also, lxml cannot support such a serialisation option because it preserves the 
order of attributes as soon as they are *created*. It previously also sorted 
dicts on input, but only dicts and not other (sequential) ways to set 
attributes, which I always considered a necessary quirk to gain reproducible 
output. I'm planning to remove the sorting for Py3.6+ in the next release. It 
is better to leave the decision about attribute order entirely to the users.

It's not difficult to get sorted dict behaviour in Py3.6+ if you really want it 
(walk the tree and sort each el.attrib if you feel like it), but it's currently 
impossible to _not_ get sorted attributes but the "expected" order of creation. 
That's what we should fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stéphane Wirtel]

Stéphane Wirtel  added the comment:

Diego,

Victor indicates that we could propose an option to the user

```
* I cannot see the change documented anywhere in 
https://docs.python.org/dev/whatsnew/3.8.html
* I don't see any simple workaround. It would be nice to add an opt-in option 
to sort attributes just to get Python 3.7 behavior.
```

--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Stéphane, According to you PR, I agree that is a good approach give the option 
to the user if wants sorted or not the attributes. But in this thread, is 
discussed that there is not necessary and leaves as default that the attributes 
preserve the order.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
pull_requests: +12319

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Change by Diego Rojas :


--
pull_requests: +12306

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Please do add an entry to WhatsNew.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

> Victor, I thought that the news changes were only for major changes.

It's a backward incompatible change. It broke at least docutils. So it should 
be mentioned at least at:
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-python-api

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Victor, I thought that the news changes were only for major changes. 

In another hand, I could retake this bug in a few days with the issues that you 
mentioned.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

This change broke docutils test suite: 
https://sourceforge.net/p/docutils/bugs/359/

Problems:

* I cannot see the change documented anywhere in 
https://docs.python.org/dev/whatsnew/3.8.html
* I don't see any simple workaround. It would be nice to add an opt-in option 
to sort attributes just to get Python 3.7 behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Julian Sivertsen]

Julian Sivertsen  added the comment:

I don't understand why this library should go out of its way to support the old 
behavior when it seems like the only thing it breaks is tests that assume 
something that was never guaranteed and where you can get the old behavior in 
just two lines of Python:

for node in root.iter():
node.attrib = dict(sorted(node.items()))

Kind regards from confused Pythoner that just wonted the attribute order to 
make sense

--
nosy: +sivert

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

I happen to use xml.dom.minidom, though I didn't mean this to be, "fix Ned's 
personal problem" :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I am working on adding a canonization support (issue13611 for ElementTree and 
perhaps separate issue for minidom), but prefer to keep this issue open for the 
case if this task turns out to be too difficult.

Ned, what module is used in your tests for serialization? minidom or 
ElementTree?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> It seemed like we were close to a consensus about adding 
> back the option to sort the attributes, or did I 
> misunderstand?

Adding a sort() option to prettyxml() seems perfectly reasonable.  I wouldn't 
extend any of the other APIs though.

Also, it is reasonable to consider adding a standards compliant 
canonicalization tool.

> Without the sorted attributes, 3.8 fails the test.

It seems to me that the test should be improved as well so that it doesn't rely 
on implementation details.  The usual way to test serialization is to verify 
that it round trips.

Unless the test is improved to not rely on implementation details, I'm 
concerned that we'll end-up having this conversation again if any other 
serialization improvements are made.

So yes, I support adding a sort() option to prettyxml() and adding a 
canonicalization tool, but no I don't agree with the premise that Py 2.7, 3.4, 
3.5, 3.6, 3.7, 3.8 were required to now and forever always produce byte-by-byte 
identical output -- afaict that was never a promised behavior, so any test that 
relies on that premise should be improved.

FWIW, when I described the existing test as being "fragile", it wasn't a 
pejorative, I meant it in the literal sense of "easily broken".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

It seemed like we were close to a consensus about adding back the option to 
sort the attributes, or did I misunderstand?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> I run my test suite on 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, PyPy2, 
> and PyPy3.  Today, one gold file suffices for all those
> versions.  Without the sorted attributes, 3.8 fails the test.

Would it make sense to rewrite the test to just make sure the XML roundtrips 
instead of relying on the exact byte sequence that is generated?  IIRC, that is 
how we test pickling and various codecs -- rather than relying on implement 
specific details for a particular byte sequence -- the test is roughly "assert 
decode(encode(thing)) == thing".  That allows the test to survive on-going 
improvements to serialization.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

Stefan, just to clarify: it isn't that I don't feel like re-serializing my gold 
files with the latest version.  I run my test suite on 2.7, 3.4, 3.5, 3.6, 3.7, 
3.8, PyPy2, and PyPy3.  Today, one gold file suffices for all those versions.  
Without the sorted attributes, 3.8 fails the test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

+1 for adding C14N support for ElementTree.

But the problem will still exist for minidom. We will need either add the 
sort_attrs parameter in prettyxml() or complete C14N support in minidom.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

> Maybe some people prefer sorting to get a more deterministic output

Note that those people are much better off with C14N. It is the one tool 
designed for all of these use cases, even usable for cryptographic signatures. 
And it's trivial to use, it's just a different serialiser.

The new option would only be for people who want the old behaviour for 
backwards compatibility reasons, like Ned who wants to keep his existing byte 
level tests working that were created by the previous version. Understandable, 
although re-serialising the test samples with C14N would probably still be the 
better approach, because it guards against more than just changes in the 
attribute order.

Maybe it's time to think about adding the ET C14N module to the stdlib again, 
see issue13611.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

I like the idea of having an opt-in option to get Python 3.7 behavior in Python 
3.8, sort=True. It would be a new parameter in Python 3.8, right?

It makes sense to me to ask explicitly to sort attributes, especially since 
Python decided to no longer sort by default :-)

Maybe some people prefer sorting to get a more deterministic output for things 
like https://reproducible-builds.org/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

One other thought:  We should add a note to the docs for all of the 
serialization formats saying that we specifically disclaim that they will 
always generate exactly the same output byte-for-byte.  When Serhiy made some 
small optimizations to the encoding of pickles, it would have broken any test 
that checked byte-level equality rather than semantic level equality (checked 
by making sure the pickle/unpickle steps would round-trip without loss of 
information).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Raymond's words convinced me, but if it is needed for preserving compatibility 
in tests, here is PR 10452 which adds support for the sort_attrs argument in 
XML serializing methods.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +9727

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> Being able to sort attributes would be a very useful addition 
> to the prettyxml method.

+1 This seems reasonable to me.

I misread it and thought it was much broader than prettyxml().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

Words like flawed from the outset, fundamentally flawed, fragile, and 
ill-conceived are being thrown around, which does not help the discussion.  Can 
we focus on the question of whether it's reasonable to add sorted attributes as 
an option?

This idea exactly parallels the sort_keys option in the json module.  It 
precisely solves the problem I have.  The code change is small.  I'll even 
volunteer to make the pull request.

I'm not suggesting that we get rid of the new feature of preserving the user's 
attributes order. I'm not even suggesting that the new feature needs to be the 
non-default option. I'd just like a way to sort attributes in XML output.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Tal Einat]

Tal Einat  added the comment:

>> ISTM that the coverage tests as currently written aren't good tests.

> Hi, I'd like to remind everyone to be open, respectful, and considerate. 
> There are ways to describe hos things that can be improved. There is no need 
> to denigrate other people's work.

I find this to be an overreaction in this case.  Sure, it could have been 
worded more positively, but the negativity was very mild; the tests weren't 
even being called "bad", not to mention overly negative wording e.g. "horrible".

Further, you omitted the followup explanation of *what about the tests isn't 
good*:

> Otherwise, the tests are relying on a non-guaranteed implementation detail.

IMO we shouldn't require ourselves to be overly careful in our wording, such as 
avoiding any negative wording entirely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

IMO adding optional keyword argument to make it compatible with previous 
version is reasonable request, and it seems early on several other core devs 
also agree with it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

> ISTM that the coverage tests as currently written aren't good tests.

Hi, I'd like to remind everyone to be open, respectful, and considerate. There 
are ways to describe hos things that can be improved. There is no need to 
denigrate other people's work.

--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> I can see that there are ways that I can change my tests. 
> I see that there are third-party libraries that can help me with this.

Please do that.  That is good design and uses all the tools the way they are 
meant to be used.


> But changing the behavior of the standard library, without a 
> way to retain the old behavior, and asking people to adapt 
> by changing their test suites, seems a bit cavalier.

Several thoughts:

* We are not cavalier.  We generally work very hard to preserve guaranteed 
behaviors.  For example, see all the work done on version numbering random 
seeding or pickle versions.  That said, we cannot be handcuffed by 
over-reliance on implementation details; otherwise, we wound never be able to 
improve the library.  The whole point of encapsulation and abstraction is to 
provide freedom to improve the internals and add capabilities without breaking 
the API contract.

* It is reasonable for folks to change tests which were flawed from the outset 
(just like when hash randomization was introduced, it scrambled key order and 
broke any tests that relied on key order).  FWIW, we remove or fix fragile or 
ill conceived tests in our own test suite all the time.

* We could add an option to sort attributes but this just clutters the API and 
turns an old quirk into a permanent fixture.  Instead, I recommend adding a 
documentation link to a standards compliant canonicalization tool.

> Let's honor Python's tradition of standard library stability, 
> and give me a way to keep the behavior I used to have.

The library IS stable with respect to documented guaranteed behaviors.  With 
respect to everything else, our tradition is make improvements where we can.  
We don't backport these changes so that mirco-releases retain their quirks.  
But point releases are allowed and encouraged to make improvements (i.e. this 
is for 3.8 and later).

In this case, there was never an API contract to sort attributes.  In a way, 
this is no different than when we improve a repr (see match objects for 
example).  That would break tests that incorrectly relied on an exact output 
match.  

When a test stops matching what a tool does, the one that is incorrect is the 
one that should be changed.  If there is an API contract, the tool needs to 
change to match.  However, if the test is fundamentally flawed, it should be 
one to change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

I can see that there are ways that I can change my tests.  I see that there are 
third-party libraries that can help me with this.

But changing the behavior of the standard library, without a way to retain the 
old behavior, and asking people to adapt by changing their test suites, seems a 
bit cavalier.

Why not add an option to keep the sorting that the standard library has always 
had?  I'll even be OK with changing the default behavior to unsorted, just give 
me a way to explicitly keep the behavior its always had in the past.

The discussion so far has 1) hypothesized that test suites would be broken, and 
2) claimed that no one needs the output in sorted order.  I can tell you that 
my test suite is broken, and that removing the sorted order is what broke it.

The json module has the option to sort keys.  toprettyxml already has options 
well outside the needs of XML semantics (indent and newl).

Let's honor Python's tradition of standard library stability, and give me a way 
to keep the behavior I used to have.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> I'm checking that my XML generating code is producing the right XML.

ISTM that the coverage tests as currently written aren't good tests.  Why not 
update the tests to check the generated XML is what the code asked it to 
generate?  Otherwise, the tests are relying on a non-guaranteed implementation 
detail. 

Alternatively, use c14n.Canonicalize[1] which implements standard compliant 
cross-language canonicalization (as opposed to non-guaranteed ad-hoc comparison 
which just happens to work).

[1] https://www.ibm.com/developerworks/xml/library/x-c14n/

Note with attribute order preservation, code now has the ability to produce 
exactly the XML it wants to produce.  The explicit intention of the user is 
honored.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

This is exactly the situation Stefan was talking about: "However, I still think 
that Serhiy is right: this change would break code, and in particular test code 
that compares XML output. Having to deal with two different "correct" 
serialisations in tests depending on the Python version is annoying at best."

He says comparison should be done with true canonicalization (C14N).  Does that 
exist in the standard library?  With sorted keys, we had a form of good-enough 
canonicalization.  Now we don't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

Diego, they are my tests in the coverage.py test suite.  I'm checking that my 
XML generating code is producing the right XML.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Ned, exactly what test fails? I ran all the test suite in local and all was ok, 
even all passed in the CI build.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

To provide a little more context: coverage.py has tests that the XML reports it 
generates are correct.  It does this by comparing the result to saved XML 
files.  On Python versions up to 3.7, the file compares correctly.  It has 
sorted attributes generated by minidom.  Now on Python 3.8, the comparison 
fails.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

(sorry, to sort the attributes.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Ned Batchelder]

Ned Batchelder  added the comment:

Hi, this broke my tests, just as earlier comments predicted.

Can we get an optional argument to sort the keys?  The json module lets us sort 
keys even though it's irrelevant to JSON.  Being able to sort attributes would 
be a very useful addition to the prettyxml method.

--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 5598cc90c745dab827e55fadded42dbe85e31d33 by Serhiy Storchaka 
(Diego Rojas) in branch 'master':
bpo-34160: Preserve order of attributes in minidom. (GH-10219)
https://github.com/python/cpython/commit/5598cc90c745dab827e55fadded42dbe85e31d33


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Thanks for doing the additional work to finish this up.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Serhiy Storchaka, Raymond. I already made the 10219 PR. Preserves order in html 
Element attributes and minidom.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 3b05ad7be09af1d4510eb698b0a70d36387f296e by Serhiy Storchaka in 
branch 'master':
bpo-34160: Preserve user specified order of Element attributes in html. 
(GH-10190)
https://github.com/python/cpython/commit/3b05ad7be09af1d4510eb698b0a70d36387f296e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Change by Diego Rojas :


--
pull_requests: +9535

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

Raymond, sure. I could do the rest, I'll start tomorrow Monday.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +9508

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[STINNER Victor]

STINNER Victor  added the comment:

Nice enhancement! I was always annoyed by XML libraries in PHP and Python which 
don't respect the "insertion order".

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset e3685fd5fdd8808acda81bfc12fb9702d4b59a60 by Raymond Hettinger in 
branch 'master':
bpo-34160: Preserve user specified order of Element attributes (GH-10163)
https://github.com/python/cpython/commit/e3685fd5fdd8808acda81bfc12fb9702d4b59a60


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Diego, I've taken care of the ElementTree case.  Would you like to make a PR 
for the other occurrences (minidom, html, etc)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Change by Raymond Hettinger :


--
keywords: +patch
pull_requests: +9486
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Okay, lets just remove sorting.

The method for creating XML file in the minidom module also sorts attributes. 
It should be changed as well.

XMLGenerator in xml.sax.saxutils doesn't sort attributes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Tal Einat]

Tal Einat  added the comment:

There is also a middle ground: Keep .write() as it is *and* have .dump() 
preserve the order in which the parameters were supplied in the constructor.  
They could both use a common *internal* method with a flag to select the 
desired ordering.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Diego, it would be premature to write an implementation until we've agreed on 
an API.  

I object to the Serhiy's proposal, and if no one agrees to my proposal to 
preserve the user's specific actions, then it  would be better to do nothing at 
all, leaving the current API (deterministic, but with no particular order being 
guaranteed).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Though it is compatible with earlier versions, I don't see any reason to keep 
sorting at all, especially as a default.  The reasonable default is that 
whatever order the user specifies is what the tool does.

The backwards compatible argument is specious.  The ordering was never 
guaranteed.  The only reason that we had sorting was to make the XML generation 
deterministic, and that is something we would still have.

I vote against the permanent API expansion with an unhelpful default.  The 
sorting is no longer necessary or desirable.  Appropriately, we never 
guaranteed it, leaving us the implementation flexibility to make exactly this 
change.

AFAICT, no XML user has ever expressed an interest in having attributes appear 
in alphabetical order -- for that matter, I can't recall any HTML user having 
expressed this preference (instead, they prefer to put class and id tags ahead 
of other tags for example).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Diego Rojas]

Diego Rojas  added the comment:

I'm working on this issue, but I have some questions:

1. If dump() function is only for debugging purpose, and since from dump() is 
where we will pass the sort_attrs keyword in False in order to conserve the 
order, I don't see how from dump() we can handle the order of the keyword 
arguments. The sorted occurs in line 926 
(https://github.com/python/cpython/blob/master/Lib/xml/etree/ElementTree.py#L926)
 and 
982(https://github.com/python/cpython/blob/master/Lib/xml/etree/ElementTree.py#L982)
 when xml or html are serialized. Or could we pass the sort_attrs to the 
functions _serialize_html/_serialize_xml and there handle this?

2. Just a comment, maybe sort_attrs is not ambiguous? For a user would be clear 
that sort_attrs will order the arguments in lexical order or maybe he/she could 
be confused and take it as that will respect the order of the keywords passed?

--
nosy: +dfrojas

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is an easy issue and I left it for beginning contributors. The PR should 
include the following changes:

* Add an optional sort_attrs parameter (True by default) in the write() method.

* Make the dump() function passing sort_attrs=False to write().

* Add tests for write() (with sort_attrs=False and sort_attrs=True) and dump() 
and/or modify existing tests.

* Document change (in the module docs, in What's New, news entry).

--
keywords: +easy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Tal Einat]

Tal Einat  added the comment:

Thanks for the clarification, Raymond.

I've spent a few minutes searching for uses of dump(), and have only come up 
with uses for debugging or printing helpful info in command line tools.  So, it 
seems that changing this as suggested wouldn't break much existing code, since 
I couldn't easily find even one such example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Consider this as a feature request.  It is perfectly reasonable for a user to 
want to generate specific XML output and to want to control the order that the 
attributes are listed.  It is perfectly reasonable for the API to preserve the 
order that the user specified rather than change it into some other order that 
they either don't expect or can't control.  Starting in Python 3.6, the 
language guarantees that the order of keyword arguments is preserved.  Knowing 
that, it will be natural for users to start viewing as buggy APIs that discard 
that order when everything else in the eco-system respects the ordering.

When I teach ElementTree in Python courses, students notice this behavior and 
ask why the order was swapped.  My only answer is that we didn't used to have a 
way to control it, so the input order was ignored (i.e. a reason that no longer 
makes sense).

--
type: behavior -> enhancement
versions:  -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Tal Einat]

Tal Einat  added the comment:

This doesn't strike me as a bug, or even clearly being a potential improvement.

If this about round-trip parsing/serializing not preserving order, that would 
be significant. As it is, if this is only intended as a debugging tool, why 
change it?

Is there a concrete reason to change this and break backwards compatibility? If 
not I suggest we close this as "won't fix".

--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Stefan Behnel]

Stefan Behnel  added the comment:

At least for lxml, attributes were never specified to have a sorted order 
(although attribute dicts are sorted on *input*, to give a *predictable* order 
as in ET), and the tutorial says: "Attributes are just unordered name-value 
pairs".

However, I still think that Serhiy is right: this change would break code, and 
in particular test code that compares XML output. Having to deal with two 
different "correct" serialisations in tests depending on the Python version is 
annoying at best. But OTOH, comparing XML output should always be based on C14N 
and not on an arbitrary serialisation. XML C14N serialisation was specifically 
designed to provide a deterministic output byte sequence, regardless of how the 
XML document was created. (This is also used for cryptographic fingerprinting.)

It is interesting that ET sorts the attributes on output. lxml does it only on 
API *input*, because the underlying tree model is order preserving. Parsed 
attributes are always written in the original document order, followed by the 
attributes set through the API in the order in which they were set.

For lxml, a serialisation flag is not going to help, because the serialisation 
order is always deterministic from the time where an attribute is added to the 
tree. Given that dicts are order preserving in Python now, ET could follow this 
path as well. The attributes are already in parse order and could easily be 
serialised that way, and attributes that were added through the API would end 
up being serialised last, also preserving the order. This would leave lxml and 
ET with the same serialisation order, which seems attractive.

In short, I like this change, it's just difficult to see a way how this could 
preserve full backwards compatibility. And it's difficult to say how important 
backwards compatibility really is here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Attributes are sorted by name when converted an element to the text 
representation for making it stable. Changing this may break existing software.

I think it makes sense now to add an optional keyword-only parameter sort_attrs 
(True by default) for functions tostring() and tostringlist() and method 
ElementTree.write(). If it is False, attributes will keep the original order.

Since dump() is used for debugging only, it could change the behavior by 
default, even in maintained releases. But currently it just uses 
ElementTree.write(), which should sort attributes by default for compatibility.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34160] ElementTree not preserving attribute order

[Raymond Hettinger]

New submission from Raymond Hettinger :

Starting with Python3.6, the order of keyword arguments has been guaranteed.  
Something in ElementTree is not respecting that order.

$ python3.7
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from xml.etree.ElementTree import Element, dump
>>> dump(Element('cirriculum', status='public', company='example'))

>>> # ^-^ These are swapped

--
components: Library (Lib)
messages: 321973
nosy: eli.bendersky, rhettinger, scoder
priority: normal
severity: normal
status: open
title: ElementTree not preserving attribute order
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com