[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: New changeset 63673916464bace8e2147357395fdf3497967ecb by Stefan Behnel (Miss Islington (bot)) in branch '3.8': [3.8] bpo-34160: explain how to deal with attribute order in ElementTree (GH-14867) (GH-14935)

[issue34160] ElementTree not preserving attribute order

Change by miss-islington : -- pull_requests: +14708 pull_request: https://github.com/python/cpython/pull/14935 ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: New changeset a3697db0102b9b6747fe36009e42f9b08f0c1ea8 by Stefan Behnel in branch 'master': bpo-34160: explain how to deal with attribute order in ElementTree (GH-14867) https://github.com/python/cpython/commit/a3697db0102b9b6747fe36009e42f9b08f0c1ea8

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: I created a PR that adds a couple of paragraphs to the documentation. Comments welcome. -- versions: +Python 3.9 ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Change by Stefan Behnel : -- pull_requests: +14654 pull_request: https://github.com/python/cpython/pull/14867 ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Change by Tal Einat : -- nosy: -taleinat ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Victor, you mean place again this code? https://github.com/python/cpython/pull/10163/files#diff-d5a064acb6ae44dcb7e01fee148c733dR926 And the recipe proposed in https://bugs.python.org/issue34160#msg338102 place it as a method of ElementTree or just as a helper

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: Other examples of projects broken by this change: * python-glyphsLib: https://bugzilla.redhat.com/show_bug.cgi?id=1705391 * pcs: https://bugzilla.redhat.com/show_bug.cgi?id=1705475 What's New In Python 3.8 only briefly mention this change: "The writexml(),

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: I implemented (most of) C14N 2.0 in issue 13611. Please give it a try if you are interested in the canonical serialisation feature. I would like to include it in Py3.8. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: I rejected the (now conflicting) PR that adds a sorting option. I also sent Victor a tentative (and trivial) patch for the pungi package. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: PR 10452 is still open. Should it be closed if you consider that we must not add an optional sort_attrs=False attribute? -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: FYI pungi project is also broken by this change and it blocks Fedora Rawhide to upgrade to Python 3.8: https://bugzilla.redhat.com/show_bug.cgi?id=1698514 -- nosy: +vstinner ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: This is done now. Thanks everyone who helped in discussing and implementing this change. I will leave Serhiy's last PR (adding the "sort_attrs" flag option) open for a while until I'm sure we have a better solution for comparing XML in 3.8, at which point I

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Stefan, as the module maintainer, you get to make the final decision on this. What you've proposed makes sense in light of the prior discussion and the python-dev discussion. Unless you think something new will come along to change your mind, just mark

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: Thanks to the discussion that rhettinger triggered on python-dev, it seems that there is a majority accordance for considering the previous ordering "undefined but deterministic" rather than "sorted", and for making the change from "arbitrarily sorted" to

[issue34160] ElementTree not preserving attribute order

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: I set the priority to release blocker to make sure that the issue got enough attention. Raymond started a thread on python-dev, so I reset the priority: https://mail.python.org/pipermail/python-dev/2019-March/156709.html Moreover, the change is now

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > how that's acceptable? For docutils, we'll most likely propose some variant of Stéphane Wirtel's script to test semantic equivalence for docutils. For other cases, Serhiy is working on a C14N canonicalization tool which is specifically designed for the

[issue34160] ElementTree not preserving attribute order

Julien Palard added the comment: > Actually, it is really easy to fix By answering this, it looks like you're currently going this way, and obviously you'll succeed fixing docutils, this still mean voluntarily leaving some (or many?) other code broken (open source and closed source), how

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > one looks tricky to fix (docutils) Actually, it is really easy to fix just by updating the target comparison files (that is the procedure described in the comments for the test). What is taking more thought is coming up with a better test that verifies

[issue34160] ElementTree not preserving attribute order

Julien Palard added the comment: We found two occurrences of external code breaking due to this change, and one looks tricky to fix (docutils), this indicates there are other library that will break. > My (somewhat educated) gut feeling is that most users simply won't care or > won't even

[issue34160] ElementTree not preserving attribute order

Stéphane Wirtel added the comment: @raymond Here is a basic comparison between two xml streams. Maybe we could propose this solution when we want to compare an XML with xml.dom.minidom instead of the byte-to-byte comparison. -- Added file:

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: I'll make a post to python-dev so that we can escalate the discussion and get broader participation. Personally, I'm not wedded to any one particular outcome and just want to do what is best for the users in the long run. --

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: > Victor, as much as I appreciate backwards compatibility, I really don't think > it's a big deal in this case. In short, the XML serialization is no longer deterministic. It depends in which order attributes are defined. Example: --- from xml.etree import

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: Victor, as much as I appreciate backwards compatibility, I really don't think it's a big deal in this case. In fact, it might not even apply. My (somewhat educated) gut feeling is that most users simply won't care or won't even notice the change. Out of

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: Please don't break the backward compatibility without an easy way to retrieve Python 3.7 behavior. I set the priority to release blocker until a consensus can be found, and I add Lukasz (Python 3.8 release manager) in the loop. Ned Batchelder: "I'm a bit

[issue34160] ElementTree not preserving attribute order

Stéphane Wirtel added the comment: in fact, there is no easy way for the fix for python-docutils. -- ___ Python tracker ___ ___

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: New changeset 06e1e688228013f411aca6309562ef80df6ce5d3 by Raymond Hettinger (Diego Rojas) in branch 'master': bpo-34160: Update news entry for XML order attributes (#12335) https://github.com/python/cpython/commit/06e1e688228013f411aca6309562ef80df6ce5d3

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > Raymond, it sounds like your end-user didn't agree that > "sorting is no longer necessary or desirable." Sorry for being unclear. The user needed to *overcome* the sorting behavior so that they could produce an ordering of their own choosing. The

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: :) The coolest thing about it is: it's not a hack at all. It's simply making use of the new feature in a suitable way. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: Thanks for the suggestion. My problem has already been fixed with my own hack workaround. Your idea is a good one to leave here so that other frustrated users will have code to copy for their hack workaround. --

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: Something like this: def sort_attributes(root): for el in root.iter(): attrib = el.attrib if len(attrib) > 1: attribs = sorted(attrib.items()) attrib.clear() attrib.update(attribs) --

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: Ned, would it solve your problem to write a helper function that walks the tree and sorts the attrib dicts? That would also work in all existing CPython versions (because they already sort attributes anyway), for both ElementTree and lxml, and you wouldn't

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: I'm a bit mystified why people are still opposed to providing sorting control, even after people are resorting to "hack work-arounds." The (two) pull requests that provide the control are simple, easy to understand, and easy to test. Raymond, it sounds

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: FWIW, this issue arose from an end-user problem. She had a hard requirement to show a security clearance level as the first attribute. We did find a work around but it was hack. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: Is not PR 12354 a duplicate of PR 10452? -- ___ Python tracker ___ ___ Python-bugs-list

[issue34160] ElementTree not preserving attribute order

Stéphane Wirtel added the comment: @scoder ok, we can close my PR and I will submit a patch to docutils. -- ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: FWIW, I second Raymond's opposition against a new option. It's really not something that users should care about. Instead of us providing a new "sort or not" option, and people adding it (conditionally!) to their code to "fix" previous or future behaviour of

[issue34160] ElementTree not preserving attribute order

Stéphane Wirtel added the comment: Diego, Victor indicates that we could propose an option to the user ``` * I cannot see the change documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html * I don't see any simple workaround. It would be nice to add an opt-in option to sort

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Stéphane, According to you PR, I agree that is a good approach give the option to the user if wants sorted or not the attributes. But in this thread, is discussed that there is not necessary and leaves as default that the attributes preserve the order.

[issue34160] ElementTree not preserving attribute order

Change by Stéphane Wirtel : -- pull_requests: +12319 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

Change by Diego Rojas : -- pull_requests: +12306 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Please do add an entry to WhatsNew. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: > Victor, I thought that the news changes were only for major changes. It's a backward incompatible change. It broke at least docutils. So it should be mentioned at least at: https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-python-api --

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Victor, I thought that the news changes were only for major changes. In another hand, I could retake this bug in a few days with the issues that you mentioned. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: This change broke docutils test suite: https://sourceforge.net/p/docutils/bugs/359/ Problems: * I cannot see the change documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html * I don't see any simple workaround. It would be nice to add an

[issue34160] ElementTree not preserving attribute order

Julian Sivertsen added the comment: I don't understand why this library should go out of its way to support the old behavior when it seems like the only thing it breaks is tests that assume something that was never guaranteed and where you can get the old behavior in just two lines of

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: I happen to use xml.dom.minidom, though I didn't mean this to be, "fix Ned's personal problem" :) -- ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: I am working on adding a canonization support (issue13611 for ElementTree and perhaps separate issue for minidom), but prefer to keep this issue open for the case if this task turns out to be too difficult. Ned, what module is used in your tests for

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > It seemed like we were close to a consensus about adding > back the option to sort the attributes, or did I > misunderstand? Adding a sort() option to prettyxml() seems perfectly reasonable. I wouldn't extend any of the other APIs though. Also, it

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: It seemed like we were close to a consensus about adding back the option to sort the attributes, or did I misunderstand? -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > I run my test suite on 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, PyPy2, > and PyPy3. Today, one gold file suffices for all those > versions. Without the sorted attributes, 3.8 fails the test. Would it make sense to rewrite the test to just make sure the XML

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: Stefan, just to clarify: it isn't that I don't feel like re-serializing my gold files with the latest version. I run my test suite on 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, PyPy2, and PyPy3. Today, one gold file suffices for all those versions. Without the sorted

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: +1 for adding C14N support for ElementTree. But the problem will still exist for minidom. We will need either add the sort_attrs parameter in prettyxml() or complete C14N support in minidom. -- ___ Python

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: > Maybe some people prefer sorting to get a more deterministic output Note that those people are much better off with C14N. It is the one tool designed for all of these use cases, even usable for cryptographic signatures. And it's trivial to use, it's just a

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: I like the idea of having an opt-in option to get Python 3.7 behavior in Python 3.8, sort=True. It would be a new parameter in Python 3.8, right? It makes sense to me to ask explicitly to sort attributes, especially since Python decided to no longer sort by

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: One other thought: We should add a note to the docs for all of the serialization formats saying that we specifically disclaim that they will always generate exactly the same output byte-for-byte. When Serhiy made some small optimizations to the

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: Raymond's words convinced me, but if it is needed for preserving compatibility in tests, here is PR 10452 which adds support for the sort_attrs argument in XML serializing methods. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Change by Serhiy Storchaka : -- pull_requests: +9727 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > Being able to sort attributes would be a very useful addition > to the prettyxml method. +1 This seems reasonable to me. I misread it and thought it was much broader than prettyxml(). -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: Words like flawed from the outset, fundamentally flawed, fragile, and ill-conceived are being thrown around, which does not help the discussion. Can we focus on the question of whether it's reasonable to add sorted attributes as an option? This idea

[issue34160] ElementTree not preserving attribute order

Tal Einat added the comment: >> ISTM that the coverage tests as currently written aren't good tests. > Hi, I'd like to remind everyone to be open, respectful, and considerate. > There are ways to describe hos things that can be improved. There is no need > to denigrate other people's work.

[issue34160] ElementTree not preserving attribute order

Mariatta Wijaya added the comment: IMO adding optional keyword argument to make it compatible with previous version is reasonable request, and it seems early on several other core devs also agree with it. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Mariatta Wijaya added the comment: > ISTM that the coverage tests as currently written aren't good tests. Hi, I'd like to remind everyone to be open, respectful, and considerate. There are ways to describe hos things that can be improved. There is no need to denigrate other people's work.

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > I can see that there are ways that I can change my tests. > I see that there are third-party libraries that can help me with this. Please do that. That is good design and uses all the tools the way they are meant to be used. > But changing the

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: I can see that there are ways that I can change my tests. I see that there are third-party libraries that can help me with this. But changing the behavior of the standard library, without a way to retain the old behavior, and asking people to adapt by

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: > I'm checking that my XML generating code is producing the right XML. ISTM that the coverage tests as currently written aren't good tests. Why not update the tests to check the generated XML is what the code asked it to generate? Otherwise, the tests

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: This is exactly the situation Stefan was talking about: "However, I still think that Serhiy is right: this change would break code, and in particular test code that compares XML output. Having to deal with two different "correct" serialisations in tests

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: Diego, they are my tests in the coverage.py test suite. I'm checking that my XML generating code is producing the right XML. -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Ned, exactly what test fails? I ran all the test suite in local and all was ok, even all passed in the CI build. -- ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: To provide a little more context: coverage.py has tests that the XML reports it generates are correct. It does this by comparing the result to saved XML files. On Python versions up to 3.7, the file compares correctly. It has sorted attributes generated

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: (sorry, to sort the attributes.) -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue34160] ElementTree not preserving attribute order

Ned Batchelder added the comment: Hi, this broke my tests, just as earlier comments predicted. Can we get an optional argument to sort the keys? The json module lets us sort keys even though it's irrelevant to JSON. Being able to sort attributes would be a very useful addition to the

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: New changeset 5598cc90c745dab827e55fadded42dbe85e31d33 by Serhiy Storchaka (Diego Rojas) in branch 'master': bpo-34160: Preserve order of attributes in minidom. (GH-10219) https://github.com/python/cpython/commit/5598cc90c745dab827e55fadded42dbe85e31d33

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Thanks for doing the additional work to finish this up. -- ___ Python tracker ___ ___

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Serhiy Storchaka, Raymond. I already made the 10219 PR. Preserves order in html Element attributes and minidom. -- ___ Python tracker ___

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: New changeset 3b05ad7be09af1d4510eb698b0a70d36387f296e by Serhiy Storchaka in branch 'master': bpo-34160: Preserve user specified order of Element attributes in html. (GH-10190)

[issue34160] ElementTree not preserving attribute order

Change by Diego Rojas : -- pull_requests: +9535 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: Raymond, sure. I could do the rest, I'll start tomorrow Monday. -- ___ Python tracker ___ ___

[issue34160] ElementTree not preserving attribute order

Change by Serhiy Storchaka : -- pull_requests: +9508 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34160] ElementTree not preserving attribute order

STINNER Victor added the comment: Nice enhancement! I was always annoyed by XML libraries in PHP and Python which don't respect the "insertion order". -- nosy: +vstinner ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: New changeset e3685fd5fdd8808acda81bfc12fb9702d4b59a60 by Raymond Hettinger in branch 'master': bpo-34160: Preserve user specified order of Element attributes (GH-10163) https://github.com/python/cpython/commit/e3685fd5fdd8808acda81bfc12fb9702d4b59a60

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Diego, I've taken care of the ElementTree case. Would you like to make a PR for the other occurrences (minidom, html, etc)? -- ___ Python tracker

[issue34160] ElementTree not preserving attribute order

Change by Raymond Hettinger : -- keywords: +patch pull_requests: +9486 stage: -> patch review ___ Python tracker ___ ___

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: Okay, lets just remove sorting. The method for creating XML file in the minidom module also sorts attributes. It should be changed as well. XMLGenerator in xml.sax.saxutils doesn't sort attributes. -- ___

[issue34160] ElementTree not preserving attribute order

Tal Einat added the comment: There is also a middle ground: Keep .write() as it is *and* have .dump() preserve the order in which the parameters were supplied in the constructor. They could both use a common *internal* method with a flag to select the desired ordering. --

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Diego, it would be premature to write an implementation until we've agreed on an API. I object to the Serhiy's proposal, and if no one agrees to my proposal to preserve the user's specific actions, then it would be better to do nothing at all,

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Though it is compatible with earlier versions, I don't see any reason to keep sorting at all, especially as a default. The reasonable default is that whatever order the user specifies is what the tool does. The backwards compatible argument is specious.

[issue34160] ElementTree not preserving attribute order

Diego Rojas added the comment: I'm working on this issue, but I have some questions: 1. If dump() function is only for debugging purpose, and since from dump() is where we will pass the sort_attrs keyword in False in order to conserve the order, I don't see how from dump() we can handle the

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: This is an easy issue and I left it for beginning contributors. The PR should include the following changes: * Add an optional sort_attrs parameter (True by default) in the write() method. * Make the dump() function passing sort_attrs=False to write(). *

[issue34160] ElementTree not preserving attribute order

Tal Einat added the comment: Thanks for the clarification, Raymond. I've spent a few minutes searching for uses of dump(), and have only come up with uses for debugging or printing helpful info in command line tools. So, it seems that changing this as suggested wouldn't break much existing

[issue34160] ElementTree not preserving attribute order

Raymond Hettinger added the comment: Consider this as a feature request. It is perfectly reasonable for a user to want to generate specific XML output and to want to control the order that the attributes are listed. It is perfectly reasonable for the API to preserve the order that the

[issue34160] ElementTree not preserving attribute order

Tal Einat added the comment: This doesn't strike me as a bug, or even clearly being a potential improvement. If this about round-trip parsing/serializing not preserving order, that would be significant. As it is, if this is only intended as a debugging tool, why change it? Is there a

[issue34160] ElementTree not preserving attribute order

Stefan Behnel added the comment: At least for lxml, attributes were never specified to have a sorted order (although attribute dicts are sorted on *input*, to give a *predictable* order as in ET), and the tutorial says: "Attributes are just unordered name-value pairs". However, I still

[issue34160] ElementTree not preserving attribute order

Serhiy Storchaka added the comment: Attributes are sorted by name when converted an element to the text representation for making it stable. Changing this may break existing software. I think it makes sense now to add an optional keyword-only parameter sort_attrs (True by default) for

[issue34160] ElementTree not preserving attribute order

New submission from Raymond Hettinger : Starting with Python3.6, the order of keyword arguments has been guaranteed. Something in ElementTree is not respecting that order. $ python3.7 Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) [Clang 6.0 (clang-600.0.57)] on darwin