[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: If there are, I can't remember. This was one or 7 or 8 untokenize issues with about 5 separate bugs between them. If there are any current untokenize issues not covered by some other open issue, then a new issue should be opened. -- resolution: ->

[issue12691] tokenize.untokenize is broken

R. David Murray added the comment: Is there anything left to do here? -- nosy: +r.david.murray ___ Python tracker ___

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset 8d6dd02a973f by Terry Jan Reedy in branch '3.3': Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The http://hg.python.org/cpython/rev/8d6dd02a973f -- ___ Python tracker

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset 0f0e9b7d4f1d by Terry Jan Reedy in branch '2.7': Issue #9974: When untokenizing, use row info to insert backslash+newline. http://hg.python.org/cpython/rev/0f0e9b7d4f1d New changeset 24b4cd5695d9 by Terry Jan Reedy in branch '3.3': Issue #9974:

[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: I fixed the assert and dropped first iter compat-mode token bugs one-by-one by writing narrow unittests that fail and code that makes them pass. I am now working on the '\' continuation issue. That is the subject of #9974, which has a nearly identical patch.

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Thanks for your work on this, Terry. I apologise for the complexity of my original report, and will try not to do it again. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691

[issue12691] tokenize.untokenize is broken

Changes by Terry J. Reedy tjre...@udel.edu: -- assignee: docs@python - terry.reedy nosy: +terry.reedy stage: test needed - patch review versions: +Python 2.7, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset c896d292080a by Terry Jan Reedy in branch '2.7': Untokenize: An logically incorrect assert tested user input validity. http://hg.python.org/cpython/rev/c896d292080a New changeset 51e5a89afb3b by Terry Jan Reedy in branch '3.3': Untokenize: An

[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: The problem of the first iterator pair token being discarded is the subject of #8478. Consider that part of this issue as being closed as a duplicate. The issue of a string being returned if there is no encoding should have been opened as a separate issue,

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: I did some research on the cause of this issue. The assertion was added in this change by Jeremy Hylton in August 2006: https://mail.python.org/pipermail/python-checkins/2006-August/055812.html (The corresponding Mercurial commit is here:

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees g...@garethrees.org: -- assignee: - docs@python components: +Documentation, Tests nosy: +docs@python ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees g...@garethrees.org: Removed file: http://bugs.python.org/file33919/Issue12691.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___

[issue12691] tokenize.untokenize is broken

Yury Selivanov added the comment: Gareth, Thanks a lot for such a comprehensive writeup and the patch. Please give me a day or two to do the review. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Yury, let me see if I can move this issue forward. I clearly haven't done a good job of explaining these problems, how they are related, and why it makes sense to solve them together, so let me have a go now. 1. tokenize.untokenize() raises AssertionError if you

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees g...@garethrees.org: -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___ ___

[issue12691] tokenize.untokenize is broken

Yury Selivanov added the comment: bump? -- nosy: +yselivanov versions: +Python 3.4 -Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___

[issue12691] tokenize.untokenize is broken

Meador Inge added the comment: I will take a look. As it stands the current patch fixes way too many issues. Patches should fix *one* issue at a time. I will look at fixing the original assert problem and at opening new issues for the others (assuming there aren't already issues for them).

[issue12691] tokenize.untokenize is broken

Thomas Kluyver added the comment: Is there anything I can do to push this forwards? I'm trying to use tokenize and untokenize in IPython, and for now I'm going to have to maintain our own copies of it (for Python 2 and 3), because I keep running into problems with the standard library module.

[issue12691] tokenize.untokenize is broken

Changes by Meador Inge mead...@gmail.com: -- nosy: +meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___ ___ Python-bugs-list

[issue12691] tokenize.untokenize is broken

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___ ___

[issue12691] tokenize.untokenize is broken

Changes by Daniel Urban urban.dani...@gmail.com: -- nosy: +durban ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___ ___ Python-bugs-list

[issue12691] tokenize.untokenize is broken

Gareth Rees g...@garethrees.org added the comment: I think I can make these changes independently and issue two patches, one fixing the problems with untokenize listed here, and another improving tokenize. I've just noticed a third bug in untokenize: in full mode, it doesn't handle

[issue12691] tokenize.untokenize is broken

Gareth Rees g...@garethrees.org added the comment: Please find attached a patch containing four bug fixes for untokenize(): * untokenize() now always returns a bytes object, defaulting to UTF-8 if no ENCODING token is found (previously it returned a string in this case). * In compatibility

[issue12691] tokenize.untokenize is broken

Gareth Rees g...@garethrees.org added the comment: Thanks Ezio for the review. I've made all the changes you requested, (except for the re-ordering of paragraphs in the documentation, which I don't want to do because that would lead to the round-trip property being mentioned before it's

[issue12691] tokenize.untokenize is broken

New submission from Gareth Rees g...@garethrees.org: tokenize.untokenize is completely broken. Python 3.2.1 (default, Jul 19 2011, 00:09:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type help, copyright, credits or license for more information. import tokenize, io

[issue12691] tokenize.untokenize is broken

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti stage: - test needed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___

[issue12691] tokenize.untokenize is broken

Sandro Tosi sandro.t...@gmail.com added the comment: Hi Gareth, would you like to provide a patch to fix the bug you spotted and add the relative case into the testsuite? -- nosy: +sandro.tosi ___ Python tracker rep...@bugs.python.org

[issue12691] tokenize.untokenize is broken

Gareth Rees g...@garethrees.org added the comment: See my last paragraph: I propose to deliver a single patch that fixes both this bug and issue12675. I hope this is OK. (If you prefer, I'll try to split the patch in two.) I just noticed another bug in untokenize(): in compatibility mode, if

[issue12691] tokenize.untokenize is broken

Sandro Tosi sandro.t...@gmail.com added the comment: The general rule would be to have separate patches. But in this case, if we have interdipendent changes, then those should be packed in a single patch (f.e. if changes to tokenize break untokenize, than those parts should be joined).

[issue12691] tokenize.untokenize is broken

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +ericsnow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12691 ___ ___ Python-bugs-list