New submission from Eric Snow: When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility mode, where only the token type and string are used from that point forward. There are two closely related problems:
* when the iterable is a sequence, the portion of the sequence prior to the 2-tuple is traversed a second time under compatibility mode. * when the iterable is an iterator, the first 2-tuple encountered is essentially gobbled up (see issue16221). Either an explicit "iterable = iter(iterable)" or "iterable = list(iterable)" should happen at the very beginning of Untokenizer.untokenize(). If the former, Untokenizer.compat() should be fixed to not treat that first token differently. If the latter, self.tokens should be cleared at the beginning of Untokenizer.compat(). I'll put up a patch with the second option when I get a chance. ---------- assignee: eric.snow components: Library (Lib) messages: 172851 nosy: eric.snow priority: normal severity: normal stage: test needed status: open title: tokenize.untokenize() misbehaves when moved to "compatiblity mode" type: behavior versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16224> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
