Terry J. Reedy added the comment: I read the manual more carefully and noticed that the guarantee is that tokenizing the result of untokenize matches the input to untokenize. " The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured."
I ran the test in 3.4 and noticed the exactness and direction of the matching does not matter because only the 2nd \t is replaced by a space, making the invalid code that raises IndentationError. So this is definitely a bug and I withdraw the close suggestion. I believe the test should be that both lines of the body begin with the same indent. from tokenize import tokenize, untokenize import io def test_tab_indent(self): code = b"if False:\n\tx=3\n\ty=3\n" codelines = untokenize(tokenize(io.BytesIO(s).readline)).split(\n) assertEqual(codelines, codelines) Of course, the test of tokenize and untokenize should be separated. ---------- stage: -> needs patch versions: +Python 3.3 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue20387> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com