Terry J. Reedy added the comment:

I read the manual more carefully and noticed that the guarantee is that 
tokenizing the result of untokenize matches the input to untokenize. " The 
result is guaranteed to tokenize back to match the input so that the conversion 
is lossless and round-trips are assured."

I ran the test in 3.4 and noticed the exactness and direction of the matching 
does not matter because only the 2nd \t is replaced by a space, making the 
invalid code that raises IndentationError.

So this is definitely a bug and I withdraw the close suggestion.

I believe the test should be that both lines of the body begin with the same 

from tokenize import tokenize, untokenize
import io

    def test_tab_indent(self):
        code = b"if False:\n\tx=3\n\ty=3\n"
        codelines = untokenize(tokenize(io.BytesIO(s).readline)).split(\n)
        assertEqual(codelines[1], codelines[2])

Of course, the test of tokenize and untokenize should be separated.

stage:  -> needs patch
versions: +Python 3.3

Python tracker <rep...@bugs.python.org>
Python-bugs-list mailing list

Reply via email to