New submission from Jason R. Coombs:

Consider this simple unit test:

def test_tokenize():
    input = "if False:\n\tx=3\n\ty=3\n"
    g = list(generate_tokens(io.StringIO(input).readline))
    assert untokenize(g) == input

According to the docs, untokenize guarantees the output equals the input to 

As this test fails on Python 2 and Python 3 (use io.BytesIO on Python 2), I 
believe the test illustrates a violation of this guarantee. The second line in 
the if block gets its tab replaced by a space.

As the input is valid Python Syntax, this behavior is particularly surprising.

messages: 209137
nosy: jason.coombs
priority: normal
severity: normal
status: open
title: tokenize/untokenize roundtrip fails with tabs
versions: Python 2.7, Python 3.4

Python tracker <>
Python-bugs-list mailing list

Reply via email to