New submission from Jason R. Coombs:
Consider this simple unit test:
def test_tokenize():
input = "if False:\n\tx=3\n\ty=3\n"
g = list(generate_tokens(io.StringIO(input).readline))
assert untokenize(g) == input
According to the docs, untokenize guarantees the output equals the input to
tokenize: http://docs.python.org/2/library/tokenize.html#tokenize.untokenize
As this test fails on Python 2 and Python 3 (use io.BytesIO on Python 2), I
believe the test illustrates a violation of this guarantee. The second line in
the if block gets its tab replaced by a space.
As the input is valid Python Syntax, this behavior is particularly surprising.
----------
messages: 209137
nosy: jason.coombs
priority: normal
severity: normal
status: open
title: tokenize/untokenize roundtrip fails with tabs
versions: Python 2.7, Python 3.4
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue20387>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
