New submission from rb <[email protected]>:
When altering tokens and thus not providing token location information,
tokenize.untokenize sometimes misses out the first token. Failure case below.
Expected output: 'import foo ,bar\n'
Actual output: 'foo ,bar\n'
$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:43:55)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import StringIO, tokenize
>>>
>>> def strip(iterable):
... for t_type, t_str, (srow, scol), (erow, ecol), line in iterable:
... yield t_type, t_str
...
>>> source = StringIO.StringIO('import foo, bar\n')
>>> print
>>> repr(tokenize.untokenize(strip(tokenize.generate_tokens(source.readline))))
'foo ,bar \n'
>>> source.seek(0)
>>> print repr(tokenize.untokenize(tokenize.generate_tokens(source.readline)))
'import foo, bar\n'
>>>
----------
components: Library (Lib)
messages: 103799
nosy: rb
severity: normal
status: open
title: tokenize.untokenize first token missing failure case
versions: Python 2.6
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue8478>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
