Serhiy Storchaka <storchaka+cpyt...@gmail.com> added the comment:
I am surprised, that removing the newline character adds a token:
>>> pprint.pprint(list(tokenize.generate_tokens(io.StringIO('#\n').readline)))
[TokenInfo(type=55 (COMMENT), string='#', start=(1, 0), end=(1, 1), line='#\n'),
TokenInfo(type=56 (NL), string='\n', start=(1, 1), end=(1, 2), line='#\n'),
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
>>> pprint.pprint(list(tokenize.generate_tokens(io.StringIO('#').readline)))
[TokenInfo(type=55 (COMMENT), string='#', start=(1, 0), end=(1, 1), line='#'),
TokenInfo(type=56 (NL), string='', start=(1, 1), end=(1, 1), line='#'),
TokenInfo(type=4 (NEWLINE), string='', start=(1, 1), end=(1, 2), line=''),
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
----------
nosy: +serhiy.storchaka
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue35107>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
