[issue25079] Tokenize generates NL instead of NEWLINE for comments
Optimal BPM added the comment: Ok, I'll work around that then. Thanks for your response! -- ___ Python tracker <http://bugs.python.org/issue25079> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25079] Tokenize generates NL instead of NEWLINE for comments
Optimal BPM added the comment: OK. But just to make me understand, would that explain why an else statement gets its DEDENT before the else, instead of before an dedented comment just before it? Even if the comment is visually dedented as the else? Example where tokenize returns the DEDENT in a separate place than where it is visually: if a=b: statement # Comment else: other statement In the above case, the DEDENT is reportet before the else, instead of before the comment. -- ___ Python tracker <http://bugs.python.org/issue25079> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25079] Tokenize generates NL instead of NEWLINE for comments
New submission from Optimal BPM: Quoting the documentation: tokenize.NL Token value used to indicate a non-terminating newline. The NEWLINE token indicates the end of a logical line of Python code; NL tokens are generated when a logical line of code is continued over multiple physical lines. This doesn't seem to be entirely true. At the end of comments, an .NL, not .NEWLINE is generated: TokenInfo(type=55 (NL), string='\n', start=(5, 22), end=(5, 23), line='# Some docs for the IF\n') As a comment cannot extend over several lines, if would appear that a NEWLINE would be appropriate? -- components: Interpreter Core messages: 250516 nosy: Optimal BPM priority: normal severity: normal status: open title: Tokenize generates NL instead of NEWLINE for comments type: behavior versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue25079> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
