Meador Inge <[email protected]> added the comment:
That syntax error is coming from the CPython parser and *not* the tokenizer.
Both CPython and the 'tokenizer' modules produce the same tokenization:
[meadori@motherbrain cpython]$ cat repro.py
if 1:
\
pass
[meadori@motherbrain cpython]$ ./python tokenize.py repro.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,2: NAME 'if'
1,3-1,4: NUMBER '1'
1,4-1,5: OP ':'
1,5-1,6: NEWLINE '\n'
2,0-2,2: INDENT ' '
3,0-3,1: NEWLINE '\n'
4,2-4,6: NAME 'pass'
4,6-4,7: NEWLINE '\n'
5,0-5,0: DEDENT ''
5,0-5,0: ENDMARKER ''
[44319 refs]
[meadori@motherbrain cpython]$ ./python -d repro.py | grep Token | tail -10
File "repro.py", line 3
^
SyntaxError: invalid syntax
[44305 refs]
Token NEWLINE/'' ... It's a token we know
Token DEDENT/'' ... It's a token we know
Token NEWLINE/'' ... It's a token we know
Token ENDMARKER/'' ... It's a token we know
Token NAME/'if' ... It's a keyword
Token NUMBER/'1' ... It's a token we know
Token COLON/':' ... It's a token we know
Token NEWLINE/'' ... It's a token we know
Token INDENT/'' ... It's a token we know
Token NEWLINE/'' ... It's a token we know
The NEWLINE INDENT NEWLINE tokenization causes the parser to choke because
'suite' nonterminals:
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
are defined as NEWLINE INDENT.
It seems appropriate that the NEWLINE after INDENT should be dropped by both
tokenizers. In other words, I think:
"""
if 1:
\
pass
"""
should produce the same tokenization as:
"""
if 1:
pass
"""
This seems consistent with with how explicit line joining is defined [2].
[1] http://hg.python.org/cpython/file/92842e347d98/Grammar/Grammar
[2] http://docs.python.org/reference/lexical_analysis.html#explicit-line-joining
----------
stage: test needed -> needs patch
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue2180>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
