On Nov 14, 2019, at 13:23, MRAB <pyt...@mrabarnett.plus.com> wrote: > >> On 2019-11-14 19:51, Andrew Barnert via Python-ideas wrote: >> On Nov 14, 2019, at 11:21, Random832 <random...@fastmail.com> wrote: >>>> On Thu, Nov 14, 2019, at 13:12, Andrew Barnert wrote: >>>> And then you can run it on a whole mess of code and verify that it’s only >>>> different in the cases where you want it to be different (what used to be >>>> an ERRORTOKEN or NEWLINE is now an NL because we’re in the middle of a >>>> with compound statement header). >>> Maybe any compound statement header? e.g. if/while with lots of and/or >>> conditions, but more to the point it doesn't really make sense to make the >>> rule work differently for different types of statements. >> Sure. People don’t bring that up as much because with it and while you can >> already just use parens, but no reason they should be different. >> And that raises a point: the if keyword can appear in other places besides >> the start of a compound statement. Does tokenize.py have enough info to >> handle that properly? I don’t know, and the answer to that might be a good >> proxy to the question of whether it can be done in the real compiler without >> making parsing complicated, even if it won’t prove the answer either way. > Keywords (reserved words) are special everywhere except in strings and > comments.
Of course, but special isn’t sufficient here. The with token can only appear in one place: the start of a with statement. So, the tokenizer rule should be as simple as “when you emit a with token, set the multiline until colon flag”. That’s even simpler than the rule for the open parens token (which has to increment a counter). The if token can appear in three places: the start of an if statement, the middle of a comprehension, and the middle of a conditional expression. So, is there a tokenizer rule “when ???, set the multiline until colon flag”? It’s not just “when you emit an if token”. Maybe “when you emit an if token as the first token on a logical line” or something? I’m not sure. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OCV3JJTJKPE2UDABZUFIRXU7X3U7GWNN/ Code of Conduct: http://python.org/psf/codeofconduct/