[issue40246] Different error messages for same error - invalid string prefixes

Change by Lysandros Nikolaou : -- resolution: -> wont fix stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: New changeset 6cb0ad20396116b5076a58b05b55286d6d5e0c94 by Lysandros Nikolaou in branch '3.9': bpo-40246: Fix test_fstring when run with the old parser (GH-20402) https://github.com/python/cpython/commit/6cb0ad20396116b5076a58b05b55286d6d5e0c94

[issue40246] Different error messages for same error - invalid string prefixes

Change by Lysandros Nikolaou : -- pull_requests: +19664 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/20402 ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: "make install" ignores compileall errors no? If I recall correctly, only an error is logged into stdlib, "make install" doesn't fail. -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Terry J. Reedy added the comment: > The >else"#fca"< code was added 6 years ago (commit > 8b95d5e0bf00d9d0098579d29fd6bb9322071879) That was my typo in turtledemo.__main__ (similar lines before had the space) and I wish that it had been caught then. Why did compileall in the test suite

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: We should run the tests with the old parser in at least one build. -- ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: The following change broke test_fstring when using the old parser. I reopen the issue. commit 846d8b28ab9bb6197ee81372820311c0abe509c0 (refs/bisect/bad) Author: Lysandros Nikolaou Date: Mon May 4 14:32:18 2020 +0300 bpo-40246: Revert reporting of

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: The error can be seen on the new cool AMD64 Arch Linux VintageParser 3.x, vintage is the new cool: https://buildbot.python.org/all/#builders/648/builds/185 -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: Ok, I'm closing this, after consulting with Guido. -- resolution: -> wont fix stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: Lysandros: > The revert is in. Now the question is if we want to take additional action to > address the original issue of this. If someone cares of that, I suggest to open an issue in pylint, pyflakes and similar tools to emit a warning in linters.

[issue40246] Different error messages for same error - invalid string prefixes

Miro Hrončok added the comment: Not that many: cpython itself (fixed via PR) demjson (fixed via PR) asn1crypto (fixed via PR) dnf (fixed via PR) freeipa (fixed via PR) I gave up sending PRs at this point. waf weasyprint virt-who thrift salt wxpython4 rosdistro mne pycairo libstoragemgmt

[issue40246] Different error messages for same error - invalid string prefixes

Miro Hrončok added the comment: I will soon come back with what Fedora package were affected by the problem. That could give some data about how to handle this. -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: The revert is in. Now the question is if we want to take additional action to address the original issue of this. -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: New changeset 846d8b28ab9bb6197ee81372820311c0abe509c0 by Lysandros Nikolaou in branch 'master': bpo-40246: Revert reporting of invalid string prefixes (GH-19888) https://github.com/python/cpython/commit/846d8b28ab9bb6197ee81372820311c0abe509c0

[issue40246] Different error messages for same error - invalid string prefixes

Change by Lysandros Nikolaou : -- pull_requests: +19199 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/19888 ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

Miro Hrončok added the comment: We also get: File "/usr/lib/python3.9/site-packages/dnf/comps.py", line 111 return'.'.join(lcl) ^ SyntaxError: invalid string prefix -- nosy: +hroncok ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Serhiy Storchaka added the comment: The original issue was about different error messages in REPL and eval(). But it is not related to string prefixes. We have the same difference without involving strings: >>> a b File "", line 1 a b ^ SyntaxError: invalid syntax >>> eval("a

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: Indeed, reverting commit 41d5b94af44e34ac05d4cd57460ed104ccf96628 makes it work with both parsers: ~/github/python/master master* ❯ ./python -X oldparser Python 3.9.0a6+ (heads/master-dirty:84724dd239, Apr 29 2020, 20:29:53) [GCC 9.3.0] on linux Type

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: > If it's all handled by the tokenizer, how come it's different in the > newparser? Is not different in the new parser: both parsers have analogous behaviour now: ~/github/python/master master ❯ ./python Python 3.9.0a6+ (heads/master:84724dd239, Apr

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: If it's all handled by the tokenizer, how come it's different in the new parser? Anyway, it seems we really have to support what the old parser supported here. I don't know exactly what rules it uses. Maybe only look for a string quote following a name that

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: > What's also possible is to handle keywords at tokenizer level Sadly, the tokenizer is unaware of keywords (maybe except async and await because their history as soft keywords) as that distinction usually is up to the parser as the parser is the one

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: What's also possible is to handle keywords at tokenizer level and return a different token type for each one of them (like with async and await), which is currently handled at parser level. This would enable us to allow reserved keywords right before a

[issue40246] Different error messages for same error - invalid string prefixes

Eric V. Smith added the comment: I think you're right, since rb obviously works, too. I just wanted to make sure we're covering all the bases. There's some code in either the stdlib or the test suite where I generate all of the valid prefixes (it's something like 80 prefixes). I can dig it

[issue40246] Different error messages for same error - invalid string prefixes

Serhiy Storchaka added the comment: I think it was a typo. You cannot combine anything with the u prefix. -- ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

Eric V. Smith added the comment: >From earlier in this issue: https://bugs.python.org/msg366164 > So a slightly shorter example uses ru''. This is an error because you can't > combine the r prefix and the u prefix (in fact you can't combine anything > with the r prefix). That's not true:

[issue40246] Different error messages for same error - invalid string prefixes

Petr Viktorin added the comment: reportlab reported failures on code like: norm=lambda m: m+(m and(m[-1]!='\n'and'\n'or'')or'\n') Note that `or` has a `r` in it. -- nosy: +petr.viktorin ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Serhiy Storchaka added the comment: > It's no different from other edge cases where for some reason the space seems > optional, like `1and 2`. The parser is not consistent here. `0or[]` is an error, while `0and[]` and `1or[]` are valid. See

[issue40246] Different error messages for same error - invalid string prefixes

Kyle Stanley added the comment: For addressing the backwards compatibility concern, I think we should just convert it to a SyntaxWarning for cases like the above to indicate that it's not really correct syntax, but not harmful enough to justify code breakage. I think it fits the documented

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: A possible solution would be to only emit a SyntaxError if the NAME directly preceding a STRING token contains one of the valid string prefixes (either one of 'f', 'r', 'u', 'b'). This would still output a nicer error message, but would not break code

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: I think that a linter can be very pedantic on such code. My concern is about backward compatibility. The >else"#fca"< code was added 6 years ago (commit 8b95d5e0bf00d9d0098579d29fd6bb9322071879) and nobody complained, it "just worked". If we keep the

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: Personally I don't think that `else"stuff"` should become a syntax error. It's no different from other edge cases where for some reason the space seems optional, like `1and 2`. -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: > I personally think the new behaviour is more consistent and more likely to > catch errors. Would it be possible to emit a SyntaxWarning in 3.9 and only emit a SyntaxError in 3.10? -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: I personally think the new behaviour is more consistent and more likely to catch errors. -- ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

STINNER Victor added the comment: I reopen the issue. The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431): clear = "NORMAL" print(dict(state=clear, bg="#d00" if clear else"#fca")) Well, the code looks

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: On bpo-40334 @vstinner reported that this change broke some code in turtledemo. The code looks like this: print(dict(state=clear, bg="#d00" if clear else"#fca")) Due to the absence of a space between `else` and `"`, the else keyword is now interpreted

[issue40246] Different error messages for same error - invalid string prefixes

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue40246] Different error messages for same error - invalid string prefixes

Pablo Galindo Salgado added the comment: New changeset 41d5b94af44e34ac05d4cd57460ed104ccf96628 by Lysandros Nikolaou in branch 'master': bpo-40246: Report a better error message for invalid string prefixes (GH-19476)

[issue40246] Different error messages for same error - invalid string prefixes

Change by Lysandros Nikolaou : -- keywords: +patch pull_requests: +18830 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19476 ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: yes On Sat, Apr 11, 2020 at 04:46 Lysandros Nikolaou wrote: > > Lysandros Nikolaou added the comment: > > I have working code that checks if there is a quotation mark right after > an identifier. Here is an example: > > ╰─ ./python > Python 3.9.0a5+

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: I have working code that checks if there is a quotation mark right after an identifier. Here is an example: ╰─ ./python Python 3.9.0a5+ (heads/pegen-dirty:502dfb719e, Apr 11 2020, 14:43:12) [GCC 9.2.1 20191008] on linux Type "help", "copyright",

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: In my understanding of the C code that's what the C tokenizer is doing as well. Here's the relevant snippet of the tokenizer (https://github.com/python/cpython/blob/4b222c9491d1700e9bdd98e6889b8d0ea1c7321e/Parser/tokenizer.c#L1358): When the tokenizer

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: But that is the tokenize.py module. It does not have 100% the same behavior as the builtin tokenizer.c module. In this case it probably doesn't. -- ___ Python tracker

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: At the moment, if the prefix is invalid, it gets tokenized as a NAME node and the string following is tokenized as a STRING node with no prefix. ╰─ echo "ur''" | ./python -m tokenize 1,0-1,2:NAME 'ur' 1,2-1,4:

[issue40246] Different error messages for same error - invalid string prefixes

Guido van Rossum added the comment: So a slightly shorter example uses ru''. This is an error because you can't combine the r prefix and the u prefix (in fact you can't combine anything with the r prefix). I declare that this is a bug to report EOF here and the code should *first* check

[issue40246] Different error messages for same error - invalid string prefixes

Lysandros Nikolaou added the comment: After some investigating, I found out that this is happening because when we execute `fur''` in the interactive interpreter there is an implicit newline, which means that EOF does not get reached. OTOH there is no newline when passing it as a string

[issue40246] Different error messages for same error - invalid string prefixes

Change by Lysandros Nikolaou : -- nosy: +gvanrossum, pablogsal ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue40246] Different error messages for same error - invalid string prefixes

Change by Karthikeyan Singaravelan : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue40246] Different error messages for same error - invalid string prefixes

New submission from Lysandros Nikolaou : While testing pegen, we found this out: Python 3.9.0a5+ (heads/pegen:502dfb719e, Apr 10 2020, 20:57:05) [GCC 9.2.1 20191008] on linux Type "help", "copyright", "credits" or "license" for more information. >>> fur'' File "", line 1 fur'' ^