[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

I am not backporting to 3.9 because the parser is different enough that 
introducing this would also introduce some unintended side effects.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 5b58db75291cfbb9b6785c9845824b3e2da01c1c by Pablo Galindo Salgado 
in branch '3.10':
[3.10] bpo-46521: Fix codeop to use a new partial-input mode of the parser 
(GH-31010). (GH-31213)
https://github.com/python/cpython/commit/5b58db75291cfbb9b6785c9845824b3e2da01c1c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +29383
pull_request: https://github.com/python/cpython/pull/31213

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 69e10976b2e7682c6d57f4272932ebc19f8e8859 by Pablo Galindo Salgado 
in branch 'main':
bpo-46521: Fix codeop to use a new partial-input mode of the parser (GH-31010)
https://github.com/python/cpython/commit/69e10976b2e7682c6d57f4272932ebc19f8e8859


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> The message is off

That's because the tokenizer sees the error before the parser even has time to 
see the other one. Not sure if is technically anything to fix here other than 
the order of reporting two different errors, which may be a bit tricky to fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Terry J. Reedy]

Terry J. Reedy  added the comment:

With my fix to the PR:

>>> a b '''
SyntaxError: unterminated triple-quoted string literal (detected at line 1)
>>> a '''
...

The message is off, and can be left for another issue (or not), but the 
behavior is correct.

With the hack removed, all the tests in test_codeop that the hack works can be 
removed and replaced by a simple test that a code object, error, or None are 
returned in 3 different cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Thank you for the PR. As I wrote on my preliminary review, I see this likely 1 
failure that might may be in the test fixture. Will test and debug later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

If we want to go with this approach, I am going to need help to fix test_idle 
as I have no idea why is failing if test_codeop passes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Ugh, the approach to do that breaks super heavily test_idle :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +29189
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/31010

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

>> Pablo, is there any possibility that the internal REPL parser could be 
>> wrapped, exposed to Python, and called with fake stdin/out/err objects?

I would really advise against this. Unfortunately, the state of affairs is that 
the REPL is somehow super entangled with the parser, to the point that is the 
tokenizer asking for more characters that triggers a read from stdin. Exposing 
this with some API would be super dangerous because we would be elevated to the 
"supported" level anything that works just because it happens to be close to 
what the interactive mode needs.

On the other side, codeop is fundamentally flawed in the way is built because 
relies on comparing error messages from the parser, which is a very poor way of 
handling semantic information.

What we need here is a mode of the parser that somehow raises a very specific 
error on incomplete input, but not an incorrect one. And that may be not 
immediate to implement given in how many places we can raise lexer and parser 
errors. I will give it a go, in any case

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46521] codeop._maybe_compile passes code with error + triple quotes

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Tushar, I am the principle IDLE maintainer and have dealt with similar 
interactive compile problems on and off for a few years. Code module uses 
codeop._maybe_compile, which in turn uses batch-mode compile().

To review: Code can be in 3 states: legal, illegal (there is an positive error 
that must be fixed), and incomplete (might become legal with more input).  
Batch mode compile has all the code it is going to get, so it raises an 
exception for both bad and incomplete code.

In command-line languages, '\n' definitely signals end-of-command. Even in 
interactive mode, incomplete commands get an error message. The user must 
retype or recall and add more.

Python is a multiline and compound statement language.  Newline may instead 
signal the end of a compound statement header or the end of a nested statement, 
or even just be present for visual formatting.  Being able to continue 
incomplete statements after newline is essential.

In interactive mode, the interpreter looks at code after each newline and 
differentiates between unrecoverable and merely incomplete errors.  In the 
latter case, it sends a prompt to enter more and then reads more.

codeop._maybe compile attempts to simulate interactive mode and make a trinary 
decision (returning None for 'incomplete') using batch-mode binary compile().  
A hack using repeated compiles, a warning filter, and a helper function 
classifies code that failed the initial compile.  I suspect that a) it has 
never been perfect and b) it cannot be (see experiment below).

The issue here is that _maybe_compile returns None instead of passing on the 
compile syntax error.  Some debug prints would reveal exactly why.

An alternate approach might be to compile just once and use the error message 
and marked error range to split.  But that would require different 
message-range pairs for the different cases.  Compile does not seem to give 
this to us.  For the current case, 3.11.a04 compile (and ast.parse) give the 
same response to both bad and incomplete lines.

>>> compile("a b '''", '', 'single')
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
a b '''
^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)
>>> compile("s='''", '', 'single')
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1
s='''
  ^
SyntaxError: unterminated triple-quoted string literal (detected at line  1

But the REPL somehow treats the two lines differently when directly entered.

>>> s = '''
...
... '''
>>> a b '''
  File "", line 1
a b '''
^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)

Pablo, is there any possibility that the internal REPL parser could be wrapped, 
exposed to Python, and called with fake stdin/out/err objects?

Or if one executes 'python -i -c "start code"', what is required of the 
standard streams for '-i' to be effective and actually shift into interactive 
mode, reading from and writing to those streams?  Just claim to be a tty?  It 
is any different than for reading responses to "input('prompt')"?

--
nosy: +terry.reedy
title: compile_command not raising syntax error when command ends with triple 
quotes -> codeop._maybe_compile passes code with error + triple quotes
versions: +Python 3.11 -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com