[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

3.6 only accepts security fixes at this point.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[A. Skrobov]

A. Skrobov  added the comment:

Is it intentional that the fix is not backported to 3.6 as well?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 00eb97b4a7d9a73b88ed7c76faee4e49204d5a00 by Pablo Galindo (Miss 
Islington (bot)) in branch '3.7':
bpo-36256: Fix bug in parsermodule when parsing if statements (GH-12488)
https://github.com/python/cpython/commit/00eb97b4a7d9a73b88ed7c76faee4e49204d5a00


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[miss-islington]

Change by miss-islington :


--
pull_requests: +12440

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 9ad15d27361eaa47b77600c7c00a9787a894 by Pablo Galindo in 
branch 'master':
bpo-36256: Fix bug in parsermodule when parsing if statements (GH-12477)
https://github.com/python/cpython/commit/9ad15d27361eaa47b77600c7c00a9787a894


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +12429
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Brett Cannon]

Brett Cannon  added the comment:

@Xavier different needs; AST and CST are at different stages of compilation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Xavier Combelle]

Xavier Combelle  added the comment:

never used the parser module nor lib2to3. Does they have any advantage over 
ast.parse and ast module ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Brett Cannon]

Brett Cannon  added the comment:

It's sounding like it might be worth the effort then to make lib2to3's parser 
not be a "hidden" thing in lib2to3, break it out as a new parser module (I have 
no stance on name), and then deprecate the old parser module. I think this was 
discussed at the last language summit when Christian proposed deprecating 
lib2to3 and everyone said the parser is too useful to lose.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
nosy:  -giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[A. Skrobov]

A. Skrobov  added the comment:

> The major problem with the parser module is that is unsynchronized with the 
> actual parser

The parser module is "sort of" synchronised with the actual parser, in that it 
uses the same _PyParser_Grammar; this doesn't mean they always behave the same, 
as this bug shows :-)

(But before #26526, it used to be much worse, with the parser module having a 
completely independent re-implementation of the parser.)

> As a side note, the problem described in this bug was more or less foreseen. 
> This is in the header of Modules/parsemodule.c:

Just to clarify, the bug is not about the cryptic exception message, it's about 
the exception firing when it shouldn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I would be curious to hear what Pablo has to say with the new parser having 
> landed and if there's something we should be exposing from that code to 
> replace what's in 'parser' today? (Either w/o semantic change or a new API.)

:)

One small clarification is that the parser is the same, what has changed is the 
parser generator. What is exposed in the parser modules today is the parse 
trees (in a very raw form). 

One thing we can do is expose the parser component that lib2to3/pgen2 has as a 
substitute/complement to the parser module (which is not exposed as part of the 
new pgen - I know, is confusing). This is very useful and complementary to the 
AST (for example, black is using a forked version of this component to obtain 
the CST as it can do round tripping - code->CST->NEW_CST->code). This piece is 
in pure Python and can read the parser tables that pgen generates. It also will 
have the advantage of forcing us to synchronize to the current grammar (black 
had to fork it among other things because the one in lib2to3 was out of date). 
This idea and all the challenges are already been discussed here:

https://bugs.python.org/issue7

The major problem with the parser module is that is unsynchronized with the 
actual parser, it has a very raw API and is moderately unmaintained (as this 
bug reveals). We would need to evaluate if we want to spend effort into 
synchronizing them, deprecating completely the parser module, substitute it 
with a new python version or wait until we have a completely new non-LL(1) C 
parser to ask these questions.

What do you think?

As a side note, the problem described in this bug was more or less foreseen. 
This is in the header of Modules/parsemodule.c:

*  To debug parser errors like
*  "parser.ParserError: Expected node type 12, got 333."
*  decode symbol numbers using the automatically-generated files
*  Lib/symbol.h and Include/token.h.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[Brett Cannon]

Brett Cannon  added the comment:

I would be curious to hear what Pablo has to say with the new parser having 
landed and if there's something we should be exposing from that code to replace 
what's in 'parser' today? (Either w/o semantic change or a new API.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36256] parser module fails on legal input

[A. Skrobov]

New submission from A. Skrobov :

Under #26526, I had optimised the validation code in parser module to use the 
actual parser DFAs, but my code considers only the token types in the input, 
and doesn't distinguish between different NAMEs (e.g. different keywords).

The result is this:

Python 3.7.0 (default, Aug 22 2018, 20:50:05) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import parser
>>> parser.sequence2st(parser.suite("if True:\n pass\nelse:\n pass").totuple())
Traceback (most recent call last):
  File "", line 1, in 
parser.ParserError: Expected node type 305, got 11.

The fix for this bug is quite simple, and in fact, it had been languishing for 
2.5 years under #26415

I can easily enough extract the fix into a PR of its own, but the bigger 
question is: parser module had been advised against using since Python 2.5; now 
that it has been broken for 2.5 years, nobody even noticed. (if-else must be 
quite a common code construct, so anybody trying to use the module would have 
noticed!) So, should perhaps the module be discontinued rather than fixed?

--
components: Extension Modules
messages: 337619
nosy: A. Skrobov, benjamin.peterson, berker.peksag, brett.cannon, fdrake, 
giampaolo.rodola, gregory.p.smith, pablogsal, python-dev, serhiy.storchaka, 
xcombelle
priority: normal
severity: normal
status: open
title: parser module fails on legal input
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com