[issue32758] Stack overflow when parse long expression to AST

Change by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: New changeset dedb99acdb5bbc179872235f975248133d3fb440 by Serhiy Storchaka (Ashley Whetter) in branch '2.7': bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960) (GH-16565)

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: New changeset 8eb27cc35489848596d9fb4b1c91fac00ae75d21 by Serhiy Storchaka (Ashley Whetter) in branch '2.7': bpo-32758: Warn that compile() can crash when compiling to an AST object (GH-6043) (GH-16566)

[issue32758] Stack overflow when parse long expression to AST

Change by Roundup Robot : -- pull_requests: +16157 pull_request: https://github.com/python/cpython/pull/16566 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Change by Roundup Robot : -- keywords: +patch pull_requests: +16156 pull_request: https://github.com/python/cpython/pull/16565 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Karthikeyan Singaravelan added the comment: This seems like an easy issue that the warnings in docs for ast need to be manually backported to 2.7 since miss Islington cannot cherry pick two PRs for 2.7 in https://bugs.python.org/issue32758#msg313511 . -- nosy: +Mariatta,

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: - [x] ast module for Python 3 - [x] compile() for Python 3 - [x] dbm.dumb.open() - [ ] ast module for Python 2 (see https://github.com/python/cpython/pull/5960) - [ ] compile() for Python 2 (see https://github.com/python/cpython/pull/6043) At

[issue32758] Stack overflow when parse long expression to AST

Change by miss-islington : -- pull_requests: +5809 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Change by Brett Cannon : -- pull_requests: +5808 ___ Python tracker ___ ___

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: @Serhiy: Correct, which is what the warning says: https://github.com/python/cpython/pull/6043/files . -- ___ Python tracker

[issue32758] Stack overflow when parse long expression to AST

Change by Yury Selivanov : -- nosy: -yselivanov ___ Python tracker ___ ___

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: In 3.7 compile() can crash not only when compiling to an AST object (due to recursive AST optimization). -- ___ Python tracker

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: Thanks for the feedback, Serhiy! Based on that, the new TODO list is: - [x] ast module for Python 3 - [x] compile() for Python 3 - [ ] dbm.dumb.open() - [ ] ast module for Python 2 (see https://github.com/python/cpython/pull/5960) - [ ]

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: I think we can ignore the inspect module. It is unlikely that it will cause a crash unintentionally, and it is hard to use this for attacks. The attacker needs to create an extension function with malicious __text_signature__,

[issue32758] Stack overflow when parse long expression to AST

Change by miss-islington : -- pull_requests: +5807 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Change by miss-islington : -- pull_requests: +5806 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Change by Brett Cannon : -- pull_requests: +5804 ___ Python tracker ___ ___

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: Actually, the TODO list is: - [x] ast module for Python 3 - [ ] compile() - [ ] eval() for >= 3.7 - [ ] exec() for >= 3.7 - [ ] dbm.dumb.open() - [ ] inspect - [ ] ast module for Python 2 (see https://github.com/python/cpython/pull/5960)

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: I have the changes in for Python 3 for the ast module. Updated TODO list: - [x] ast module - [ ] compile() - [ ] eval() for >= 3.7 - [ ] exec() for >= 3.7 - [ ] dbm.dumb.open() - [ ] inspect -- ___

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: New changeset f2fffd41b42d88fe36b483852ae33d5a415b7082 by Brett Cannon (Miss Islington (bot)) in branch '3.7': bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960) (GH-6041)

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: New changeset b316c44b0105d11a80ff971636143735f3655bbf by Brett Cannon (Miss Islington (bot)) in branch '3.6': bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960) (GH-6042)

[issue32758] Stack overflow when parse long expression to AST

Change by miss-islington : -- pull_requests: +5803 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Change by miss-islington : -- pull_requests: +5802 ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: New changeset 7a7f100eb352d08938ee0f5ba59c18f56dc4a7b5 by Brett Cannon in branch 'master': bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960)

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: You're probably right and it's worth propagating the warning a bit wider. -- assignee: docs@python -> brett.cannon ___ Python tracker

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: Thank you Brett! The comment LGTM. Is it worth to add warnings to other functions? * compile(), exec() and eval(). They are crashed due to recursion in the AST optimizer. This is a regression of 3.7. compile(..., PyCF_ONLY_AST)

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: The PR adds the documentation warnings. Serhiy, can you double-check that I have the appropriate functions and the comment is acceptable? -- ___ Python tracker

[issue32758] Stack overflow when parse long expression to AST

Change by Brett Cannon : -- keywords: +patch pull_requests: +5728 stage: needs patch -> patch review ___ Python tracker ___

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: A consequence of this is that ast.literal_eval() can crash. >>> import ast >>> ast.literal_eval("+0"*20) Segmentation fault (core dumped) It should be documented that ast.literal_eval() is not safe. --

[issue32758] Stack overflow when parse long expression to AST

Change by Serhiy Storchaka : -- assignee: -> docs@python components: +Documentation -Interpreter Core nosy: +docs@python stage: -> needs patch ___ Python tracker

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: On Sun, Feb 11, 2018, 16:26 Serhiy Storchaka, wrote: > > Serhiy Storchaka added the comment: > > > The other option is to simply not worry about it and acknowledge you can > > crash the

[issue32758] Stack overflow when parse long expression to AST

Terry J. Reedy added the comment: If ast_parse returns, a correct tree (A) rather than a buggy tree is a hard requirement. If ast_parse does not return, an exception (B) rather than a crash is strongly desired. We should not risk A to get B. I presume that Serhiy is

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: > The other option is to simply not worry about it and acknowledge you can > crash the compiler with a crazy-sized expression. Agreed, this is the most practical option. Do you want to write such acknowledgement? --

[issue32758] Stack overflow when parse long expression to AST

Brett Cannon added the comment: The other option is to simply not worry about it and acknowledge you can crash the compiler with a crazy-sized expression. Option 1 is too much work and option 2 takes us from an AST to more of an s-expression format which is a significant shift

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: On Linux the limit is much larger that 2**15. This depends on the stack size, it is smaller on Windows. The stack is overflowed by recursive call of ast2obj_expr in Python/Python-ast.c. The same problem exists in other

[issue32758] Stack overflow when parse long expression to AST

Terry J. Reedy added the comment: Ditto as to the limit for ast.parse. >>> import ast; ast.parse('+chr(33)'*32000) RESTART: Shell = >>> import ast; ast.parse('+chr(33)'*31000) <_ast.Module object

[issue32758] Stack overflow when parse long expression to AST

Terry J. Reedy added the comment: Experimenting on Windows with IDLE's current 3.7 Shell, where a user process crash restarts Shell, compile('+a'*31365, '?', 'eval') consistently gives RecursionError, values a bit larger sometimes crash, and values much larger (32000, at

[issue32758] Stack overflow when parse long expression to AST

Serhiy Storchaka added the comment: There is also a regression in 3.7 when compile a long expression. In 3.6: >>> compile('+a'*100, '?', 'eval') Traceback (most recent call last): File "", line 1, in RecursionError: maximum recursion depth exceeded during

[issue32758] Stack overflow when parse long expression to AST

New submission from Serhiy Storchaka : Python 2 can crash when compile long expression. >>> x = eval('""' + '+chr(33)'*10) Segmentation fault (core dumped) This was fixed in Python 3. RecursionError is raised now. >>> x = eval('""' + '+chr(33)'*10)