[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: Thank you for your review Antoine. I have merged the PR since resolving issue24340 allowed to simplify the code for the stack depth calculation. -- resolution: -> fixed stage: patch review -> resolved status: open ->

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: New changeset 782d6fe4434381c50e0c7ec94a1ef9c6debbc333 by Serhiy Storchaka in branch 'master': bpo-31113: Get rid of recursion in the compiler for normal control flow. (#3015)

[issue31113] Stack overflow with large program

Change by Serhiy Storchaka : -- assignee: -> serhiy.storchaka dependencies: +co_stacksize estimate can be highly off ___ Python tracker

[issue31113] Stack overflow with large program

Change by Stefan Nordhausen : -- nosy: +snordhausen ___ Python tracker ___

[issue31113] Stack overflow with large program

Ned Deily added the comment: I agree with Antoine's comment on the PR: this seems like this should only go into 3.7. From the description here, it sounds like this is an edge-case problem that hasn't come up before as an issue. Let's do the right thing in master (for 3.7) and try to come up

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: Ned, Benjamin, what do you think about backporting this change to 3.6 and 2.7? On one hand, the change is not trivial and looks like a new feature. On other hand, this can be considered as a platform specific bug, the compiler works with a large generated

[issue31113] Stack overflow with large program

Changes by Yury Selivanov : -- nosy: -yselivanov ___ Python tracker ___ ___

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: Updated patch makes stackdepth() not consuming the C stack for recursion. The new code is not strictly equivalent to the old code, but I think they should produce the same results in common cases (the existing stack depth calculation algorithm is not free

[issue31113] Stack overflow with large program

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: > The PR resolved the stack overflow in dfs(), however it now fails in the > stackdepth() routine (technically, the stackdepth_walk() helper). Indeed. The compiler crashes for longer sequences on Linux too. I'm trying to rewrite stackdepth_walk() too, but

[issue31113] Stack overflow with large program

Eryk Sun added the comment: Here are a couple of workarounds for the crash in Windows. The default stack reservation size is a field in the PE/COFF header, which you can edit using editbin.exe, e.g.: editbin /stack:[size_in_bytes] "path\to\python.exe" The distributed python.exe has a

[issue31113] Stack overflow with large program

Brett Cannon added the comment: One fix at a time.  On Mon, Aug 7, 2017, 07:52 Jeremy Kloth, wrote: > > Jeremy Kloth added the comment: > > The PR resolved the stack overflow in dfs(), however it now fails in the > stackdepth() routine (technically, the

[issue31113] Stack overflow with large program

Jeremy Kloth added the comment: The PR resolved the stack overflow in dfs(), however it now fails in the stackdepth() routine (technically, the stackdepth_walk() helper). -- ___ Python tracker

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: PR 3015 gets rid of recursion for normal control flow in the compiler. This fixes a stack overflow for this case. -- ___ Python tracker

[issue31113] Stack overflow with large program

Changes by Serhiy Storchaka : -- pull_requests: +3048 ___ Python tracker ___ ___

[issue31113] Stack overflow with large program

Antoine Pitrou added the comment: > exec('if a: b = 1\n' * 25150) I need to increase it to 20 and it fails with a stack overflow in dfs() here (git master on Ubuntu 16.04 in pydebug mode). -- ___ Python tracker

[issue31113] Stack overflow with large program

Changes by Antoine Pitrou : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list

[issue31113] Stack overflow with large program

Jeremy Kloth added the comment: Using master to debug, the (first) offending part of the generated file is the get_match_iter() function. The problem is not that there is too much nesting, rather it is simply the fact of too many if's period. Simple testing at the command prompt (using

[issue31113] Stack overflow with large program

Manuel Krebber added the comment: @Serhiy That would require me to compile Python myself though, right? Is there a reason why the limit is only for try/for and not for if? @Antoine Well, the goal is to be able to generate Python 2 compatible code . I will try to split the code into more

[issue31113] Stack overflow with large program

Antoine Pitrou added the comment: By the way, since you're using Python 3, you can probably workaround this issue by delegating some of the work to helper functions using `yield from`. -- ___ Python tracker

[issue31113] Stack overflow with large program

Serhiy Storchaka added the comment: I concur with Stefan. Some parts of the compiler are recursive. The crash is expected for enough complex programs, and the size of C stack is platform depended. There are few hard-coded limits (MAXINDENT, CO_MAXBLOCKS) that may prevent the crash by

[issue31113] Stack overflow with large program

Terry J. Reedy added the comment: We know that compile has undocumented size limits of various sorts that are usually more than sufficient for normal human written code but which can be overwhelmed by machine-generated code. We do not regard this as a bug. However, 20 levels of if-nesting

[issue31113] Stack overflow with large program

Antoine Pitrou added the comment: I'm not an expert in the compiler myself. -- nosy: +benjamin.peterson, brett.cannon, ncoghlan, serhiy.storchaka, yselivanov ___ Python tracker

[issue31113] Stack overflow with large program

Manuel Krebber added the comment: I have already tried to reduce the nesting, but it still crashes. I have to admit that ~20 levels of nesting are still quite a lot. But I am surprised that so few levels of nesting already are a problem for the parser... I have attached the generated code

[issue31113] Stack overflow with large program

Stefan Behnel added the comment: Regarding the user side of the problem, you might(!) be able to work around the crash by merging nested if-conditions into and-expressions if they have no elif/else. That's also why the split into multiple files doesn't help, it's the depth of the nesting and

[issue31113] Stack overflow with large program

Stefan Behnel added the comment: I've looked at the file and it contains a huge amount of deeply nested if-statements. Given that parsers and compilers are typically recursive, I can well imagine that this is a problem, and my guess is that it's most likely just the different C level stack

[issue31113] Stack overflow with large program

Manuel Krebber added the comment: 1) Yes. 2) A .pyc file was not generated. 3) It is always the same location where the error occurs. 4) It did not crash on the linux machine that I tested it on. I have tried to split the code up into multiple files, but it still crashes. I have uploaded the

[issue31113] Stack overflow with large program

Stefan Behnel added the comment: 1) Is this reproducible? 2) Getting a crash in compile.c indicates that this is happening at parse/compile time and not when your Python code is executing. Can you confirm that? Does it generate a .pyc file on import that would indicate a successful byte code

[issue31113] Stack overflow with large program

New submission from Manuel Krebber: With a pattern matching library I am generating some Python code that matches patterns. For a very big pattern set I generate a Python file which is about 20MB and has ~300K LOC. When I try to execute the file with Python 3.6.2 on Windows 10 (64bit), the