[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Raymond Hettinger added the comment: I think this particular issue can be closed in now. After it was written, the AST optimizer has been built-out and some peephole logic has moved. Other adjustments and improvements can be continue to be done in the other open issues. --

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Mateusz Bysiek : -- nosy: +mbdevpl ___ Python tracker ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: I created the issue #29471: "AST: add an attribute to FunctionDef to distinguish functions from generators and coroutines". -- ___ Python tracker

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: I submit new issues: * #29463 for AST change (change docstring from first statement to attribute). * #29469 for constant folding Note that this issue contains more peephole -> AST optimization changes. But I want to start these two patch to ease review and

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Alex Gaynor : -- nosy: -alex ___ Python tracker ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: Naoki: Can you please open a new issue for your ast-docstring.patch change? I like it, but this issue became too big, and I'm not sure that everyone in the nosy list is interested by this specific change. -- ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: I've tried to update ast_opt.c[t] without changing AST. But I can't find clear way to solve "foo" + "bar" docstring problem. This patch adds only docstring to AST. -- Added file: http://bugs.python.org/file46542/ast-docstring.patch

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Gregory P. Smith : -- nosy: -gregory.p.smith ___ Python tracker ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: INADA Naoki added the comment: > My motivation is improve speed, Ah, if the motivation is performance, I would like to see benchmark results :-) I understand that an AST optimizer would help to produce more efficient bytecode, right? > reduce memory usage, I

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: My motivation is improve speed, reduce memory usage, and quicker startup time for real world applications. If some optimization in FAT optimizer has significant speedup, I want to try it. But this time, my motivation is I felt "everyone think constant folding

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: Yes, doing optimizations on AST in CPython is unlikely to give any sizable speed improvements in real world programs. Python as a language is not suited for static optimization, and even if you manage to inline a function, there's still CPython's interpreted

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: At the AST level, you have a wide range of possible optimizations. See the optimizations that I implemented in fatoptimizer (FAT Python) to have an idea: http://fatoptimizer.readthedocs.io/en/latest/optimizations.html FAT Python adds guards checked at runtime,

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Brett Cannon added the comment: Dropping ast.Lit is fine. As for the docstring part, I'm torn. Yes it's nice as that will show up semantically in the Python code, but it's also easy to check for by just looking if the first statement is a Str (or Constant if that's what we make all strings).

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: Before trying advanced optimizations, I want move suspended obvious optimizations forwards. For example, removing unused constants is suspended because constant folding should be moved from peephole to AST. This is why I found this issue. After that, I'm

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: If you would like to implement constant folding at the AST level, I suggest you to look at my fatoptimizer project: https://github.com/haypo/fatoptimizer/blob/master/fatoptimizer/const_fold.py The tricky part is to avoid operations when we know that it will

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: >> We have already Constant and NameConstant. So it seems there are no need for >> None, Bool, TupleConst, SetConst nodes. > Yes, Constant is Victor's version of Lit. Then, may I remove ast.Lit, and use Constant and NameConstant? >> I think converting Num, Str,

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: > We have already Constant and NameConstant. So it seems there are no need for > None, Bool, TupleConst, SetConst nodes. Yes, Constant is Victor's version of Lit. > I think converting Num, Str, Bytes, Ellipsis into Constant in folding stage > is easier than

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: > 1. Changes to AST I'm working on updating this part. There are some failing tests remains. But I doubt this stage is worth enough for now. > a) Merge Num, Str, Bytes and Ellipsis constructors into a single Lit > (literal) that can hold anything. For one

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: Hugo, Serhiy, and Victor: I think you're all agreeing with each other, but to make sure I'm understanding the point correctly: 1. ast.literal_eval() is currently safe from malicious code like "10 ** 10" or "1073741824 * 'a'" because it only traverses

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: > Currently there is no a bug in ast.literal_eval() because the '**' operator > is not accepted. The doc says "This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Serhiy Storchaka added the comment: > Since the Python compiler doesn't produce ast.Constant, there is no change in practice in ast.literal_eval(). If you found a bug, please open a new issue. Currently there is no a bug in ast.literal_eval() because the '**' operator is not accepted. >>>

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Serhiy Storchaka added the comment: Good point Hugo. Yes, this should be taken into account when move constant folding to AST level. Thank you for the reminder. -- ___ Python tracker

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: Hugo Geoffroy added the comment: > I would like to point out that the changes in `ast.literal_eval` may have > some security risk for code that do not expect this function to return an > object with user-controlled length (for example, with `2**32*'X'`).

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Hugo Geoffroy added the comment: I would like to point out that the changes in `ast.literal_eval` may have some security risk for code that do not expect this function to return an object with user-controlled length (for example, with `2**32*'X'`). AFAIK, this is not possible with the current

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: > @haypo, how do you think about ast.Lit and ast.Constant? I already made two changes in Python 3.6 :-) I added ast.Constant to Python 3.6. While it's not used by .py=>AST compiler, it is understood by the AST->bytecode compiler (ex: handle correctly special

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

INADA Naoki added the comment: @haypo, how do you think about ast.Lit and ast.Constant? Is this patch updated to use ast.Constant? Or ast.Constant should be used only for some transform like constant folding? -- versions: +Python 3.7 -Python 3.6 ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by INADA Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: > I also see that Victor has been doing some of the same work, e.g. #26146. ast.Constant idea directly comes from your work. The implementatiln may ve different. It's a first step for AST optimizers. -- ___ Python

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: Serhiy: Nice! Yes, _PyCode_ConstantKey solved the problem. But #16619 went in the opposite direction of this patch, and introduced a new type of literal node instead of unifying the existing ones. Kind of a shame, since *this* patch, I believe, both fixes that

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Serhiy Storchaka added the comment: Yes, the patch is outdated, conflicts with current code (and would conflict even more after pushing the wordcode patch) and contains bugs. But it moved in right direction, I think your _PyCode_ConstantKey() could help to fix bugs. I'm going to revive this

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: Fairly sure it's 5 years old. -- ___ Python tracker ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: "issue11549.patch: serhiy.storchaka, 2016-05-11 08:22: Regenerated for review" diff -r 1e00b161f5f5 PC/os2emx/python33.def --- a/PC/os2emx/python33.defWed Mar 09 12:53:30 2011 +0100 +++ b/PC/os2emx/python33.defWed May 11 11:21:24 2016 +0300 The

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Serhiy Storchaka added the comment: Regenerated for review. -- nosy: +serhiy.storchaka Added file: http://bugs.python.org/file42810/issue11549.patch ___ Python tracker

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Nick Coghlan ncogh...@gmail.com: -- versions: +Python 3.6 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Berker Peksag berker.pek...@gmail.com: -- nosy: +berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: Just noting for the record (since it appears it was never brought back to the comments): it is expected that programs that manipulate the AST may require updates before they will work on a new version of Python. Preserving AST backwards compatbility is too

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: (That was the outcome of the suggested AST discussions on python-dev that were mentioned earlier) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Phil Connell pconn...@gmail.com: -- nosy: +pconnell ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Martin Morrison m...@ensoft.co.uk: -- nosy: +isoschiz ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Brett Cannon added the comment: http://bugs.python.org/issue17515 might also play into this if it happens to go in. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: I'm working on a AST optimizer for Python 2.6-3.3: https://bitbucket.org/haypo/astoptimizer It is implemented in Python and is able to optimize much more cases than the current bytecode peepholer. -- nosy: +haypo

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: All of the optimisations that assume globals haven't been shadowed or rebound are invalid in the general case. E.g. print(1.5) and print(1.5) are valid for *our* print function, but we technically have no idea if they're equivalent in user code. In short, if

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: All of the optimisations that assume globals haven't been shadowed or rebound are invalid in the general case. My main idea is that the developer of the application should be able to annotate functions and constants to declare them as optimizable (constant).

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

STINNER Victor added the comment: Any stdlib AST optimiser would need to be substantially more conservative by default. FYI The test suite of Python 2.7 and 3.3 pass with astoptimizer... except some minor (?) failures: * test_math fails for the reason explained above * test_pdb: it looks

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: No, you're assuming global program analysis and *that is not a valid assumption*. One of the key features of Python is that *monkeypatching works*. It's not encouraged, but it works. You simply cannot play games with name lookups like this without creating

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: You also have to be very careful of the interface to tracing functions, such as profilers and coverage analysis tools. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: Method calls on literals are always fair game, though (e.g. you could optimise a b c.split()) What about optimizations that do not change behavior, except for different error messages? E.g. we can change y = [1,2][x] to y = (1,2)[x] where the tuple is constant

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan added the comment: The peephole optimiser already makes optimisations like that in a couple of places (e.g. set - frozenset): def f(x): ...if x in {1, 2}: pass ... f.__code__.co_consts (None, 1, 2, frozenset({1, 2})) It's name lookup semantics that are the real minefield.

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder added the comment: If I'm not missing something, changing x in [1,2] to x in (1,2) and x in {1,2} to x in frozenset([1,2]) does not change any error messages. Agreed that without dynamic compilation we can pretty much only track literals (including functions and lambdas) assigned

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Terry J. Reedy added the comment: In msg132312 Nick asked where do we stand in regards to backwards compatibility of the AST? The current ast module chapter, second sentence, says The abstract syntax itself might change with each Python release; this module helps to find out programmatically

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Meador Inge mead...@gmail.com: -- nosy: +meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Jeremy Hylton jhyl...@gmail.com: -- nosy: +Jeremy.Hylton ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Gregory P. Smith g...@krypto.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan ncogh...@gmail.com added the comment: Bumping the target version to 3.4. This is still a good long term idea, but it's a substantial enough change that we really want to land it early in a development cycle so we have plenty of time to hammer out any issues. -- versions:

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Raymond Hettinger raymond.hettin...@gmail.com added the comment: Good call, Nick. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +ericsnow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan ncogh...@gmail.com added the comment: Marking the PEP 380 implementation as a dependency, as I expect it to be easier to update this patch to cope with those changes than it would be the other way around. -- dependencies: +PEP 380 reference implementation for 3.3

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Changes by John O'Connor tehj...@gmail.com: -- nosy: +jcon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11549 ___ ___ Python-bugs-list mailing

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder elto...@gmail.com added the comment: I found a problem in constant de-duplication, already performed by compiler, that needs to be fixed before this change can be merged. Compiler tries to eliminate duplicate constants by putting them into a dict. However, duplicate in this case

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Nick Coghlan ncogh...@gmail.com added the comment: As Raymond noted though, some of the block stack fiddling doesn't make sense until after the bytecode has already been generated. It's OK to have multiple optimisers at different layers, each taking care of the elements that are best suited

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Eugene Toder elto...@gmail.com added the comment: Nick, if there's an interest in reviewing the patch I can update the it. I doubt it needs a lot of changes, given that visitor is auto-generated. Raymond, the patch contains a rewrite of low-level optimizations to work before byte code

[issue11549] Build-out an AST optimizer, moving some functionality out of the peephole optimizer

Raymond Hettinger raymond.hettin...@gmail.com added the comment: Updated the title to reflect that the peephole optimizer will likely continue to exist but in a much simpler form. Some complex peephole optimization such as constant folding can be handled more easily and more robustly at the