[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Roundup Robot]

Roundup Robot added the comment:

New changeset 16f60cd918e0 by Victor Stinner in branch 'default':
PEP 511
https://hg.python.org/peps/rev/16f60cd918e0

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Roundup Robot]

Roundup Robot added the comment:

New changeset c5df914e73ad by Victor Stinner in branch 'default':
Fix a refleak in validate_constant()
https://hg.python.org/cpython/rev/c5df914e73ad

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Roundup Robot]

Roundup Robot added the comment:

New changeset 0d0d0a05 by Victor Stinner in branch 'default':
Issue #26146: remove useless code
https://hg.python.org/cpython/rev/0d0d0a05

New changeset 58086f0a953a by Victor Stinner in branch 'default':
Issue #26146: enhance ast.Constant error message
https://hg.python.org/cpython/rev/58086f0a953a

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

Patch version 3: fix a reference leak in validate_constant().

--
Added file: http://bugs.python.org/file41708/constant-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Roundup Robot]

Roundup Robot added the comment:

New changeset 27e5437f442c by Victor Stinner in branch 'default':
Add ast.Constant
https://hg.python.org/cpython/rev/27e5437f442c

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

I pushed an enhanced version of constant-3.patch:

* fix the compiler: don't emit LOAD_CONST for ast.Constant if the value is a 
string (str) or a number (int, float, complex), as done previously for ast.Str 
and ast.Num

* add unit tests on: the compiler (ensure that LOAD_CONST is emitted for 
constants), ast.get_docstring() and ast.literal_eval()

--
components: +Interpreter Core
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5de1bd759f3b by Victor Stinner in branch 'default':
Issue #26146: marshal.loads() now uses the empty frozenset singleton
https://hg.python.org/cpython/rev/5de1bd759f3b

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

> New changeset 5de1bd759f3b by Victor Stinner in branch 'default':
> Issue #26146: marshal.loads() now uses the empty frozenset singleton
> https://hg.python.org/cpython/rev/5de1bd759f3b

This change is not directly related to this issue.

It's required by test_singleton_empty_frozenset() of test_set when an AST 
transformer replaces frozenset(), frozenset([]), etc. calls with an empty 
frozenset constant.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

@Serhiy: Would you mind reviewing constant-2.patch? I reviewed my own patch and 
added some comments, I found a refleak ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

Patch version 2:

- rework the whole patch
- add unit tests
- fix AST validation: the code was completly wrong in patch 1, I don't 
understand how it worked :-p Validate also correctly nested tuple and nested 
frozenset.
- add a comment to explain why obj2ast_constant() doesn't have to Py_INCREF() 
singletons
- handle also Ellipsis
- revert changes on set_context(). It seems like set_context() is only called 
with code emited directly by the compiler (not by compile(custom_ast_tree, 
...)). If someone finds a way to call set_context() with an ast.Constant, we 
can support this case later.

--
Added file: http://bugs.python.org/file41689/constant-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

> We also make no backwards-compatibility guarantees about the AST, so if it 
> simplifies things to switch entirely to Constant from Num, etc. then I said 
> do it.

Replacing Num/Str/... with Constant causes bug in markerlib (used by pip, it 
breaks indirectly the venv module of the stdlib) and Chameleon benchmark. I 
only found these projects by mistake. I'm quite sure that much more code rely 
on the AST "API" even if it's unstable.

I prefer to not break the API for free :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

Changes by STINNER Victor :


--
dependencies: +code_richcompare() don't use constant type when comparing code 
constants

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

To support emiting constants from ast.Constant, we will also need the fix for 
the issue #25843. Currently, the compile merges constants (0, 0) and (0.0, 0.0) 
because they are equal, but item types are different.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[Brett Cannon]

Brett Cannon added the comment:

Would it make sense to tag the type of the constant in the node somehow?

We also make no backwards-compatibility guarantees about the AST, so if it 
simplifies things to switch entirely to Constant from Num, etc. then I said do 
it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

New submission from STINNER Victor:

Currently, the Python parser emits ast.Tuple AST nodes which are compiled to 
multiple LOAD_xxx instructions followed a final BUILD_TUPLE instruction. The 
Python peephole optimizer detect LOAD_CONST x n + BUILD_TUPLE instructions to 
replace it directly with a tuple constant.

IHMO it's better to implement this optimization early at the AST level in an 
AST optimizer. The PEP 511 proposes to add a new ast.Constant AST node for that.

With this new AST node, the AST optimizer can emit tuple constants, but also 
any kind of constant like frozenset. For example, it's also possible to 
optimize "x in {1, 2}" to "x in frozenset({1, 2})" where frozenset({1, 2}) is a 
constant (don't call frozenset type at runtime). (Again, this optimization is 
already implemented in the peephole optimizer, it's just an example.)

Attached patch adds the new ast.Constant type but update also code consuming 
AST to handle this new kind of node:

* add the node: Parser/Python.asdl, Parser/asdl.py, Parser/asdl_c.py
* generated changes: Include/asdl.h, Include/Python-ast.h, Python/Python-ast.c
* accept the node kind: Python/symtable.c, Python/ast.c
* accept Constant instead of Num or Str: Lib/ast.py, Python/future.c, 
Tools/parser/unparse.py
* emit constants to bytecode: Python/compile.c

I didn't change the compiler to emit directly ast.Constant nodes to reduce the 
impact on the backward compatibility. This change can be done. An AST optimizer 
is responsible to convert NameConstant (False, True, None), Num (int, float, 
complex), Str, Bytes, Tuple to Constant. Example with fatoptimizer:
https://github.com/haypo/fatoptimizer/blob/2d794f511fe23ccde320725c6d12ce5ce8ffbdfe/fatoptimizer/convert_const.py

ast.Constant also simplifies AST optimizers: no need to check each time if a 
node is constant or not.

Adding a new kind of node was already proposed in the old issue #11549: the 
patch adds ast.Lit (it was proposed to rename it to ast.Literal).

--
files: constant.patch
keywords: patch
messages: 258528
nosy: benjamin.peterson, brett.cannon, haypo, serhiy.storchaka
priority: normal
severity: normal
status: open
title: PEP 511: Add ast.Constant to allow AST optimizer to emit constants
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file41647/constant.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26146] PEP 511: Add ast.Constant to allow AST optimizer to emit constants

[STINNER Victor]

STINNER Victor added the comment:

Brett Cannon: "Would it make sense to tag the type of the constant in the node 
somehow?"

It's easy to get the type of a constant: type(node.value). I like using 
isinstance().

Brett Cannon: "We also make no backwards-compatibility guarantees about the 
AST, so if it simplifies things to switch entirely to Constant from Num, etc. 
then I said do it."

Oh, I forgot an important point from the PEP: "[ast.Constant] does not contain 
line number and column offset informations on tuple or frozenset items." I 
don't know if it's an issue or not.

I prefer to move step by step. As I wrote, we can decide to use directly 
ast.Constant later, even before the Python 3.6 release.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com