[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-37253: "PyCompilerFlags got a new cf_feature_version field".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

Thanks for taking care of this!
-- 
--Guido (mobile)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b2fd32b2f041402bb9fc8607f01566c055aafed9 by Victor Stinner in 
branch '3.8':
bpo-35766: compile(): rename feature_version parameter (GH-13994) (GH-14001)
https://github.com/python/cpython/commit/b2fd32b2f041402bb9fc8607f01566c055aafed9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13864
pull_request: https://github.com/python/cpython/pull/14001

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 3ba21070c6ecab83c23cea41a92b42fa651c7ea2 by Victor Stinner (Miss 
Islington (bot)) in branch '3.8':
bpo-35766: Change format for feature_version to (major, minor) (GH-13992) 
(GH-13993)
https://github.com/python/cpython/commit/3ba21070c6ecab83c23cea41a92b42fa651c7ea2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[miss-islington]

Change by miss-islington :


--
pull_requests: +13858
pull_request: https://github.com/python/cpython/pull/13995

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:


New changeset efdf6ca90f7702824e7aeee1ceca949e7c20288a by Victor Stinner in 
branch 'master':
bpo-35766: compile(): rename feature_version parameter (GH-13994)
https://github.com/python/cpython/commit/efdf6ca90f7702824e7aeee1ceca949e7c20288a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13857
pull_request: https://github.com/python/cpython/pull/13994

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[miss-islington]

miss-islington  added the comment:


New changeset 10b55c1643b512b3a2cae8ab89c53683a13ca43e by Miss Islington (bot) 
(Guido van Rossum) in branch 'master':
bpo-35766: Change format for feature_version to (major, minor) (GH-13992)
https://github.com/python/cpython/commit/10b55c1643b512b3a2cae8ab89c53683a13ca43e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[miss-islington]

Change by miss-islington :


--
pull_requests: +13856
pull_request: https://github.com/python/cpython/pull/13993

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +13855
pull_request: https://github.com/python/cpython/pull/13992

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:

> But this is the format currently used by typed_ast. I think it would just 
> cause a cascade of annoying failures for tools that switch between typed_ast 
> and the 3.8 version of ast. Such as mypy.

typed_ast is a 3rd party project. But here we are talking about Python stdlib. 
If a project moves from typed_ast to stdlib ast, IMHO it should be quite easy 
to replace 4 with (3, 4), especially if 4 raises an error, no?

IMHO it's a good opportunity to fix it.

I would prefer to not have to batch many backward incompatible changes into 
Python 4...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

But this is the format currently used by typed_ast. I think it would just cause 
a cascade of annoying failures for tools that switch between typed_ast and the 
3.8 version of ast. Such as mypy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:

> I'm sure we can come up with a sufficiently backwards-compatible API.  (I'd 
> prefer not to go the hex route, because printing the value would show as 52, 
> which is pretty meaningless.)

I propose to replace 4 with (3, 4) to be more future-proof. This version format 
is already used by sys.version_info for example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[miss-islington]

miss-islington  added the comment:


New changeset 785688832de699270530a9418e99c5c4caedbffb by Miss Islington (bot) 
in branch '3.8':
bpo-35766: What's new in the ast and typing modules (GH-13984)
https://github.com/python/cpython/commit/785688832de699270530a9418e99c5c4caedbffb


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 9b33ce48a7846dbdad32d4f8936b08e6b78a2faf by Guido van Rossum in 
branch 'master':
bpo-35766: What's new in the ast and typing modules (#13984)
https://github.com/python/cpython/commit/9b33ce48a7846dbdad32d4f8936b08e6b78a2faf


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[miss-islington]

Change by miss-islington :


--
pull_requests: +13850
pull_request: https://github.com/python/cpython/pull/13987

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

> What will happen with Python 4? Why not using "full" Python version like (3, 
> 4) or 0x304?

I don't think Python 4 is just around the corner, so I prefer to kick that 
problem down the road. I'm sure we can come up with a sufficiently 
backwards-compatible API.  (I'd prefer not to go the hex route, because 
printing the value would show as 52, which is pretty meaningless.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:

> ``feature_version=N`` allows specifying the minor version of an earlier 
> Python 3 version.  (For example, ``feature_version=4`` will treat ``async`` 
> and ``await`` as non-reserved words.)

What will happen with Python 4? Why not using "full" Python version like (3, 4) 
or 0x304?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +13848
pull_request: https://github.com/python/cpython/pull/13984

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

I'll create a PR that updates the What's New docs with news about all of these.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:

Typo: "mypy todo list" should be "my todo list" :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:

> Would it be possible to document the change somewhere?

Most of these things are in the docs for ast module.

But indeed none of static typing related features have been added to What's 
New. I have an item on mypy todo list to add four typing PEPs plus new AST 
features (also adding `end_lineno` and `end_col_offset`) also there, but didn't 
have time last weekend.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[STINNER Victor]

STINNER Victor  added the comment:

I don't see anything in What's New in Python 3.8?:
https://docs.python.org/dev/whatsnew/3.8.html

Would it be possible to document the change somewhere?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

See https://bugs.python.org/issue35975

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

I have some follow-up wishes but I'll create a new issue.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 4b250fc1da9c893803cf724a4974450b5e10bd8a by Guido van Rossum in 
branch 'master':
bpo-35766 follow-up: Add an error check to new_type_comment() (#11766)
https://github.com/python/cpython/commit/4b250fc1da9c893803cf724a4974450b5e10bd8a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11723, 11724, 11725

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11723

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11723, 11724

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 3a32e3bf880f0842ac73d18285f5a1caa902a2ca by Guido van Rossum in 
branch 'master':
bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod 
(#11714)
https://github.com/python/cpython/commit/3a32e3bf880f0842ac73d18285f5a1caa902a2ca


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11573, 11574, 11575

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11573, 11574

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
pull_requests: +11573

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c by Łukasz Langa (Guido 
van Rossum) in branch 'master':
bpo-35766: Merge typed_ast back into CPython (GH-11645)
https://github.com/python/cpython/commit/dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

Also the tests now all pass; for now I am happy with the solution I found for 
the indentation error (see 
https://github.com/python/cpython/pull/11645#issuecomment-456627216).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

The PR is ready for reviews now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Ethan Smith]

Change by Ethan Smith :


--
nosy: +Ethan Smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
keywords: +patch, patch, patch
pull_requests: +11427, 11428, 11429
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
keywords: +patch, patch
pull_requests: +11427, 11428
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Change by Guido van Rossum :


--
keywords: +patch
pull_requests: +11427
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Gregory P. Smith]

Change by Gregory P. Smith :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Anthony Sottile]

Anthony Sottile  added the comment:

> You’d be surprised how tenacious old versions are. 

heh that's true, at my last job we finally got rid of python2.6 in 2016 :'(

anyway -- I don't mean to discourage this, definitely seems valuable to the 
maintenance of mypy!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

Guido van Rossum  added the comment:

You’d be surprised how tenacious old versions are. 

Anywa, I am working on this on my copious spare time — you can follow my 
progress at 
https://github.com/gvanrossum/cpython/tree/ast-type-comments?files=1 .

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Anthony Sottile]

Anthony Sottile  added the comment:

Seems also related to https://bugs.python.org/issue24119

with python2 / python3.5 (hopefully) rapidly falling off in usage I would 
assume the specialized treatment of `# type: ...` comments would become less 
and less necessary

Though I guess there's still `# type: ignore`, though if I recall correctly 
ignores are bubbled up to the module level and are applied on a line-by-line 
basis and wouldn't necessarily need specialized AST treatment (though a second 
parse over the token stream is a bit unfortunate)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Anthony Sottile]

Change by Anthony Sottile :


--
nosy: +Anthony Sottile

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Ivan Levkivskyi]

Change by Ivan Levkivskyi :


--
nosy: +levkivskyi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

New submission from Guido van Rossum :

(This started at 
https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's 
somewhat related to https://bugs.python.org/issue7.) 

I now have a thorough understanding of what typed_ast does, and I think it 
would be straightforward to port it upstream. We’d need to define two new 
tokens to represent `# type: ignore` and `# type: `, and tokenizer 
code to recognize these. Then we need a new flag to be passed to the tokenizer 
(via the parser) that enables this behavior. We make a small number of changes 
to `Grammar` (inserting optional `TYPE_COMMENT` tokens and to `Python.asdl` 
(adding fields to a few node types to hold the optional type comment), and a 
fair number of changes to `ast.c` to extract the type comments. We have similar 
patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 
3.8.

By default, `ast.parse()` should not return type comments, since this would 
reject some perfectly good Python code (with sonething looking like a type 
comment in a place where the grammar doesn’t allow it). But passing an new flag 
will cause the tokenizer to process type comments and the returned tree will 
contain them.

I could produce a PR with this in a few days (having just gone over most of the 
process for porting typed_ast from 3.6 to 3.7).

There’s one more feature I’d like to lobby for – a feature_version flag that 
modifies the grammar slightly so it resembles an older version of Python (going 
back to 3.4). This is used in mypy to decouple the Python version you’re 
running from the Python version for which you’re checking compatibility (useful 
when checking code that will be deployed on a system with a different Python 
version installed). I imagine this would be useful to other linters as well, 
and the implementation is mostly manipulating whether `async` and `await` are 
keywords. But if there’s pushback to this part I can live without it – the rest 
of the work is still useful.

In the next few days I will produce a PR so people can see for themselves.

In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, 
Łukasz offered to merge my PR.

--
components: Library (Lib)
messages: 333919
nosy: gvanrossum
priority: normal
severity: normal
stage: needs patch
status: open
title: Merge typed_ast back into CPython
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com