[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Irit Katriel]

Irit Katriel  added the comment:

Created issue45544 to close all open issues and list them there.

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset fdbdf3f7359832820a11ece4c4b01581004d6fe7 by Gregory P. Smith in 
branch 'main':
bpo-40360: Make the 2to3 deprecation more obvious. (GH-29064)
https://github.com/python/cpython/commit/fdbdf3f7359832820a11ece4c4b01581004d6fe7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +27331
pull_request: https://github.com/python/cpython/pull/29064

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Łukasz Langa  added the comment:

The "pending" deprecation status of lib2to3 in 3.9 and 3.10 is no worse than a 
vanilla deprecation in terms of visibility. It will appear just the same when 
run with pytest or `-X dev`.

However, upgrading the deprecation between 3.10.0rc1 and 3.10.0rc2 really was 
too late. So we'll have it deprecated fully in 3.11 and removed in 3.13. One 
more year doesn't hurt us much but might be helpful for library maintainers to 
have more time to move off of the included lib2to3.

Invoking `2to3` also generates the warning so I guess this can be closed:

$ ./python.exe Tools/scripts/2to3
/private/tmp/cpy2/Tools/scripts/2to3:3: DeprecationWarning: lib2to3 package is 
deprecated and may not be able to parse Python 3.10+
  from lib2to3.main import main
At least one file or directory argument required.
Use --help to show usage.

Thanks for the patches, Gregory, Carl, and Victor! ✨  ✨  

Now we just have to remember to actually remove the damn thing in 3.13 

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

STINNER Victor  added the comment:

I close the issue: lib2to3 is now deprecated in Python 3.11. I propose to open 
a new issue in Python 3.13 or newer to remove it.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[miss-islington]

miss-islington  added the comment:


New changeset 559af7434668e2950c08389515a52eba697ef6af by Miss Islington (bot) 
in branch '3.10':
bpo-40360: [doc] Rephrase deprecation note about lib2to3 (GH-28122)
https://github.com/python/cpython/commit/559af7434668e2950c08389515a52eba697ef6af


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[miss-islington]

Change by miss-islington :


--
pull_requests: +26566
pull_request: https://github.com/python/cpython/pull/28127

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset f0b63d5b56a6324f5f86807d9548c7b38aa2a8f7 by Łukasz Langa in 
branch 'main':
bpo-40360: [doc] Rephrase deprecation note about lib2to3 (GH-28122)
https://github.com/python/cpython/commit/f0b63d5b56a6324f5f86807d9548c7b38aa2a8f7


--
message_count: 45.0 -> 46.0
pull_requests: +26566
pull_request: https://github.com/python/cpython/pull/28127

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Łukasz Langa  added the comment:

> We can add to the 3.10 docs that it is deprecated without any code change.

This was already the case:
https://docs.python.org/3/library/2to3.html#module-lib2to3

The wording was a bit clumsy so I rephrased in GH-28122.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Change by Łukasz Langa :


--
pull_requests: +26563
pull_request: https://github.com/python/cpython/pull/28122

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset d589a7e7eb56196c05337d37417479375878b127 by Victor Stinner in 
branch 'main':
bpo-40360: Deprecate the lib2to3 package (GH-28116)
https://github.com/python/cpython/commit/d589a7e7eb56196c05337d37417479375878b127


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

We can add to the 3.10 docs that it is deprecated without any code change. And 
in 3.11 we can add a warning.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

STINNER Victor  added the comment:

Guido: How come the deprecation didn't happen in 3.10? Were people just not 
interested?

Well, if nobody deprecates it, it's not deprecated. It is simple as it it :-)

IMO it's ok to only deprecate it in Python 3.11, unless Pablo *really* wants to 
deprecate lib2to3 before just Python 3.10 final.

I dislike adding new warnings between a beta1 release and the final release :-( 
In my experience, it *does* break projects which care of warnings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:

I think we just forgot to make the change in time.  3.11 is fine.  We're not 
_maintaining_ lib2to3 or describing it as fit for any modern purpose 
regardless.  It's just code that'll sit around in the back of the 3.10 stdlib 
but not be able to parse the new syntax in 3.10.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

How come the deprecation didn't happen in 3.10? Were people just not
interested?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

STINNER Victor  added the comment:

I retarget this issue to Python 3.11, since lib2to3 is *not* deprecated in 
Python 3.10.

--
versions: +Python 3.11 -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

STINNER Victor  added the comment:

I created PR 28116 to deprecate the lib2to3 package: replace 
PendingDeprecationWarning to DeprecationWarning.

In 2021, I don't think that we should keep the 2to3 tool in the stdlib. Python 
2 reached end of line 1 year ago.

For the lib2to3 *parser*, IMO it would be better to maintain it outside the 
stdlib, and collaborate with other existing forks, like the parser used by 
Black.

The change is only about *deprecating* lib2to3, not *remove* it. For the 
removal, we should check if major projects which used it moved to something 
else.

It's kind of a shame that Python stdlib (lib2to3) cannot parse valid Python 
3.10 code :-( IMO it's better to deprecate (and then remove) lib2to3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner
nosy_count: 11.0 -> 12.0
pull_requests: +26557
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28116

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:

status: lib2to3 PendingDeprecationWarning shipped in 3.9.  Since we don't have 
a specific release planned for the final deprecation, I'll leave this issue 
open while we figure that out.  Once we do, we should promote this to a regular 
DeprecationWarning in whichever release is next at that time.

--
versions:  -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee: gregory.p.smith -> 
stage: patch review -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset fe928b32daca184e16ccc0ebdc20314cfa776b98 by Karthikeyan 
Singaravelan in branch '3.9':
[3.9] bpo-40360: Handle PendingDeprecationWarning in test_lib2to3. (GH-21694) 
(GH-21697)
https://github.com/python/cpython/commit/fe928b32daca184e16ccc0ebdc20314cfa776b98


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests: +20841
pull_request: https://github.com/python/cpython/pull/21697

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:


New changeset cadda52d974937069eeebea1cca4229e2bd400df by Karthikeyan 
Singaravelan in branch 'master':
bpo-40360: Handle PendingDeprecationWarning in test_lib2to3. (GH-21694)
https://github.com/python/cpython/commit/cadda52d974937069eeebea1cca4229e2bd400df


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 10.0 -> 11.0
pull_requests: +20840
pull_request: https://github.com/python/cpython/pull/21696

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests: +20838
pull_request: https://github.com/python/cpython/pull/21694

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[David Halter]

David Halter  added the comment:

@gvanrossum

> Does parso have to be pure Python? If not, we could generate C code like we 
> do for CPython's parser. 

I would rather write the parser either in C or Rust. So no, parso does not need 
to be pure Python.

> Now, that doesn't work for incremental parsing, but I did an alternative 
> implementation that uses a stack machine, also in C, that's only 2x slower 
> than the PEG parser. Maybe that could be adapted to incremental parsing 
> (because it's a stack machine). Error recovery is still a research project 
> (at least for me -- I'm actually reading papers :-).

Makes sense! I was also thinking about GLL parsing. Obviously GLL does not 
cover all cases where PEG could potentially work, but I doubt that Python ever 
moves to a place where GLL would not be sufficient.

I'm also doing a bit of research on Rust parsers and trying to find a solution 
for my parsing needs in the future. (I'd rather have a Rust parser than a C 
one, because I like the language better and both should still work in Python).

Please let me know if you're making a lot of progress with PEG parsers and 
error recovery/incremental parsing. I'm definitely interested in copying an 
approach if it works :).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

Okay, so if you know what to do please do it. ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I was referring to PR https://github.com/python/cpython/pull/19663 
(commit-503de7149d03bdcc671dcbbb5b64f761bb192b4d) that was merged as part of 
this issue. It started emitting PendingDeprecationWarning but was not silenced 
in the test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

Which patch are you referring to? Is it already merged?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

After this patch test_lib2to3 generates a PendingDeprecationWarning. It can be 
silenced as it's intentional to avoid test failures while running tests with 
-Werror.

./python.exe -Wall -m test test_lib2to3
0:00:00 load avg: 2.31 Run tests sequentially
0:00:00 load avg: 2.31 [1/1] test_lib2to3
/Users/kasingar/stuff/python/cpython/Lib/test/test_lib2to3.py:1: 
PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to 
parse Python 3.10+
  from lib2to3.tests import load_tests

== Tests result: SUCCESS ==

1 test OK.

Total duration: 9.0 sec
Tests result: SUCCESS

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

I guess the design space is wide open.

Does parso have to be pure Python? If not, we could generate C code like we do 
for CPython's parser. Now, that doesn't work for incremental parsing, but I did 
an alternative implementation that uses a stack machine, also in C, that's only 
2x slower than the PEG parser. Maybe that could be adapted to incremental 
parsing (because it's a stack machine). Error recovery is still a research 
project (at least for me -- I'm actually reading papers :-).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[David Halter]

David Halter  added the comment:

Parso's incremental parser is a terrible idea. It also works and is pretty 
fast, but the design is pretty terrible (it took me a lot of fuzzing to make 
sure that it works decently well).

The basic problem is that it's reusing nodes in a mutable way. If I were to 
redo it, I would probably choose a similar approach to Roslyn's red/green 
trees. It's probably also possible to use these approaches in Python, but they 
might be quite a bit slower than what I'm using (because recreating nodes can 
be quite expensive).

I imagine that one of the biggest issues with parsing PEG in parso would be to 
do it with error recovery AND incremental parsing. That combination can be 
quite annoying, but it's definitely still possible.

I'm not really sure about the future of parso with PEG. I'm definitely going to 
have to find a way to parse 3.10+ (so Jedi is going to keep working), however I 
feel like that it's hard to achieve a fast parser in pure Python. Parso is like 
20% faster, but still more than ten times slower than the CPython parser...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[wyz23x2]

Change by wyz23x2 :


--
versions: +Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

Thanks for joining in! How do you do incremental parsing with LL1 currently? 
FWIW I found 
https://ohmlang.github.io/pubs/sle2017/incremental-packrat-parsing.pdf which 
may have some useful ideas.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[David Halter]

David Halter  added the comment:

I'm the maintainer of parso. Feel free to addd me to the Nosy List if we have 
these discussions in the future.

Parso is indeed a lib2to3 fork with error recovery, round tripping and 
incremental parsing as its features. Most pgen2 code has been rewritten since 
for various reasons, but it's essentially still LL(1). We're currently trying 
to think how to proceed with Non-LL(1). For very simple cases a few hacks could 
suffice, but for larger stuff we will probably need to implement some form of 
PEG parsing.

I'm mostly worried about incremental parsing breaking if we replace the PEG 
parser, not about writing the PEG parser. But I guess we'll find a way, because 
I don't want to abandon Jedi (which depends on parso).

--
nosy: +davidhalter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Peter Ludemann]

Peter Ludemann  added the comment:

Yes, I'm thinking of doing this as a wrapper, in such a way that it could be 
incorporated into Lib/ast.py eventually. (Also, any lib2to3-ish capabilities 
would probably not be suitable for inclusion in the stdlib, at least not 
initially ... but I have no plans to work on something to replace lib2to3's 
fixers.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

Can that be done as a 3rd party wrapper? Then you would be able to support 
older Python versions, and typed_ast (which can parse older Python grammars 
with a newer Python that's older than 3.8). Plus it would be much easier to get 
your code released -- no waiting for core devs to review it or waiting for the 
next CPython (bugfix or feature) release to get a bug fixed or small feature 
added.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Peter Ludemann]

Peter Ludemann  added the comment:

I've written up a proposal for adding "whitespace" handling to the ast module:
https://mail.python.org/archives/list/python-id...@python.org/thread/X2HJ6I6XLIGRZDB27HRHIVQC3RXNZAY4/

I don't think it's a "summer-of-code-sized project", mainly because I already 
have various bits of code that handle the fiddly byte/str offset conversions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

There's no python-dev discussion; if you want more feedback I recommend 
starting on python-ideas first (on either forum you may expect pushback because 
this is not about a proposed change to Python or its workflow).

The Lib/ast.py module will continue to be the official API for the standard 
AST. It is a simple wrapper around the builtin parser (at least in CPython -- I 
don't actually know to what extent other Python implementations support it, but 
they certainly *could*). And in 3.9 and later the AST is already being produced 
using the *new* parser.

We want to deprecate lib2to3 because nobody is interested in maintaining it., 
Having it in the stdlib, with its strict backwards compatibility requirements, 
makes it difficult to do a good job at updating it. This is why it's been 
forked repeatedly -- once forked, the owner of the fork can make changes 
easily, preserving the API perfectly (if so desired) and maintaining 
compatibility with older Python versions.

My own thoughts are that libraries like LibCST and parso have two sides: an API 
for the AST, and a way to parse source code into an AST. Usually the parsing 
API is incredibly simple -- e.g. a function to parse a file and another 
function to parse a string. And there's no reason for the AST API to change 
just because the parsing algorithm has changed.

Finally, we already have a (rough) Python implementation of the PEG parser too 
-- in fact it's included in Tools/peg_generator (and used to regenerate the 
metaparser). This reads the same grammar format (i.e. Grammar/python.gram) and 
generates Python code instead of C code to do the parsing. It's easy to 
retarget the tokenizer of the generated Python code.

So a decent way forward might be to pick one of the 3rd party libraries 
(perhaps parso, which is itself a fork of lib2to3 and what LibCST builds on) 
and update its parser to use a PEG parser generated using the PEG generator 
from Tools/peg_generator (which people are welcome to fork).

This might be a summer-of-code-sized project.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Peter Ludemann]

Peter Ludemann  added the comment:

Looking at the suggested successor tools (redbaron, libCST, parso, awpa) ... 
all of them appear to use some variant of pgen2. But at some point Python will 
be using a PEG approach (PEP 617), and therefor the pgen2 approach apparently 
won't work.

For a number of projects, it's important to have a parse tree that contains all 
the "whitespace" information (indent, dedent, comment, newline, etc.) As far as 
I can tell, the new PEG parser won't provide that, and it seems that none of 
the successor tools will be able to handle future versions of Python syntax.

So, three questions:
1. Am I right that all proposed replacements (redbaron, libCST, parso, awpa) 
use some variation of the LL(1) and therefore will have trouble in the future?
2. Are there any plans (either part of the core development or as a project) 
for one of these replacements that is PEG-based? (Or a new project?)
3. Is Lib/ast.py going to continue being supported? (I infer that it will, with 
the change from LL(1) to PEG being mostly transparent - 
https://mail.python.org/archives/list/python-...@python.org/thread/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/#4D3B2NM2JMV2UKIT6EV5Q2A6XK2HXDEH
 )

If Lib/ast.py continues to be supported, I think I can see a way of providing 
functionality similar to lib2to3 (in terms of an AST-ish thing with 
"whitespace" from the source, sufficient for tools such as yapf, black, 
pykythe, pytype, mypy, etc.) as a kind of wrapper to ast.py. 
I suppose I should discuss this idea on python-dev? Is there an ongoing 
discussion? (I couldn't find any but might have been using the wrong search 
terms)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[STINNER Victor]

STINNER Victor  added the comment:

FYI the autopep8 project uses lib2to3.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 18f1c60a1625d341a905c7e07367c32c08f222df by Miro Hrončok in 
branch 'master':
bpo-40360: Add a What's New entry for lib2to3 pending deprecation (GH-19898)
https://github.com/python/cpython/commit/18f1c60a1625d341a905c7e07367c32c08f222df


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Miro Hrončok]

Change by Miro Hrončok :


--
pull_requests: +19209
pull_request: https://github.com/python/cpython/pull/19898

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Miro Hrončok]

Miro Hrončok  added the comment:

Thanks for the explanation.

I plan to send a PR to add this to the What's new in 3.9 page early next week. 
Anyone, feel free to beat me to it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

IIRC PendingDeprecationError does not mean that the decision hasn't been made 
yet. It just means it's less urgent for folks to worry about. I believe we tend 
to change PendingDeprecationError to DeprecationError in the last release 
before something is removed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Miro Hrončok]

Miro Hrončok  added the comment:

> Getting rid of PendingDeprecationWarning seems like an orthogonal decision; 
> if it happens, this can trivially be upgraded to DeprecationWarning as part 
> of a removal sweep.

My thought was that the decision was already made to do so. Hence adding new 
PendingDeprecationWarnings goes against that decision.

But maybe I misunderstand and that decision was not made.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

A "What's New" entry would go into Doc/whatsnew/3.9.rst and is much more 
visible to users looking for exciting bits in the new release (the NEWS file is 
very large, see e.g. 
https://docs.python.org/3/whatsnew/changelog.html#changelog.

The What's New doc typically has a section collecting all the deprecations, 
e.g. https://docs.python.org/3/whatsnew/3.8.html#deprecated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Carl Meyer  added the comment:

> Coul you please add a what's new entry for this change?

The committed change already included an entry in NEWS. Is a "What's New" entry 
something different?

> I don't understand why there is a PendingDeprecationWarning and not a 
> DeprecationWarning.

Purely because I was following gps' recommendation in the first comment on this 
issue. Getting rid of PendingDeprecationWarning seems like an orthogonal 
decision; if it happens, this can trivially be upgraded to DeprecationWarning 
as part of a removal sweep.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Miro Hrončok]

Miro Hrončok  added the comment:

I don't understand why there is a PendingDeprecationWarning and not a 
DeprecationWarning.


See 
https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038/4 
and issue36404

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Miro Hrončok]

Miro Hrončok  added the comment:

Coul you please add a what's new entry for this change?

--
nosy: +hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Carl Meyer  added the comment:

Right, although I think it still makes sense to link both LibCST and parso 
since they provide different levels of abstraction that would be suitable for 
different types of tools (e.g. I would rather write an auto-formatter on top of 
parso, because LibCST's careful parsing and assignment of whitespace would 
mostly just get in the way, but I'd rather write any kind of refactoring 
tooling on top of LibCST.)

Another tool that escaped my mind when writing the PR that should probably be 
linked also is Baron/RedBaron (https://github.com/PyCQA/redbaron); 457 stars 
makes it slightly more popular than LibCST (but it's also been around a lot 
longer.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

It's typically not up to the core devs to pick a winning third party library; 
we tend to recommend libraries that are already essentially category winners, 
like requests. In a sense pointing to LibCST *and* parso is redundant because 
LibCST builds on parso. Comparing stars on GitHub:
- LibCST: 423
- parso: 296
- awpa: 10

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Peter Ludemann]

Peter Ludemann  added the comment:

The documentation change gives two possible successors:

https://libcst.readthedocs.io/ (https://github.com/Instagram/LibCST)
https://parso.readthedocs.io/

And I've also seen this mentioned: https://github.com/pyga/awpa

Is it possible to settle on one of these as the successor to the lib2to3 
parser? It would be nice to avoid a 2nd deprecation in the future ...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Peter Ludemann]

Change by Peter Ludemann :


--
nosy: +Peter Ludemann

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests:  -19032

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +corona10
nosy_count: 5.0 -> 6.0
pull_requests: +19032
pull_request: https://github.com/python/cpython/pull/18245

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:

I think what we're doing with the documentation update is fine.  We can add a 
warning on stderr to the tool in 3.11.  But I don't expect people will be using 
the tool _from_ the latest CPython 3.x by then.

2to3 is already included with Python 2.7 and the only real use for it is for 
people who still have code they maintain on 2.7 so they've got a copy already.  
There is no value in running a 2to3 shipped with Python 3 vs the latest 2.7.  
Meaningful updates to it were already back ported to 2.7 over time as it was 
intentionally exempt from feature freeze.

We should have sorted out a PyPI home for lib2to3 by 3.11 time and can also 
create a PyPI package for the 2to3 tool itself at that point.

I _think_ there is support for running 2to3 on sources at package install time 
from setup.py?  But I don't expect anything actually maintained and widely used 
to require that by the time this deprecation lands.  If it does, that becomes a 
plumbing issue within package tools to know that requiring 2to3 at either build 
or install time adds an implicit tool dependency on the new pypi package to get 
it.

Maybe I'm just in a good mood about all of this, but none of this seems 
worrisome.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Carl Meyer  added the comment:

@gregory.p.smith

What do you think about the question I raised above about how to make this 
deprecation visible to users of the 2to3 CLI tool, assuming the plan is to 
remove both?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 503de7149d03bdcc671dcbbb5b64f761bb192b4d by Carl Meyer in branch 
'master':
bpo-40360: Deprecate lib2to3 module in light of PEP 617 (GH-19663)
https://github.com/python/cpython/commit/503de7149d03bdcc671dcbbb5b64f761bb192b4d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Okay,the pending deprecation is in.  Keeping open as a reminder to turn that 
into a real DeprecationWarning in 3.10 after the 3.9 branch is cut.

We'll then want to track reminding us to remove it in 3.12.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Carl Meyer  added the comment:

I opened a PR. It deprecates the lib2to3 library to discourage future use of it 
for Python3, but not the 2to3 tool. This of course means that the lib2to3 
module will in practice stick around in the stdlib as long as 2to3 is still 
bundled with Python.

It seems like the idea in this issue is to deprecate and remove both. I'm not 
sure what we typically do to deprecate a command-line utility bundled with 
Python. Given warnings are silent by default, the deprecation warning for 
lib2to3 won't be visible to users of 2to3. Should I add something to its 
`--help` output? Or something more aggressive; an unconditionally-printed 
warning?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Change by Carl Meyer :


--
pull_requests: +18987
pull_request: https://github.com/python/cpython/pull/19663

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Eric Snow]

Change by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Carl Meyer]

Carl Meyer  added the comment:

I volunteered in the python-dev thread to write a patch to the docs clarifying 
future status of lib2to3; happy to include the PendingDeprecationWarning as 
well.

Re linking to alternatives, we want to make sure we link to alternatives that 
are committed to updating to support newer Python versions' syntax. This 
definitely includes LibCST; I can inquire with the parso maintainer about 
whether it also includes parso. In future it could also include a 
third-party-maintained copy of lib2to3, if someone picks that up.

--
nosy: +carljm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Guido van Rossum]

Guido van Rossum  added the comment:

I am in favor of this. We could promote LibCST, which is based on Parso, which 
uses a forked version of pgen2 (the parser in lib2to3). I believe one of these 
could switch to a fork of pegen as its parser, so it will be able to handle new 
PEG based syntax in 3.10+.

Removal by 3.12 might be feasible.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Change by Gregory P. Smith :


--
keywords: +patch
pull_requests: +18969
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/19645

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

Change by Gregory P. Smith :


--
components: +2to3 (2.x to 3.x conversion tool)
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40360] Deprecate lib2to3 (and 2to3) for future removal

[Gregory P. Smith]

New submission from Gregory P. Smith :

Based on the PEP 617 acceptance thread on python-dev, lib2to3 is eventually 
going to run into trouble parsing modern syntax a few releases from now.

It would be better off maintained outside of the standard library.  It gets 
used by a lot of things and is generally useful, but would make a lot more 
sense as a PyPI project than as something only quasi-maintained within the 
stdlib (it only gained the ability to parse a couple modern syntax features in 
via bugfix contributions to the stdlib the past month or two...  meaning a lot 
of versions of it out there cannot)

Black has already forked it.

goal:  PendingDeprecationWarning and documentation as such in 3.9.  Move to 
DeprecationWarning in 3.10 or 3.11 and remove it by ~3.12.  Subject to our 
existing deprecation process guidelines.

--
assignee: gregory.p.smith
messages: 366973
nosy: gregory.p.smith
priority: normal
severity: normal
status: open
title: Deprecate lib2to3 (and 2to3) for future removal
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com