[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Alternate PR 10370 generates all files from a single file Grammar/Tokens using 
a single script Tools/scripts/generate_token.py.

In addition, the script doesn't write files when the content is not changed. 
Thus it can be used with read-only sources.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +9671

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Emily Morehouse]

Change by Emily Morehouse :


--
pull_requests: +8783

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests:  -5479

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Zachary Ware]

Change by Zachary Ware :


--
keywords: +patch
pull_requests: +5479

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The regular expression tokenize.Funny also can be generated. Information is not 
enough for distinguish between Operator, Bracket and Special, but seems this 
isn't needed.

Some token names can be generated from Grammar/Grammar. But needed an 
additional mapping for relations between token strings and names ('+' <-> PLUS, 
etc).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Albert-Jan Nijburg]

Albert-Jan Nijburg added the comment:

I think this covers all the changes from PR #1608. Looks a lot nicer too, 
building it every time from the make file. 

You may want to add to the docs that token.py is now the source of the tokens.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

PR 1860 makes following files be generated from token.py:

* Include/token.h
* Parser/token.c. New file containing the array of token names 
_PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), 
PyToken_ThreeChars(), moved from Parser/tokenizer.c.
* Doc/library/token-list.inc. New file containing the list of token.py 
constants, it is included in Doc/library/token.rst.

New Makefile target regen-token regenerates these files.

The dict EXACT_TOKEN_TYPES that maps operator strings to token names now is 
automatically generated and moved from tokenize.py to token.py. Tokens COMMENT, 
NL and ENCODING used only in tokenize.py now are added in token.py as in 
issue25324.

--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +1943

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I already write a patch.

--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

I can work on it

--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[STINNER Victor]

STINNER Victor added the comment:

I like the idea.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30455] Generate C code from token.py and not vice versa

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Currently Lib/token.py is generated from Include/token.h. This contradicts 
common practice when the C code is generated from the Python code (see for 
example opcode.py and sre_constants.py). In additional the table in 
Parser/tokenizer.c should be manually supported matching Include/token.h.

Generating Include/token.h and Parser/tokenizer.c from Lib/token.py would be 
simpler and more reliable.

--
components: Interpreter Core, Library (Lib)
messages: 294350
nosy: Albert-Jan Nijburg, benjamin.peterson, haypo, meador.inge, 
r.david.murray, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Generate C code from token.py and not vice versa
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com