[issue30455] Generate C code from token.py and not vice versa

Serhiy Storchaka added the comment: Alternate PR 10370 generates all files from a single file Grammar/Tokens using a single script Tools/scripts/generate_token.py. In addition, the script doesn't write files when the content is not changed. Thus it can be used with read-only sources.

[issue30455] Generate C code from token.py and not vice versa

Change by Serhiy Storchaka : -- pull_requests: +9671 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue30455] Generate C code from token.py and not vice versa

Change by Emily Morehouse : -- pull_requests: +8783 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue30455] Generate C code from token.py and not vice versa

Change by Serhiy Storchaka : -- pull_requests: -5479 ___ Python tracker ___

[issue30455] Generate C code from token.py and not vice versa

Change by Zachary Ware : -- keywords: +patch pull_requests: +5479 ___ Python tracker ___

[issue30455] Generate C code from token.py and not vice versa

Serhiy Storchaka added the comment: The regular expression tokenize.Funny also can be generated. Information is not enough for distinguish between Operator, Bracket and Special, but seems this isn't needed. Some token names can be generated from Grammar/Grammar. But needed an additional

[issue30455] Generate C code from token.py and not vice versa

Albert-Jan Nijburg added the comment: I think this covers all the changes from PR #1608. Looks a lot nicer too, building it every time from the make file. You may want to add to the docs that token.py is now the source of the tokens. -- ___ Python

[issue30455] Generate C code from token.py and not vice versa

Serhiy Storchaka added the comment: PR 1860 makes following files be generated from token.py: * Include/token.h * Parser/token.c. New file containing the array of token names _PyParser_TokenNames, and functions PyToken_OneChar(), PyToken_TwoChars(), PyToken_ThreeChars(), moved from

[issue30455] Generate C code from token.py and not vice versa

Changes by Serhiy Storchaka : -- pull_requests: +1943 ___ Python tracker ___ ___

[issue30455] Generate C code from token.py and not vice versa

Serhiy Storchaka added the comment: I already write a patch. -- assignee: -> serhiy.storchaka ___ Python tracker ___

[issue30455] Generate C code from token.py and not vice versa

Stéphane Wirtel added the comment: I can work on it -- nosy: +matrixise ___ Python tracker ___ ___

[issue30455] Generate C code from token.py and not vice versa

STINNER Victor added the comment: I like the idea. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue30455] Generate C code from token.py and not vice versa

New submission from Serhiy Storchaka: Currently Lib/token.py is generated from Include/token.h. This contradicts common practice when the C code is generated from the Python code (see for example opcode.py and sre_constants.py). In additional the table in Parser/tokenizer.c should be manually