Solved!
The astute reader will notice that while I built the latest version of
python3.10 2022-05-24 11:38, the one I was testing was built May 15 2022,
12:44:05.
This was left over from an earlier iteration, and not cleaned up with a
subsequent make clean that was run after another ./configur
Sent from Mail for Windows
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archi
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started messing around with the tokenizer and the grammar. I was
wondering, is there a way to just print out the results of the tokenizer
(as in just the stream of tokens it generates) in a human readable
format? It woul
Le 30/05/2022 à 00:59, Jack a écrit :
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started messing around with the tokenizer and the grammar. I was
wondering, is there a way to just print out the results of the
tokenizer (as in just the stream of tokens it genera
Thanks! I didn't even know about that module. Does this take into
account your local changes to the tokenizer, though? I've added a new
token type to Grammar/Tokens, and some code to tokenizer.c to return
that token type in appropriate circumstances. I've stepped through the
tokenizer in the de
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the
tokenizer, it does not call the one used for python code.
Eric
On 5/29/2022 6:59 PM, Jack wrote:
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started mess
Well, I just stuck a print statement in _PyTokenizer_Get() and it's done
the job for me, right now.
Thanks,
Jack
On 30/05/2022 00:36, Eric V. Smith wrote:
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the
tokenizer, it does not call
