Well, I just stuck a print statement in _PyTokenizer_Get() and it's done
the job for me, right now.
Thanks,
Jack
On 30/05/2022 00:36, Eric V. Smith wrote:
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the
tokenizer, it does not call
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the
tokenizer, it does not call the one used for python code.
Eric
On 5/29/2022 6:59 PM, Jack wrote:
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started mess
Thanks! I didn't even know about that module. Does this take into
account your local changes to the tokenizer, though? I've added a new
token type to Grammar/Tokens, and some code to tokenizer.c to return
that token type in appropriate circumstances. I've stepped through the
tokenizer in the de
Le 30/05/2022 à 00:59, Jack a écrit :
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started messing around with the tokenizer and the grammar. I was
wondering, is there a way to just print out the results of the
tokenizer (as in just the stream of tokens it genera
Hi, I'm just getting into the CPython codebase just for fun, and I've
just started messing around with the tokenizer and the grammar. I was
wondering, is there a way to just print out the results of the tokenizer
(as in just the stream of tokens it generates) in a human readable
format? It woul
Sent from Mail for Windows
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archi
Solved!
The astute reader will notice that while I built the latest version of
python3.10 2022-05-24 11:38, the one I was testing was built May 15 2022,
12:44:05.
This was left over from an earlier iteration, and not cleaned up with a
subsequent make clean that was run after another ./configur
