Just be aware that the C tokenizer interface is NOT a public interface and is there only so we can test the C tokenizer itself. This can and will break at any point without previous warning in any way.
Pablo Galindo Salgado On 3 Nov 2022, at 18:10, David J W <ward.dav...@gmail.com> wrote:
Following up, Pablo spotted my problem with the mixup of NL & NEWLINE tokens. I was using tokenize.py in cPython's stdlib with a simple python script to build ridiculously strict unit tests.
My solution to that problem was originally to figure out how to access cPython's internal c tokenizer but someone else did that in 3.11. The parser is passing basic tests but I need to redo all of the tests for my tokenizer as they are flawed and also do some major housekeeping to clean up all the warnings and TODO's sprinkled throughout my code base.
To hopefully avoid future problems, is Lib/symtable.py trustworthy as a way of building unit tests when I start implementing my own symbols graph/table?
Thanks, David
_______________________________________________Python-Dev mailing list -- python-dev@python.orgTo unsubscribe send an email to python-dev-le...@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TBWLY6AEAV6BLJDO2UHCHG5F7YLLVTQT/Code of Conduct: http://python.org/psf/codeofconduct/
|
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/KGSJTD3RRC3YBWYNVRLQ5YS4G6CQT22J/
Code of Conduct: http://python.org/psf/codeofconduct/