There is no *public* one but there is a private one accesible from Python I added for testing purposes.
On Mon, 30 May 2022, 15:17 Victor Stinner, <vstin...@python.org> wrote: > On Mon, May 30, 2022 at 1:40 AM Eric V. Smith <e...@trueblade.com> wrote: > > python -m tokenize < file-to-parse.py > > > > See the comment at the top of tokenize.py. IIRC, it re-implements the > > tokenizer, it does not call the one used for python code. > > Ah right, I would be surprised that there would be a public Python API > to get the tokenizer output, since there is no public C API for that > :-) > > I just removed <token.h> header file since it was never usable outside > Python C internals: there is no public C API to just run the tokenizer > and gets its output. > > Victor > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5AWZ4NJCS5TVWMR2KZOWLJT7AYDCXVIY/ Code of Conduct: http://python.org/psf/codeofconduct/
