Is on the main branch but as I mentioned is **exclusively** for internal consumption:
https://github.com/python/cpython/blob/8136606769661c103c46d142e52ecbbbb88803f6/Lib/tokenize.py#L685 On Mon, 30 May 2022 at 17:37, Jack <jack.jjmill...@gmail.com> wrote: > Hi Pablo, could you clarify please? Is that on the main branch, or would > you be willing to share the code? > On 30/05/2022 16:23, Pablo Galindo Salgado wrote: > > There is no *public* one but there is a private one accesible from Python > I added for testing purposes. > > On Mon, 30 May 2022, 15:17 Victor Stinner, <vstin...@python.org> wrote: > >> On Mon, May 30, 2022 at 1:40 AM Eric V. Smith <e...@trueblade.com> wrote: >> > python -m tokenize < file-to-parse.py >> > >> > See the comment at the top of tokenize.py. IIRC, it re-implements the >> > tokenizer, it does not call the one used for python code. >> >> Ah right, I would be surprised that there would be a public Python API >> to get the tokenizer output, since there is no public C API for that >> :-) >> >> I just removed <token.h> header file since it was never usable outside >> Python C internals: there is no public C API to just run the tokenizer >> and gets its output. >> >> Victor >> _______________________________________________ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UXPSZFOKCKGHUERUVO7UPLZK3L53CGFW/ Code of Conduct: http://python.org/psf/codeofconduct/
