The file tbo-in-rust.leo (in my ekr-tbo-in-rust
<https://github.com/edreamleo/ekr-tbo-in-rust> repo) now contains the
sources for the Rust and Python tokenizers, as shown by Leo's import
commands. This work concludes (for now) my work with Rust.
As you may recall, Python's tokenizer is much faster than Rust's. The
sources show why. Python's tokenizer is highly optimized C code, complete
with critical sections!
*Code overview*
- Most of Lib/tokenize.py pertains only to the (non-critical) *untokenize*
method.
- Tokenize.py delegates the *tokenize* method to the
*_generate_tokens_from_c_tokenizer* in the *_tokenize* module.
- It took several hours to *find* the _tokenize module! It's in the file
*cpython/Python/Python-tokenize.c* (!!)
The magic happens in the function:
*PyMODINIT_FUNC PyInit__tokenize(void);*
which is part of Python's C-language interface.
*Summary*
*tbo-in-rust.leo* now contains the sources for the Python and Rust
tokenizers. The Python tokenizer is highly optimized C code.
Today's work completes my study of Rust for now.
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/leo-editor/f590dcd4-6540-4fc6-b10e-45e2f40b3764n%40googlegroups.com.
