This Engineering Notebook post tells how to avoid using Python's parser in Leo's beautifier. Leo issue #3744 <https://github.com/leo-editor/leo-editor/issues/3744> provides the background. To summarize:
- The new *Orange *class (in leoAst.py) will use neither an ast <https://docs.python.org/3/library/ast.html> (parse tree) nor Leo's *TokenOrderGenerator *class. - The new version will be a prototype for transliterating Orange from Python to Nim. Writing Python's Parser in Nim is out of the question, but writing Python's tokenizer module <https://docs.python.org/3/library/tokenize.html> in Nim should be straightforward. tokenize.py <https://github.com/python/cpython/blob/3.12/Lib/tokenize.py> contains about 500 lines of code. Eliminating the (complex!) TokenOrderGenerator class will save a lot of work. *Discovering context* Four tokens require context knowledge not immediately available from nearby tokens: colons, minus signs, equal signs, and stars ('*' and '**'). The legacy version of the Orange class gets the context from the parse stack: *token.node* is an Ast node. The new version will use *token scanning* to discover the larger context. Function definitions will require a (token-based) scan of the entire statement. This *look-ahead* scan will start at the "def" token and continue *forward. *It will patch context data into later colon, equal sign, and star tokens. The look-ahead scan will work like a recursive descent parser. It's no big deal because the scanner will understand only a fraction of Python. Otherwise, a simple *backward* scan will find the required context. *Summary* Leo's legacy beautifier required Python's ast module and Leo's TokenOrderGenerator class. This approach was elegant but required complex machinery behind the scenes. It would be infeasible to transliterate the legacy code into the Nim language. The new (token-based) beautifier will be slightly less elegant but will require no additional support code. The new code will be the basis for a super-fast beautifier written in Nim. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/3bc3099f-0216-4a10-a06f-bb56ec5aaf0fn%40googlegroups.com.
