This Engineering Notebook post introduces a new core module: *leoTokens.py*. This file contains Leo's new token-oriented beautifier, discussed in this ENB <https://groups.google.com/g/leo-editor/c/sfNK5EM3yhw/m/TJevU7IvAwAJ>. I plan to transliterate leoTokens.py into Nim.
*Background* PR #3757 <https://github.com/leo-editor/leo-editor/pull/3757> contains the present code. It is still a work in progress. PR #3757 is complex! Three earlier PRs contributed to the work. I could not have refactored the code without Leo. *leoAst.py* contains Leo's legacy beautifier. This beautifier uses the *TokenOrderGenerator* (*TOG*) class, also defined in leoAst.py. *Goals for leoTokens.py* - Fixed type annotations of all data, functions, and methods. - Minimal code size. - Minimal dependence on other Leo modules. *Aha: keep leoAst.py* My initial thought was to *replace* the legacy beautifier with the tokens-only beautifier. But this was a bad idea for at least three reasons: - The Orange class (the legacy beautifier) is an excellent demo of how to use the TOG. - We want to be able to compare the old and new beautifiers. - Adding another class to leoAst.py would complicate an already too-complex module. *leoTokens.py* leoTokens.py started life as a complex merger of the PRs mentioned above. Details omitted! I then removed unnecessary classes, methods, and functions. En route, I simplified annotations. This pruning process ended today: leoTokens.py contains the minimum code required! I saw that some helpers in *leoAst.py* were unnecessary. Again, details omitted. *Summary* Separating *leoTokens.py* from *leoAst.py* has simplified both files. Similar remarks apply to the corresponding unit test files. leoTokens.py is now as small as possible. It is also fully annotated. This preliminary work will make transliterating leoTokens.py into Nim much easier. leoTokens.py is still a work in progress. I must demonstrate that a tokens-only beautifier can infer context *without* the help of a parse tree. I am confident that token scanning can do the job. The existing unit tests will be my guide! Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/d3a7bc63-bb4d-4fcd-bb6d-d7bf39283fa2n%40googlegroups.com.
