Two days ago, all unit tests passed for Leo's new beautifier in leoTokens.leo. My celebrations were premature. Beautifying Leo's sources revealed unexpected (and unwelcome!) changes.
This Engineering Notebook post summarizes the remaining issues and suggests possible fixes. *Background* Leo's legacy colorizer (in leoAst.py) uses data from parse trees to discover proper spacing around the colon, minus sign, star, and 'import' tokens. leoTokens.py does not have that data, so it must compute new *context data* to resolve ambiguities. Both colorizers contain *visitors *and *generators*. Visitors handle *input tokens*; generators create *output tokens*. The new generator adds *scanners* that discover context. *Unexpected scanning problems* The new beautifier fails because several ad-hoc scanners can move beyond statement boundaries. The unit tests didn't catch such situations because they focused on GvR's single-line pet peeves. *Solutions* I spent yesterday noodling potential solutions. As I went to bed, I saw that the existing scanners are *parts* of a recursive-descent parser. The new beautifier needs a *good enough* parser. This morning, the details became clear: - New *scan_statements* and *scan_statement *methods will discover statement boundaries. They are the top levels of the parser. - The problematic scanners will use these boundaries to avoid mistakes. *Summary* New scanners will provide the necessary context to problematic code. Various questions remain, but the new scanners are unlikely to impact performance significantly. The new scanners complete a *good enough parser*. The final code will look as though the parser was an obvious choice :-) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/5f947d80-2295-4479-bad8-310d5d562137n%40googlegroups.com.
