All Leo's beautification commands now use the new token-based code, except for the beautify-c command. I switched over to the new code "early" to simplify packaging.
The new beautifier has problems with Leonine syntax. *Such problems have existed in all previous beautifiers*. For example, beautifying a copy of leoBeautifier.py generates 3 errors out of 140 nodes. These errors just prevent beautification for a single node, so no great harm is done. Alas, some users may use @doc parts and section references more widely than does Leo's own code, so it's important to fix these problems. This will be done today as follows... *Handling Leonine syntax* Python's tokenizer will generate gobbledegook for code containing Leo directives, section references and @doc parts. Similarly, Python's parser will throw SyntaxError's or IndentationError's on such code. I considered modifying Python's tokenizer module (it's written in Python!) to generate Leo-specific tokens, but there is a simpler, better way: - On a line-by-line basis, the beautifier's top-level *driver* method will convert Leo directives, section references and @doc parts to *special comments*. Such comments will have the form '#' + '!' * N where N >= 5 is chosen so that special comments appear nowhere in the incoming source text. *Note*: decorators must be left alone so that the beautifier properly generates preceding blank lines for def's and class's. - Python's tokenizer and parser will then accept the code containing the special comments. - The token-based beautifier will work as before. The handler for comment tokens will never change or move special comments. - When beautification is complete, the driver will convert special comments back to Leo-specific lines. *Summary* The beautifier will soon be able to handle *all *nodes, regardless of Leonine syntax. The beautifier should be fast enough that @file/@clean nodes can optionally be beautified every time they are written. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
