On Thursday, May 21, 2015 at 9:58:29 AM UTC-5, Edward K. Ream wrote: Much progress to report. Feel free to ignore the rest of this post. The details are entries in the ENB.
> The ast-based approach seems more natural. It *would *be more more natural if tokens could be associated with ast statement nodes. Rev 8f9563f contain a proof of concept. The new AddTokensToTree class in leoBeautify.py contains the code. It is fast, straightforward and flexible. The general idea is to generate all the tokens, make various data structures describing the tokens, and then use these data to inject summary data into ast nodes for statements. The base AstFullTraverser class does all the actual traversing. Only the visit method was overridden and it just calls set_tokens. set_tokens uses precomputed data to inject data. Associating tokens with parse-tree nodes is fast. For example: test @file leoFileCommands.py run AddTokensToTree: n2 2449 test nodes: 8226 test tokens: 15281 test total: 0.18 sec. test parse: 0.02 sec. test tokenize: 0.09 sec. test add toks: 0.07 sec. parse is the time taken by ast.parse. tokenize is the time taken by list(tokenize.generate_tokens(readlines)) add toks is the time taken by AddTokensToTree. In other words, AddTokensToTree runs in time proportional to the C code in tokenize.generate_tokens. In any event, AddTokensToTree takes less than 0.1 sec. for a large file. No doubt the code will jiggle over time, but I have no further doubts that the project is feasible. After all these years of complaining to myself about missing data, it's great to know that the problem isn't serious. The next step will be to create a *Null Formatter*. The will return exactly the same code as the input code. When this works the project will be almost complete :-) Furthermore, at every stage of the project, asserting that the pre and post parse trees are identical provides an excellent unit test. But this test does test comments at all, so other tests will be needed. In short, this project is already well on its way. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
