I have just completed a spectacular simplification of the TokenOrderGenerator class.
The tog.advance_str and and tog.advance_if machinery is gone. These methods "polluted" all visitors involving "else" tokens. No more. This would never have happened were it not for a hole I found yesterday. Once again, the parse tree can not distinguish between two different sets of tokens. The following generate exactly the same parse trees: a = [ b : ] a = [ b : : ] My first thought was to implement tog.advance_slice, but then, Aha!, I saw that a *single *pointer suffices to sync tokens with parse tree nodes. This pointer is tog.px, set *only* by tog.sync_tokens. It's just a bit tricky to use tog.px to look ahead in the token list. That's what tog.find_next_significant_token and tog.get_concatenated_string_tokens do. *Details* Instead of three or four indices that step through the token list, a single index, tog.px suffices. Multiple indices suffered from theoretical as well as practical problems. It was hardly obvious that the various pointers would play well together. Only visitors for ast.Bytes, ast.If, ast.JoinedStr, ast.Slice and ast.Str ever need to look ahead in the token list. This simplifies several other visitors. *Summary* The fstringify branch now converts all Leo files to f-strings without obvious problems. The new code should be much easier to understand and maintain. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/58ad92f6-6553-4e4d-b000-34dd6a081e0e%40googlegroups.com.
