This morning I took a second look at asttokens, fstringify and black. What I found astounded me.
I had studied the sources asttokens awhile back, but now, looking at the sources with expert's eyes, I see that the asttokens people have solved all the problems that took me so much work. I then looked again at the latest versions of fstringify and black. As explained below, there seems to be no way to recover fstringify's history. It may have been rewritten using a token-based approach. I may be suffering from false memories concerning black. The code is much better than I remember, and the git logs show only incremental improvements. *asttokens* To repeat, the asttokens code solves all the problems that it took me so long to solve: - They somehow discovered that the tokens list contains the primary data. The parse tree is kinda "along for the ride". - The asttokens traversal is *not* done in token order. MarkTokens.def _visit_after_children contains this comment: # Note how this doesn't assume that we iterate through children in order that corresponds to occurrence in source code. - The asttokens code must handle what I call "insignificant" tokens with special code, just as my TOG class does. In particular, there must be special cases for commas and parens. - The asttokens code must "reassign" parens in some cases, just as in the ReassignTokens class in leoAst.py. - MarkTokens._expand_to_matching_pairs is similar to the match_parens function in leoAst.py. There is a *lot* to like in asttokens: - asttokens hands astroid trees as well as ast trees. - asttokens works for both python 2 and 3. - The visitors are much more compact than the visitors in the TOG class. *fstringify* To repeat, fstringify is much different from what I had remembered. To find out more, I git cloned the fstringify code. Alas, gitk is almost useless: apparently, all the code was dumped into the v.0.1 rc (#1) version. gitk transform.py does show one change. transform.py must handle ast.Call nodes on the RHS of the ast.Binop. My newly-expert eyes understands exactly why the revised are necessary. I made the same mistake myself. *black* Again, black is much different from what I had remembered. To find out more, I git cloned the black code. This time there is a full gitk history. This only increases my confusion: all code changes appear to be incremental. I'm starting to think I have somehow conflated important memories. *Questions* 1. After studying asttokens again, I asked myself, "Why didn't the fstringify and black people use asttokens?" After studying the fstringify and black again, I'm not sure that either fstringify or black needs extra help. 2. "Why doesn't the Fstringify class in leoAst.py use asttokens?" I will certainly investigate using asttokens instead of the TOG class. I'll also investigate how thoroughly asttokens attaches token lists to nodes. I had remembered that only ast nodes for statements have attached tokens, but now I am not sure at all about that. 3. "Can the Fstringify class in leoAst.py use ast.walk?" It would seem that it could. The code only needs to handle ast.Binop nodes, so traversal order probably doesn't matter! *Summary* The asttokens, fstringify and black tools are much different than I remember. The fstringify and black tools are under intense development, but that may not explain my faulty memories. The surprises recorded in this post are *good* news. The work so far has not been wasted: - It has been intensely enjoyable. - It has produced what imo is the best possible definition of token order. - It has given me expert's eyes, and created expert-level questions. I must somehow digest today's revelations. The announcement of leoAst.py must wait for days, or weeks. It may never happen. Before doing anything else, I'll thoroughly compare how asttokens and the TOG class assigns tokens to nodes. This must be done before I can form even tentative conclusions about the TOG class. All comments welcome. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/546cb8ae-a286-4c1c-a9e7-4e6a9725b827%40googlegroups.com.
