On Tue, Jan 14, 2020 at 10:59 PM Brian Theado <[email protected]> wrote:
I like your second comment > <https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-573661883>. > It is well written and informative. Nice job. > Thanks. > You stress "token order traversal". How does that contrast the traversal > provided by the ast library? > ast.walk is feeble. Formally, it visits nodes "in no particular order", which actually means that it visits nodes in a useless order ;-) In contrast, TOT.traverse (which requires the links injected by the TOG class), visits nodes in token order, which is the order that will be most useful for any tool that modifies python source code. Does your library still traverse the ast nodes in the same order as the ast > library does, just that with each visited node you will have access to all > the tokens corresponding to that node? > The ast module (ast.walk) defines no order at all. Furthermore, the TOT.traverse method must run *after* the TOG class has created all links. So when links *are* available, they are available to all nodes. You have, however, put your finger on something that probably should be documented. The Fstringify class in leoAst.py replaces many tokens with a single, new, "fstringified" string. In the process, it converts all the old tokens to "do-nothing" tokens. This is easy to do. I'll omit the details, except to say that all links are adjusted properly. However, the Orange class, may *increase *the net number of tokens. It would be possible to readjust links from nodes to tokens, but that would be messy. Instead, I'll probably replace one or more tokens with an *extension token*. Details not clear yet. Back to the first comment: > > A *children* array from each ast node to its children. Order matters! > > > In contrast, how does the ast library link the ast nodes? Not at all? > Linked, but out of order? Something else? > Something else. Some (but not all!) nodes have lineno, col_offset, end_lineno end_col_offset fields. For further fun, previous versions of Python did not compute these fields properly. And this: > > - For each token, *token.node* contains the ast.node "responsible" for > the token. > - For each ast node, *node.first_i* and *node.last_i* are indices into > the token list. > These indices give the range of tokens that can be said to be > "generated" by the ast node. > > Does this imply the user can choose between traversing each token (and > from the token reach the node) and traversing each node (and from the node > reach each token)? In what case would the user want to do one over the > other? > Another great question. My long answer to your previous reply mostly answered this question. Imo, the *token* view is usually the most natural view for tools that modify python sources. The more I work with ast trees, the more their deficiencies become apparent. I suspect the authors of fstringify and black overvalue parse trees and undervalue tokens. All the big Ahas involved in creating the TOG class involved realizing that tokens were essential to help the traversal of the parse tree. I had no idea that parse trees were so feeble. It was a big big change in my attitude. The Fstringify class is implicitly based on tokens, not parse trees. The changes are made to the token list, and the result is the tokens_to_string(self.tokens). This is so elegant that readers may not understand what is going on! The Orange class will also be based on tokens. This will be much more obvious in the Orange class because most of the "beautifying" will be done on tokens. The beautifier converts input tokens to output tokens. My present plan is to use parse-tree views only for difficult cases: 1. Determine the exact spacing for slices. The present code doesn't have access to enough context. Using the parse tree should be the easy way to get that context. 2. Determine how to break long lines. I'm not sure what will happen. Perhaps a purely token-oriented view will suffice. We shall see. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS2iGup_zCEQ_Cv2%3DnCqk-LT8%3DpPMwzQ5-h5z3PKU8Pv3Q%40mail.gmail.com.
