I like your second comment <https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-573661883>. It is well written and informative. Nice job.
You stress "token order traversal". How does that contrast the traversal provided by the ast library? Does your library still traverse the ast nodes in the same order as the ast library does, just that with each visited node you will have access to all the tokens corresponding to that node? Back to the first comment: A *children* array from each ast node to its children. Order matters! In contrast, how does the ast library link the ast nodes? Not at all? Linked, but out of order? Something else? And this: - For each token, *token.node* contains the ast.node "responsible" for the token. - For each ast node, *node.first_i* and *node.last_i* are indices into the token list. These indices give the range of tokens that can be said to be "generated" by the ast node. Does this imply the user can choose between traversing each token (and from the token reach the node) and traversing each node (and from the node reach each token)? In what case would the user want to do one over the other? On Tue, Jan 14, 2020 at 7:06 PM Brian Theado <brian.the...@gmail.com> wrote: > In the theory of operation: > > "The notion of a *token order traversal* of a parse tree is the > foundation of this project" > > In contrast, what traversal order do parse trees provide? How is token > order different/better? What does it allow me to do that I can't otherwise > do with parse trees? > > "This help is essential, because the following two source files generate > identical parse trees!" > > This implies to me that with a parse tree you can't do a round trip from > source code, to a parse tree and back to the original source code. Is that > correct? If so is this just one externally visible benefit to your library > or is it the main benefit? If it is the main benefit, then I think it > should be made much more clear earlier in the document like in the Overview > near here: > > "These links promise to collapse the complexity of any code that changes > text, including the asttokens <https://pypi.org/project/asttokens/>, > fstringify <https://pypi.org/project/fstringify/>, and black > <https://pypi.org/project/black/> projects." > > something like: > > "These links allow portions of python source code to be transformed while > leaving the rest untouched. Much of the complexity of the asttokens > <https://pypi.org/project/asttokens/>, fstringify > <https://pypi.org/project/fstringify/>, and black > <https://pypi.org/project/black/> projects comes from not being able to > link between the text and structure of the Python code. Using the code of > this library can collapse the complexity of these and any projects which > change Python text" > > On Tue, Jan 14, 2020 at 6:10 PM Brian Theado <brian.the...@gmail.com> > wrote: > >> Ok, now I see in your proposed email response in the other thread you do >> mention the asttokens project. And also later in #1440 you said "The >> asttokens <https://asttokens.readthedocs.io/en/latest/> project assigns >> tokens to ast nodes, but only for statement-level ast nodes." That's useful >> to know. >> >> One thing I really like about the front page of that project ( >> https://github.com/gristlabs/asttokens) is the front-and-center example >> which shows in a few lines of code how to call the code using concrete >> inputs and shows exactly what the outputs are. >> >> I find that very valuable. Examples are usually the first thing I want to >> see. Can you have something similar in #1440? >> >> On Tue, Jan 14, 2020 at 5:59 PM Brian Theado <brian.the...@gmail.com> >> wrote: >> >>> I just followed the stackoverflow link ( >>> https://stackoverflow.com/questions/16748029/how-to-get-source-corresponding-to-a-python-ast-node#) >>> and someone posted that they created >>> https://github.com/gristlabs/asttokens which "Annotates Python AST >>> trees with source text and token information". Is that different from >>> what your code does? >>> >>> On Tue, Jan 14, 2020 at 8:02 AM Edward K. Ream <edream...@gmail.com> >>> wrote: >>> >>>> In a day or three I'll be announcing #1440 >>>> <https://github.com/leo-editor/leo-editor/issues/1440>, the >>>> unification of parse trees and tokens, to python's core developers. I'll >>>> do that by making a comment in this related python issue >>>> <https://bugs.python.org/issue33337>. >>>> >>>> In preparation for that announcement, I have revised/created the first >>>> three comments of #1440: >>>> >>>> - The first (main) comment >>>> <https://github.com/leo-editor/leo-editor/issues/1440> gives an >>>> overview of the project and its status. >>>> - The second comment >>>> <https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-573661883> >>>> is the Theory of Operation. >>>> - The third comment >>>> <https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-574145510> >>>> contains the Project's history. >>>> >>>> The announcement itself will be fairly brief, and will refer readers to >>>> #1440 for more details. So I want these three comments to be "perfect". >>>> >>>> Please read and critique all three comments. I am interested in your >>>> comments and suggestions. >>>> >>>> Edward >>>> >>>> P. S. There are similar discussions in LeoDocs.leo, but I'll be >>>> referring to #1440 in the upcoming public announcements. The links above >>>> are good looking and are available to anyone, including those with no >>>> access to Leo. >>>> >>>> EKR >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "leo-editor" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to leo-editor+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/leo-editor/5c6da99b-360f-4ab2-bae9-f9a4d589e860%40googlegroups.com >>>> <https://groups.google.com/d/msgid/leo-editor/5c6da99b-360f-4ab2-bae9-f9a4d589e860%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAO5X8CxvYsoC92DzSQgAsMFiOvUHizmdfXssuqVi%3DOdAAd6XBw%40mail.gmail.com.
