Re: leoInspect: a hobby with a future?
On Nov 17, 10:26 am, Edward K. Ream edream...@gmail.com wrote: I expect to spend at least the rest of today on format and the AstFormatter class. It was an interesting day, not least because the Fool (capitalized, the skeptical part of me) kept screaming at me. leoInspect is so elegant, I started doubting it! As an antidote, and as a way to honor the skeptical part of me, here is a dialog between the Fool and the excited part of me: Fool: Why all the excitement? All you've done is made a trivial front end for ASTs! Me: The front end is important, for several reasons. First, it provides a way to examine *dead* code more easily than Python's inspect module allows you to examine *live* code. It is also a spectacular collapse in code complexity. Fool: Of *course* it's a collapse in complexity, you've *given up* trying to do a proper lint! What did you expect? Me: Actually, almost all the code is unchanged! Only my *understanding* has changed. Giving up just means that a few algorithms are no longer part of leoInspect itself. Fool: Let's get back to the interface. The getters just deliver ASTs. That's no big deal. Me: Getters deliver Context objects, not ASTs. User query code quite naturally starts with a call to the module getter:: m = leoInspect.module(fn=path to a file) # For users. m = leoInspect.module(s=s) # For testing leoInspect itself. The module getter preprocesses all the code, creating semantic information (Context objects) that speed up and enrich all future getters. **leoInspect adds rich data structures to ASTs**. Fool: How useful can these data structures be? They cost almost nothing to make: there is no such thing as a free lunch. Me: That's my second point. As far as the users of leoInspect are concerned, Context classes *are* query objects. The leoInspect API shows that this change in point of view is a huge advance for users. Fool: Well, what about you? You wasted all day yesterday writing the AstFormatter class, yet another AST-to-Python converter. You haven't done your homework: 2to3lib already has such a thing. Me: Yes, 2to3lib already has something similar. But it doesn't fit in with leoInspect's Context or AstTraverser classes. AstFormatter is essential. Fool: AstFormatter is always going to be buggy. Me: Writing unit tests for AstFormatter will be easy. AstFormatter doesn't need to preserve whitespace exactly. The only requirement is that the *tokenized* version of the input must be equivalent to the *tokenized* version of leoInspect.module(s=s).format(). The new g.python_tokenizer function will suffice as a tokenizer. Fool: Well, writing AstFormatter was a big distraction. Me: Not really. It has given me a chance to look again at the old LintTraverser class, now called InspectTraverser. I used InspectTraverser as a reference while writing AstFormatter. In the process I found some horrible code, namely InspectTraverser.attribute_to_string. This is a wretched hack, supposedly for the benefit of the symbol table classes. It can't possibly be correct. The new code will simply use the AST to represent itself. Fool: Using ASTs to represent themselves? That's a step backwards! It makes the code harder to use. Me: No, it doesn't, because the getters don't get more complex. Yes, the assign_to and assign_using getters must do some work, but that work hides all the AST-related blah-blah-blah from the user. If other getters are needed to hide AST details, I'll put them in. Fool: Maybe the getters will be useful for the naive user, but they will never be good enough to implement a real lint. Me: Wrong, on two counts. First, *I* will be a naive user when it comes to writing unit tests based on leoInspect. I want a dead-simple interface in which to build up significant assertions about Leo's own code base. Second, the o.tree() getter provides a fast trap door to any part of the AST. The leoInspect API *can* be used as the basis for a new lint. Actually, the old sudoku-like (data-driven) lint algorithm could use the underlying Context classes as before. The new leoInspect API doesn't hide the old API. Fool: But you are going to gut attribute_to_string. Isn't that going to ruin some old code? Me: Now you're nit-picking. InspectTraverser.attribute_to_string will be sound, which is kinda important for a lint! Fool: But you are going to waste even more time revising InspectTraverser. Me: It's never a waste of time to put code on sound foundation. And the big collapse in complexity creates further opportunities for simplifications. Don't even *think* about complaining about that. Furthermore, the run-marked-unit-tests-externally command has amplified the Stupendous Aha. As a result, I seem to have gotten smarter--I find even more ways to collapse complexity. This happened several times yesterday, and I expect to simplify InspectTraverser even more today. All the great mathematicians revisit their old proofs, seeking
Re: leoInspect: a hobby with a future?
On Nov 18, 2:35 am, Edward K. Ream edream...@gmail.com wrote: Me: Time to get back to work on leoInspect! Actually, it's time for sleep. I'm a bit short due to the excitement of the leoInspect project. EKR -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Fri, Nov 18, 2011 at 2:35 AM, Edward K. Ream edream...@gmail.com wrote: leoInspect is so elegant, I started doubting it! The quick summary of leoInspect: 1. leoInspect inspects text; Python's inspect module inspects live objects. 2. leoInspect is much easier to use than inspect, and more scriptable. 3. Most importantly, leoInspect is a view of **symbol tables**, not ASTs. These symbol tables are designed to answer important, practical questions about source code, including Leo's sources. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Nov 16, 8:25 pm, Edward K. Ream edream...@gmail.com wrote: When the format method is written, we would replace:: print(z.sd.dump_ast(z.tree())) by: print(z.format()) but that is just syntactic sugar. And rather than printing the raw AST tree shown above, format will print it as it would look in the actual source file. It will take a bit of work tomorrow... The first draft of Context.format was:: def format(self,brief=True): cx = self return ast.dump(cx._tree,annotate_fields=True,include_attributes=not brief) This produces output such as:: Assign(targets=[Attribute(value=Name(id='self', ctx=Load()), attr='c', ctx=Store())], value=Name(id='c', ctx=Load())) Call(func=Attribute(value=Name(id='self', ctx=Load()), attr='beginCommandHelper', ctx=Load()), args=[], keywords=[keyword(arg='ch', value=Str(s='')), keyword(arg='undoType', value=Name(id='undoType', ctx=Load())), keyword(arg='w', value=Attribute(value=Name(id='self', ctx=Load()), attr='w', ctx=Load()))], starargs=None, kwargs=None) This is just a squished version of the ast tree. Alas, ast.dump has no option to reproduce the tree's source. It's a big hole in the ast class, imo. But there is no use complaining. Early this morning I started work on the AstFormatter class, yet another AST traversal class. It's coming along nicely. Here is format's present output corresponding to the dumps above:: self.c = c self.beginCommandHelper(ch='',undoType=undoType ,w=self.w) In other words, format is nearing completion. There are a lot of picky details to handle, however: the formatter must generate code for virtually all AST nodes. In addition, the present code adds spaces around all identifiers, so that constructions such as a or b don't get turned into aorb. This can lead to too many spaces, as you can see above. Eventually, format will beautify its result, perhaps using Leo's token-oriented beautify-python code. I expect to spend at least the rest of today on format and the AstFormatter class. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Nov 16, 5:01 am, Edward K. Ream edream...@gmail.com wrote: As I write this, I see that the problem is to create a query language that can be translated into specific AST traversals. By query language I mean any mechanism that could produce specific searches. API might be a better word for what I imagine. In any case, creating queries of AST trees creates a new world of invention... Here is the first draft of a design. Feel free to skip, but if you are interested, don't miss the last line of this post. The inspiration will be Mathematica's language: simple and task oriented. All details in the background. In fact, there won't be any special query language, just a simple api. AST trees are part of the plumbing. Queries will be made against the **query objects** representing the semantic data created when parsing AST trees. The present new-lint code is already pretty much what we need. Getter functions return query objects, o, or lists of query objects:: o = module(file_name) # Parses the file into a query object, o. aList = assignments(o) aList = classes(o) aList = defs(o) aList = statements(o) ast_tree = tree(o) Assignments are especially important. Two more getters deliver the right-hand-side and left-hand-side of an assignment a:: o = lhs(a) o = rhs(a) The assignments getter will split composite assignments a,b = x,y into separate assignments. Finally, we'll need a way to view query objects, that is, the underlying AST. We'll want a rich set of options so that we can customize the view:: aString = format(o,options) Other bells and whistles may become useful or necessary, but this is a good start, imo. Aha! With these getters, *Python* becomes the query language! So simple. So powerful. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
Just the kind of stuff I like, looking forward to being able to try it without having to understand how it works :-] Probably off topic, but I'm reminded of my reaction to lots of testing tools, especially the coverage stuff which produces listings of execution paths. The intent of the listing is to determine which lines of code are tested, but that's not what I see. When I see that, I don't think of testing, I think what a wonderful, automatically generated, explanation of how the pieces of the app work together, and what the control flow looks like I think the one I've seen is Figleaf, found here http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy Thanks, Kent On Wed, Nov 16, 2011 at 5:01 AM, Edward K. Ream edream...@gmail.com wrote: I'd like a module, call it leoInspect, which would, in effect, provide answers to questions about Leo's source code such as: - Where are all assignments to 'w' in leoEditCommands.py? - Which of those assignments are unusual or suspect (in ways to be specified)? This is, in essence, a re-imagining of the new-pylint project, which has been a hobby project of mine for several years. Rather than attempting global proofs of difficult propositions, as in new- pylint, the leoInspect module answer specific pattern-oriented questions about specific files. We could use such answers while debugging, or as documentation, or especially as the foundation for *fast* unit tests. My first thought was that the simplest thing that could possibly work would be a souped-up search command. As a quick prototype, yesterday I wrote a short script, based on a simple Python tokenizer, that would find non-comment, non-string lines containing assignments. Suppose we are interested, for whatever reason, in assignments to i. Such a script, when run on itself, yields assignments to i like:: result,i,line_number=[],0,0 kind,i= S,i+1 i+= 1 kind,i= S,g.skip_to_end_of_line(s,i) This illustrates the pitfalls of text-based approaches. Finding the real assignments to i will be ugly. In retrospect, this is hardly surprising. An approach based on AST trees will be cleaner and more flexible in the long run. The AST traversal code in new-pylint will form the foundation of leoInspect. Traversals creates semantic data (symbol tables, etc.) that will also be useful in leoInspect. As I write this, I see that the problem is to create a query language that can be translated into specific AST traversals. By query language I mean any mechanism that could produce specific searches. API might be a better word for what I imagine. In any case, creating queries of AST trees creates a new world of invention... Happily, speed is *not* a big problem. The kind of queries I envisage can certainly be done in time proportional to O(N) where N is the size of the code being examined. More sophisticated queries might require multiple passes over the tree (or data structures built from the tree) but that's still O(N). I expect most queries to take less than a second, even on Leo's largest source files. For example, new-lint takes 1.5 seconds to create AST trees for *all* of Leo's sources, and another 2 seconds to traverse all the trees. Your comments please, Amigos. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en. -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Wed, Nov 16, 2011 at 9:20 AM, Edward K. Ream edream...@gmail.com wrote:
Assignments are especially important. Two more getters deliver the
right-hand-side and left-hand-side of an assignment a::
o = lhs(a)
o = rhs(a)
Instead of this, or in addition, there will be the following getters::
aList = assignments_to(arg)
aList = assignments_using(arg)
Here arg can be either a query object or name (string).
With this in place, the script to discover all assignments to 'w' in
leoEditCommands.py is::
import leo.core.leoInspect as inspect
m = inspect.module('leoEditCommands')
for z in m.assignments_to('w'):
print(inspect.format(z))
This is simple enough to be exciting! Some notes:
1. Yes, it really will be possible to use just 'leoEditCommands' as
the argument to module. leoInspect will know about Leo's files.
2. By default, format will delete leading whitespace, line numbers,
file names, etc, but there will be keyword options to enable all that
and more.
3. The script prints all assignments to 'w' anywhere in
leoEditCommands.py. But it would be trivial to zero in on
particular classes or methods with one or two more lines of code.
This is way too good to ignore. I expect to have something working in
an hour or three...
Edward
--
You received this message because you are subscribed to the Google Groups
leo-editor group.
To post to this group, send email to leo-editor@googlegroups.com.
To unsubscribe from this group, send email to
leo-editor+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Wed, Nov 16, 2011 at 9:43 AM, Kent Tenney kten...@gmail.com wrote: Just the kind of stuff I like, looking forward to being able to try it without having to understand how it works :-] That's why I am excited about it as well. This simple, high-level description provides the best possible road map for how the code should work. There is no need for lower-level specs! Probably off topic, but I'm reminded of my reaction to lots of testing tools, especially the coverage stuff which produces listings of execution paths. The intent of the listing is to determine which lines of code are tested, but that's not what I see. When I see that, I don't think of testing, I think what a wonderful, automatically generated, explanation of how the pieces of the app work together, and what the control flow looks like I think the one I've seen is Figleaf, found here http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy Thanks for these remarks and the link. I myself basically never think in terms of execution paths. They are irrelevant to design, or rather, the *only* robust designs are designs that work for *all* execution paths. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Nov 16, 10:33 am, Edward K. Ream edream...@gmail.com wrote: On Wed, Nov 16, 2011 at 9:43 AM, Kent Tenney kten...@gmail.com wrote: Just the kind of stuff I like, looking forward to being able to try it without having to understand how it works :-] That's why I am excited about it as well. This simple, high-level description provides the best possible road map for how the code should work. There is no need for lower-level specs! The clarity makes everything simple. I see now that a query object is simply a new-lint Context object. The getters will be members of the base Context object, and will be very simple iterators. EKR -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Wed, Nov 16, 2011 at 12:11 PM, Edward K. Ream edream...@gmail.com wrote:
The clarity makes everything simple. I see now that a query object is
simply a new-lint Context object.
Rev 4813 contains the first successful demo of leoInspect!
Here is the Leo script::
import leo.core.leoInspect as inspect
dump_modules = False
m = inspect.module(c,'leoEditCommands.py',print_stats=True,print_times=True)
for o in m.classes:
if dump_modules:
o.dump(verbose=False)
else:
print(o)
for f in o.functions:
print(' %s' % f)
And here is the output::
Q
Dump of statistics...
errors: 0
contexts: 732
modules: 1
assignments: 2476
calls: 3241
classes: 22
defs: 602
fors: 85
globals: 5
imports: 18
lambdas: 2
list_comps: 20
withs: 0
attributes: 4493
ivars: 0
names: 5012
del_names: 0
load_names: 5747
param_names: 2
store_names: 2512
parse_time: 0.13
pass1_time: 0.26
total_time: 0.39
class(baseEditCommandsClass)
def(__init__)
def(finishCreate)
def(init)
def(beginCommand)
def(beginCommandWithEvent)
def(beginCommandHelper)
def(endCommand)
def(editWidget)
def(getPublicCommands)
def(getWSString)
def(oops)
def(_chckSel)
def(_checkIfRectangle)
def(getRectanglePoints)
def(keyboardQuit)
class(abbrevCommandsClass)
def(__init__)
def(finishCreate)
def(getPublicCommands)
def(expandAbbrev)
def(dynamicCompletion)
def(dynamicExpansion)
def(dynamicExpandHelper)
def(getDynamicList)
def(addAbbrevHelper)
def(addAbbreviation)
def(addInverseAbbreviation)
def(killAllAbbrevs)
def(listAbbrevs)
def(readAbbreviations)
def(readAbbreviationsFromFile)
def(toggleAbbrevMode)
def(writeAbbreviations)
[big snip]
.
--
Ran 1 test in 0.642s
OK
leoDynamicUnittest.py: 1.55sec
Q
Important: the class and def entries are not just strings. They
are representations of real Context classes, which means that they
already support, or soon will support, the getters I have been talking
about.
In short, we are *very* close to having a fully functional leoInspect module.
Edward
--
You received this message because you are subscribed to the Google Groups
leo-editor group.
To post to this group, send email to leo-editor@googlegroups.com.
To unsubscribe from this group, send email to
leo-editor+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Wed, Nov 16, 2011 at 1:06 PM, Edward K. Ream edream...@gmail.com wrote: Rev 4813 contains the first successful demo of leoInspect! This rev also includes a new version of leoPyRef.leo that contains leoInspect.py. EKR -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Nov 16, 9:20 am, Edward K. Ream edream...@gmail.com wrote: Getter functions return query objects, o, or lists of query objects:: o = module(file_name) # Parses the file into a query object, o. aList = assignments(o) aList = classes(o) aList = defs(o) aList = statements(o) ast_tree = tree(o) Clearly, these getters should be methods of o:: aList = o.assignments() aList = o.classes() etc. EKR -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
Re: leoInspect: a hobby with a future?
On Nov 16, 8:25 pm, Edward K. Ream edream...@gmail.com wrote: All the getters have a simple pattern. Here is the assignments() getter: the others are virtually the same code, with straightforward modifications::: I should emphasize that these methods are part of the base Context class, and are inherited completely unchanged by all subclasses. It's totally elegant. Edward -- You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com. To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
