On Nov 17, 10:26 am, "Edward K. Ream" <edream...@gmail.com> wrote:
> I expect to spend at least the rest of today on format and the AstFormatter
> class.
It was an "interesting" day, not least because the Fool (capitalized,
the skeptical part of me) kept screaming at me. leoInspect is so
elegant, I started doubting it!
As an antidote, and as a way to honor the skeptical part of me, here
is a dialog between the Fool and the excited part of me:
Fool: Why all the excitement? All you've done is made a trivial front
end for ASTs!
Me: The front end is important, for several reasons. First, it
provides a way to examine *dead* code more easily than Python's
inspect module allows you to examine *live* code. It is also a
spectacular collapse in code complexity.
Fool: Of *course* it's a collapse in complexity, you've *given up*
trying to do a proper lint! What did you expect?
Me: Actually, almost all the code is unchanged! Only my
*understanding* has changed. "Giving up" just means that a few
algorithms are no longer part of leoInspect itself.
Fool: Let's get back to the interface. The getters just deliver ASTs.
That's no big deal.
Me: Getters deliver Context objects, not ASTs.
User query code quite naturally starts with a call to the module
getter::
m = leoInspect.module(fn=<path to a file>) # For users.
m = leoInspect.module(s=s) # For testing leoInspect itself.
The module getter preprocesses all the code, creating semantic
information (Context objects) that speed up and enrich all future
getters. **leoInspect adds rich data structures to ASTs**.
Fool: How useful can these data structures be? They cost almost
nothing to make: there is no such thing as a free lunch.
Me: That's my second point.
As far as the users of leoInspect are concerned, Context classes *are*
query objects. The leoInspect API shows that this change in point of
view is a huge advance for users.
Fool: Well, what about you? You wasted all day yesterday writing the
AstFormatter class, yet another AST-to-Python converter. You haven't
done your homework: 2to3lib already has such a thing.
Me: Yes, 2to3lib already has something similar. But it doesn't fit in
with leoInspect's Context or AstTraverser classes. AstFormatter is
essential.
Fool: AstFormatter is always going to be buggy.
Me: Writing unit tests for AstFormatter will be easy. AstFormatter
doesn't need to preserve whitespace exactly. The only requirement is
that the *tokenized* version of the input must be equivalent to the
*tokenized* version of leoInspect.module(s=s).format(). The new
g.python_tokenizer function will suffice as a tokenizer.
Fool: Well, writing AstFormatter was a big distraction.
Me: Not really. It has given me a chance to look again at the old
LintTraverser class, now called InspectTraverser. I used
InspectTraverser as a reference while writing AstFormatter. In the
process I found some horrible code, namely
InspectTraverser.attribute_to_string. This is a wretched hack,
supposedly for the benefit of the symbol table classes. It can't
possibly be correct. The new code will simply use the AST to
represent itself.
Fool: Using ASTs to represent themselves? That's a step backwards!
It makes the code harder to use.
Me: No, it doesn't, because the getters don't get more complex. Yes,
the assign_to and assign_using getters must do some work, but that
work hides all the AST-related blah-blah-blah from the user. If other
getters are needed to hide AST details, I'll put them in.
Fool: Maybe the getters will be useful for the "naive" user, but they
will never be good enough to implement a real lint.
Me: Wrong, on two counts. First, *I* will be a "naive" user when it
comes to writing unit tests based on leoInspect. I want a dead-simple
interface in which to build up significant assertions about Leo's own
code base.
Second, the o.tree() getter provides a fast trap door to any part of
the AST. The leoInspect API *can* be used as the basis for a new
lint. Actually, the old "sudoku-like" (data-driven) lint algorithm
could use the underlying Context classes as before. The new
leoInspect API doesn't hide the old API.
Fool: But you are going to gut attribute_to_string. Isn't that going
to ruin some old code?
Me: Now you're nit-picking. InspectTraverser.attribute_to_string will
be sound, which is kinda important for a lint!
Fool: But you are going to waste even more time revising
InspectTraverser.
Me: It's never a waste of time to put code on sound foundation. And
the big collapse in complexity creates further opportunities for
simplifications. Don't even *think* about complaining about that.
Furthermore, the run-marked-unit-tests-externally command has
amplified the Stupendous Aha. As a result, I seem to have gotten
smarter--I find even more ways to collapse complexity. This happened
several times yesterday, and I expect to simplify InspectTraverser
even more today.
All the great mathematicians revisit their old proofs, seeking deeper
understanding and greater clarity. And great authors like Joyce Carol
Oates rewrite their novels many times before they publish. I'm doing
the same thing.
Fool: Anyway, this is keeping you from fixing bugs.
Me: You do have a point. However, leoInspect is part of Leo's test
plan. I'd like to wrap up the leoInspect work asap. Otoh, I'm not
going to rush: this is way too good a project to leave in a half-
finished state.
Fool: Well, we'll have to see how it turns out.
Me: Yes, we will. Time to get back to work on leoInspect!
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To post to this group, send email to leo-editor@googlegroups.com.
To unsubscribe from this group, send email to
leo-editor+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/leo-editor?hl=en.
