Work is going much faster than expected. The checkin log for d2efae5:
Two milestones:
1. Strings are now spelled properly.
2. LeoTidy processes all files in Leo's core in 14.6 sec.
Stats for @file leoCommands.py:
nodes: 27085
tokens: 49809
code_list 66661
len(s2): 239621
parse: 0.09 sec.
tokenize: 0.29 sec.
add toks: 0.28 sec.
format: 0.42 sec.
total: 1.07 sec
In fact, *all *token data is now easily available to Ast nodes. That is,
the AddTokensToTree class is complete.
A big collapse in complexity has made this class rock solid. There are no
significant "if" statements in this class! This shows that all necessary
tokens can be associated with Ast nodes.
The rest of this post is an ENB entry. Feel free to ignore.
On my bicycle ride today I saw that there must be a 1-1 correspondence
between string *tokens *and Ast.Str *nodes*. So add.make_tokens_data just
creates a global string_list.
The crucial AddTokensToTree.set_str_token method pops items from the start
of this list and injects them into the node.str_spelling field of Ast.Str
node:
def set_str_token (self,node):
'''Associate a token with a ast.Str node.'''
data = self.strings_list.pop(0)
n,s = data
node.str_spelling = s
And here is LeoTidy.do_Str:
def do_Str (self,node):
'''This represents a string constant.'''
self.lit(node.str_spelling)
Hehe. It doesn't get any better than this!
The AddTokensToTree.visit method makes this all work:
def visit(self,node):
'''AddTokentsToTree.visit.'''
self.n_visits += 1
name = node.__class__.__name__
if hasattr(node,'lineno'):
# ast.expr and ast.stmt nodes
# have lineno attributes.
if name == 'Str':
self.set_str_token(node)
elif name in self.statements_d:
self.set_tokens(node)
method = getattr(self,'do_' + name)
method(node)
As a result, only the do_Str method must be overridden, as shown above.
If you think this code is inherently easy, take a look at the corresponding
PythonTidy code...
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.
