On Saturday, May 23, 2015 at 7:47:01 AM UTC-5, Edward K. Ream wrote:
> *Descriptive code generation*
> *A global code list*
> *A peephole*
It is almost unbelievable how simple everything has become.
*A peephole pass is not needed*
"Look behind" suffices. Most code generators need only examine the
previous token, if any. The blank_lines generator scans backwards over a
contiguous range of blank-lines and line-end tokens.
Here is LeoTidy.blank, the crucial code generator for blanks:
def blank(self):
'''Add a blank request on the code list.'''
prev = self.code_list[-1]
if prev.kind not in (
'blank','blank-lines',
'file-start','line-start','line-end',
'lit-no-blanks','lt',
):
self.add_token('blank',' ')
Most generators are even simpler. Some examples:
def clean(self,kind):
'''Remove the last item of token list if
it has the given kind.'''
prev = self.code_list[-1]
if prev.kind == kind:
self.code_list.pop()
def lit(self,s):
'''Add a request for a literal to the code list.'''
assert s and g.isString(s),repr(s)
self.add_token('lit',s)
def lit_blank(self,s):
'''Add request for a literal (no previous blank)
followed by a blank.'''
self.clean('blank')
self.lit(s)
self.blank()
def lit_no_blanks(self,s):
'''Add a request for a literal *not* surrounded by blanks.'''
self.clean('blank')
self.add_token('lit-no-blanks',s)
def word(self,s):
'''Add a word request to the code list.'''
assert s and g.isString(s),repr(s)
self.blank()
self.add_token('word',s)
self.blank()
Note: the code list starts with a file-start token, so self.token_list[-1]
always exists.
*Look behind can handle post processing*
LeoTidy.format ends the file with a file-end token. Its code generator
ensures a single trailing newline:
def file_end(self):
'''
Add a file-end token to the code list.
Retain exactly one line-end token.
'''
while True:
prev = self.code_list[-1]
if prev.kind in ('blank-lines','line-end'):
self.code_list.pop()
else:
break
self.add_token('line-end')
self.add_token('file-end')
More interestingly, the FunctionDef visitor could insert arg-start and
arg-end tokens into the token list. The code generator for arg-end could
scan backwards for the corresponding arg-start token. All the intervening
tokens are the actual function arguments. These can then be split into
separate lines as needed.
*The OutputToken class represents all tokens without the need for
subclasses*
class OutputToken:
'''A class representing items on the LeoTidy code list.'''
def __init__(self,kind,level,value):
self.kind = kind
self.level = level
self.value = value
def __repr__(self):
return 'OutputToken %10s %s' % (self.kind,repr(self.value))
__str__ = __repr__
def to_string(self):
'''Convert an output token to a string.'''
return self.value if g.isString(self.value) else ''
*Flattening the token list is trivial*
return ''.join([z.to_string() for z in self.code_list])
*Debugging is easy*The tokens in the code list represent the intentions of
the node visitors. This is a breakthrough in debugging.
*The code is fast*
Here is a recent run:
test @file leoFileCommands.py
run AddTokensToTree: n2 2449
format (LeoTidy) ===== self.new: True
test nodes: 8226
test tokens: 15281
test code_list 19720
test len(s2): 70211
test parse: 0.03 sec.
test tokenize: 0.09 sec.
test add toks: 0.07 sec.
test format: 0.18 sec.
test total: 0.36 sec.
I haven't gotten around to comparing this with PythonTidy, but I suspect it
is much faster. It is easily 50x faster than autopep8.
*Summary*
This project has already succeeded beyond my wildest dreams. You need only
compare this code with the PythonTidy code to know that something truly
remarkable has happened.
It will be easy to extend the line-end handling:
- to add comments associated with each line.
- to retain blank lines in the source, if desired.
The g.List functions are now deprecated. This code, of which I was so
proud, is now seen to be grossly inferior to a token-based approach.
That's all for now. I expect to have most of the code complete in a day or
so.
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.
