Re: Wow: a second look at asttokens, fstringify and black

[Edward K. Ream]

On Wednesday, January 15, 2020 at 11:40:23 AM UTC-5, Edward K. Ream wrote:

> I'll investigate how thoroughly asttokens attaches token lists to nodes. 
I had remembered that only ast nodes for statements have attached tokens, 
but now I am not sure at all about that.

Well, I haven't forgotten everything about asttokens.  This unit test:

def test_compare_tog_vs_asttokens(self):
"""Compare asttokens token lists with TOG token lists."""
tag = 'test_compare_tog_vs_asttokens'
try:
import asttokens
except Exception:
self.skipTest('requires asttokens')
contents = """g.blue('wrote %s' % p.x())"""
expected = """g.blue(f'wrote {p.x()}')"""
contents, tokens, tree = self.make_data(contents, description=tag)
# Dump GOT data.
dump_contents(contents)
dump_tree(tokens, tree)
# Dump asttokens data
print('= asttokens =\n')
atok = asttokens.ASTTokens(contents, parse=False, filename=tag)
atok.mark_tokens(tree)
for node in asttokens.util.walk(tree):
print(f"{node.__class__.__name__:>10} {atok.get_text(node)!s}")

Produces this output:

Contents...

1g.blue('wrote %s' % p.x())

Tree...

parent   lines  nodetokens
==   =  ==
 1..2   0.Module:   newline.15(1:0)
  0.Module1.Expr:
  1.Expr2.Call:
  2.Call 13.Attribute:  op.2=. name.3(blue)
  3.Attribute1  4.Name: id='g'  name.1(g)
  2.Call 15.BinOp: op=% op.7=%
  5.BinOp1  6.Str: s='wrote %s' string.5('wrote %s')
  5.BinOp   7.Call:
  7.Call 18.Attribute:  op.10=. name.11(x) op.12
=( op.13=)
  8.Attribute1  9.Name: id='p'  name.9(p)

= asttokens =

Module g.blue('wrote %s' % p.x())
  Expr g.blue('wrote %s' % p.x())
  Call g.blue('wrote %s' % p.x())
 Attribute g.blue
  Name g
 BinOp 'wrote %s' % p.x()
   Str 'wrote %s'
  Call p.x()
 Attribute p.x
  Name p

As you can see, the TOG class assigns tokens to only one node, which is 
much better than the asttokens way, shown above.

This concludes the round of exploration of asttokens. I'll figure out what 
the next steps are tomorrow.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/425e1563-642a--a6df-cded076a6e22%40googlegroups.com.

Re: Wow: a second look at asttokens, fstringify and black

[Edward K. Ream]

On Wednesday, January 15, 2020 at 11:40:23 AM UTC-5, Edward K. Ream wrote:

> I will certainly investigate using asttokens instead of the TOG class. 

I only plan to compare how asttokens attaches token lists to nodes. 
Actually, asttokens uses begin/end token indices instead of actually 
creating variable-length lists of tokens.  The TOG does the same. For the 
comparison I'll change the names of the ivars, so that the AstDumper class 
in leoAst.py will report the token lists created by *asttokens.* 

> "Can the Fstringify class in leoAst.py use ast.walk?"

Clearly, the answer is "yes", because all ast.Binop nodes (and their 
associated tokens) are disjoint. Actually changing the code would be make 
work.

The surprises recorded in this post are *good* news. The work so far has 
> not been wasted:
>
> - It has been intensely enjoyable.
> - It has produced what imo is the best possible definition of token order.
> - It has given me expert's eyes, and created expert-level questions.
>

I forgot to mention the big improvements in TDD that result from using 
pytest and "traditional" unit tests. I have just created #1478 
: deprecate @test 
nodes.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/ca5c7109-1121-4697-a61c-b700358a9a8b%40googlegroups.com.