On Wednesday, January 15, 2020 at 11:40:23 AM UTC-5, Edward K. Ream wrote:
> I'll investigate how thoroughly asttokens attaches token lists to nodes.
I had remembered that only ast nodes for statements have attached tokens,
but now I am not sure at all about that.
Well, I haven't forgotten everything about asttokens. This unit test:
def test_compare_tog_vs_asttokens(self):
"""Compare asttokens token lists with TOG token lists."""
tag = 'test_compare_tog_vs_asttokens'
try:
import asttokens
except Exception:
self.skipTest('requires asttokens')
contents = """g.blue('wrote %s' % p.x())"""
expected = """g.blue(f'wrote {p.x()}')"""
contents, tokens, tree = self.make_data(contents, description=tag)
# Dump GOT data.
dump_contents(contents)
dump_tree(tokens, tree)
# Dump asttokens data
print('===== asttokens =====\n')
atok = asttokens.ASTTokens(contents, parse=False, filename=tag)
atok.mark_tokens(tree)
for node in asttokens.util.walk(tree):
print(f"{node.__class__.__name__:>10} {atok.get_text(node)!s}")
Produces this output:
Contents...
1 g.blue('wrote %s' % p.x())
Tree...
parent lines node tokens
====== ===== ==== ======
1..2 0.Module: newline.15(1:0)
0.Module 1.Expr:
1.Expr 2.Call:
2.Call 1 3.Attribute: op.2=. name.3(blue)
3.Attribute 1 4.Name: id='g' name.1(g)
2.Call 1 5.BinOp: op=% op.7=%
5.BinOp 1 6.Str: s='wrote %s' string.5('wrote %s')
5.BinOp 7.Call:
7.Call 1 8.Attribute: op.10=. name.11(x) op.12
=( op.13=)
8.Attribute 1 9.Name: id='p' name.9(p)
===== asttokens =====
Module g.blue('wrote %s' % p.x())
Expr g.blue('wrote %s' % p.x())
Call g.blue('wrote %s' % p.x())
Attribute g.blue
Name g
BinOp 'wrote %s' % p.x()
Str 'wrote %s'
Call p.x()
Attribute p.x
Name p
As you can see, the TOG class assigns tokens to only one node, which is
much better than the asttokens way, shown above.
This concludes the round of exploration of asttokens. I'll figure out what
the next steps are tomorrow.
Edward
--
You received this message because you are subscribed to the Google Groups
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/leo-editor/425e1563-642a-4444-a6df-cded076a6e22%40googlegroups.com.
