For several years now I have been attempting to do something interesting re type checking. Yesterday, that effort finally bore fruit. This post explains the make-stub-files command (see recent revs), tells why it is important and how it can be improved.
*Executive summary* The make-stub-files does for type/design analysis what the c2py command did for converting C to python. The command eliminates most of the drudgery associated with creating python stub (.pyi) files <https://www.python.org/dev/peps/pep-0484/#stub-files>. This makes it possible to use mypy without changing Leo's source code. *What the command does* The make-stub-files command makes python stub (.pyi) files for @<file> x.py nodes "close" to the selected node. Stub files are discussed in pep484 (type hints) <https://www.python.org/dev/peps/pep-0484/> and also this page <http://www.mypy-lang.org/wiki/CreatingStubsForPythonModules> in the mypy documentation <http://mypy.readthedocs.org/en/latest/>. Stub files contain function annotations <https://www.python.org/dev/peps/pep-3107/> (pep 3107) for the mypy type checker. Leo can't use annotations because they raise syntax errors in Python 2.x. The make-stub-files command is *very *simple. It walks the ast (parse) tree <https://docs.python.org/2/library/ast.html> for x.py, writing annotations for all defs and classes to the corresponding x.pyi file in the ~/stubs directory. This is the default directory for mypy stubs. The make-stub-files command knows *nothing* about types! It only knows about naming conventions. At present, this knowledge is hard-wired into this Python dictionary: { 'aList': 'Sequence', 'aList2': 'Sequence', 'c': 'Commander', 'i': 'int', 'j': 'int', 'k': 'int', 'node': 'ast.Ast', 'p': 'Position', 's': 'str', 's2': 'str', 'v': 'VNode', } This is a good start for Leo's own source code. It will be generalized using one or more @data nodes. We want to be able to specify default conventions that can be overridden on a file-by-file basis. It's easy for the command to use these conventions for function arguments. For example, the dict above says to replace: def foo(s): return s by def foo(s: str): return s The command handles function returns is in a pragmatic way. The tree walker simply associates a *list* of return *expressions *with each def. For example, here is the output at the start of leoAst.pyi: class AstDumper: def dump(self, node: ast.Ast, level=number) -> str%(name,sep,sep1.join(aList)), str%str.join(str%(sep,self.dump(z,level+number)) for z in node), str%(name,str.join(aList)), repr(node): ... def get_fields(self, node: ast.Ast) -> result: ... def extra_attributes(self, node: ast.Ast) -> aList: ... Notes: 1. The stub for the dump function is not syntactically correct because there are 4 returns listed. It is up to a *human *to specify a proper return type. In this case, all the returns are obviously (to a human) strings, so a human could replace the first stub by: def dump(self, node: ast.Ast, level=number) -> str: ... 2. At present, the command makes no substitutions for aList, etc. within return values. It will do so fairly soon. So the stub for extra_attributes would become: def extra_attributes(self, node: ast.Ast) -> Sequence: ... 3. The present dictionary contains only class names, but type aliases could also be used to make the results clearer. The definitions of these aliases could be added by hand at the start of the stub file, or could be automatically inserted somehow. 4. Not all types are obvious from naming conventions. In that case, the human will have to update the stub using the actual source code of the def as a guide. For example, the type of "result" in get_fields could be just about anything. Actually, it is a list of strings. *Why this command is important* 1. The command eliminates most of the drudgery from creating stub files. Creating a syntactically correct stub file from the output of the command is straightforward. The command will be made more flexible using @data nodes. 2. The make-stub-files code does *no* type inference. This is a giant breakthrough. We can gain the benefits of mypy without doing *anything* clever. The make-stub-files command is much simpler than c2py. The StubTraverser class overrides only three (simple!) visitors. The base AstFullTraverser class does all the other work. The format_returns method could be improved as discussed above, but even as is the command is extremely useful. 3. Stub files are real data. Mypy will check the syntax for us. More importantly, mypy will do its type inference on the stub files. That means that mypy will a) find errors in the stubs and b) actual type errors in Leo. T*here is now a simple way to use mypy*! 4. Stubs express design intentions as well as types. In most cases, programmers *do* know the expected types of arguments. Up until now, there has been no official/practical way of expressing this knowledge in Python 2. 5. One could imagine a "reverse" command that would inject function annotations into files from stub files. The command would be about as simple as make-stub-files. Edward P.S. The make-stub-files command works as follows: 1. It creates the stub file, x.pyi, if c.p (the selected node) is an @<file> node for x.py. 2. If not,the command creates sub files for all @<file> nodes in c.p.subtree(). 3. If steps 1 and 2 did not find any @<file> nodes, the command looks at p.parents() for the first @<file> node. This is typical behavior for Leo's file-oriented commands. EKR -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
