This post introduces the stand-alone make_stub_files.py script, explaining what it does, how it works and why it is important. The public version of this script is at GitHubGist <https://gist.github.com/edreamleo/5c2d625e223e3d04c11d>. You may also find this script, and an example config file, in scripts.leo. All contributions are welcome.
*Executive summary* The make_stub_files.py script eliminates much of the drudgery of creating python stub (.pyi) files <https://www.python.org/dev/peps/pep-0484/#stub-files> from python source files. To my knowledge, no such tool presently exists. The script does *no *type inference. Instead, it creates function annotations using user-supplied *type conventions*, pairs of strings of the form "name: type-annotation". A *configuration file*, ~/stubs/make_stub_files.cfg, specifies the *source list*, (a list files to be processed), the type conventions, and a list of *prefix lines* to be inserted verbatim at the start of each stub file. This script should encourage more people to use mypy. Stub files can be used by people using Python 2.x code bases. As discussed below, stub files can be thought of as design documents or as executable and checkable design tools. *What the script does* For each file in source list (glob wildcards are allowed), the script creates a corresponding stub file in the output directory. By default, this is ~/stubs, the default directory for mypy stubs. For each source file, the script does the following: 1. The script writes all prefix lines verbatim. This makes it easy to add common code to the start of stub files. For example: from typing import TypeVar, Iterable, Tuple T = TypeVar('T', int, float, complex) 2. The script walks the parse (ast) tree for the source file, generating stub lines for each function, class or method. The script generates no stub lines for defs nested within other defs. For example, given the naming conventions: aList: Sequence i: int c: Commander s: str and a function: def scan(s, i, x): whatever the script will generate: def scan(s: str, i:int, x): --> (see below) *Handling function returns* The script handles function returns pragmatically. The tree walker simply writes a list of return expressions for each def. For example, here is the output at the start of leoAst.pyi: class AstDumper: def dump(self, node: ast.Ast, level=number) -> repr(node), str%(name,sep,sep1.join(aList)), str%(name,str.join(aList)), str%str.join(str%(sep,self.dump(z,level+number)) for z in node): ... def get_fields(self, node: ast.Ast) -> result: ... def extra_attributes(self, node: ast.Ast) -> Sequence: ... The stub for the dump function is not syntactically correct because there are four returns listed. You must edit stubs to specify a proper return type. For the dump method, all the returns are obviously strings, so its stub should be: def dump(self, node: ast.Ast, level=number) -> str: ... Not all types are obvious from naming conventions. In that case, the programmer will have to update the stub using the actual source code as a guide. For example, the type of "result" in get_fields could be just about anything. In fact, it is a list of strings. *The configuration file* As mentioned above, the configuration file, make_stub_files.cfg, is located in the ~/stubs directory. This is mypy's default directory for stubs. The configuration file uses the .ini format. It has two sections: The [Global] section specifies the files list, the output directory and prefix lines. The [Types] section specifies naming conventions. For example: [Global] files: ~/leo-editor/leo/core/*.py output_directory: ~/stub prefix: from typing import TypeVar, Iterable, Tuple T = TypeVar('T', int, float, complex) [Types] aList: Sequence c: Commander i: int j: int k: int n: int node: ast.Ast p: Position result: str s: str v: VNode *Why this script is important* The script eliminates most of the drudgery from creating stub files. Creating a syntactically correct stub file from the output of the script is straightforward. Stub files are real data. mypy will check the syntax for us. More importantly, mypy will do its type inference on the stub files. That means that mypy will discover both errors in the stubs and actual type errors in the program under test. There is now a simple way to use mypy! Stubs express design intentions and intuitions as well as types. We programmers think we *do *know most of the types of arguments passed into and out of functions and methods. Up until now, there has been no practical way of expressing and *testing *these assumptions. Using mypy, we can be as specific as we like about types. For example, we can simply say that d is a dict, or we can say that d is a dict whose keys are strings and whose values are executables with a union of possible signatures. In short, stubs are the easy way to play with type inference. Most importantly, from my point of view, stub files clarify issues that I have been struggling with for many years. To what extent *do *we understand types? mypy will tell us. How dynamic (RPython-like) *are* our programs? mypy will tell us. Could we use type annotation to convert our programs to C. Heh, not likely, but the data in the stubs will tell where things get sticky. Finally, stubs can simplify the general type inference problem. Without type hints or annotations, the type of everything depends on the type of everything else. Stubs could allow robust, maybe even complete, type inference to be done locally. We might expect stubs to make mypy work faster. *Summary* The make-stub-files script does for type/design analysis what Leo's c2py command did for converting C sources to python. It eliminates much of the drudgery associated with creating stub files, leaving the programmer to make non-trivial inferences. Stub files allow us to explore type checking using mypy as a guide and helper. Stub files are both a design document and an executable, checkable, type specification. Stub files allow those with a Python 2 code base to use mypy. One could imagine a similar insert_annotations script that would inject function annotations into source files using stub files as data. This "reverse" script should be about as straightforward as the make-stub-files script. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
