Hello fellow developers,
Pytype is a Python type checker able to do infer some of the missing type annotations. This is quite valuable in a codebase as large as ours and catch real bugs that a patch would introduce on a regular basis. It is also invaluable for secondary tooling, like TortoiseHG that have been extensively using it.
However it is quite slow to work on such a large code base, and the associated CI job routinely take 45 to 60 minutes.
Some of the slowness can be accounted for implementation issues within the pytype tool itself, but some other comes from a recurrent pattern in the Mercurial code base. Import cycles.
For quite a while, we forbid "top level" circular imports. However we introduce many "inline" circular import over the year. Where we import a module that would create a cycle directly in the function using it. This trick does not help pytype, that detect the dependencies in the typing and deal with is poorly.
First, pytype decide to skip one item of the cycle, to break it. And doing so, it won't be able to cache reusable inference information for item that are part of the cycle… and all their dependencies. As a result, even if the CI has a cache setup for pytype, it remain unused for about 90% of the file…
Second,. while pytype process file in parallel, it handle all item in the cycle sequentially. So the big cycle in Mercurial (~100 files) take probably around 80% of the total runtime for the full ~550 files that pytype is checking.
So I propose that we stop accepting "local import" to prevent cycle from now.
In addition, the current cycles needs to be broken through
refactoring¹. This usually result in simpler and saner module
graph anyway, for example scmutil.py use such import a lot and
breaking this 2500 line module apart cannot hurt.
(as a side note, we could start using faster tool, like mypy or
ty to check do some of the type checking. While being less
advanced than pytype, it can run in a few second which is quite
useful)
-- Pierre-Yves David
[1] (e.g. https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1362 )
_______________________________________________ Mercurial-devel mailing list Mercurial-devel@lists.mercurial-scm.org https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel
