I didn't quite get what the big effect is. Saving 30 milliseconds?
Also, how is the now-split-off funcionality to be invoked? Does it require two or more imports now, or it's imported on demand when one
invokes an appropriate test.support entry?
On 23.03.2021 4:29, Victor Stinner wrote:
Hi,
tl;dr Tests of the Python Test Suite now start with 131 imported
modules, instead of 233. For example, asyncio, logging,
multiprocessing and warnings are no longer imported by default.
--
The Python test suite is run by a custom test runner called
"libregrtest" (test.libregrtest). It can detect reference leaks, write
the output as JUnit XML, run tests in parallel with multiple
processes, etc. Tests are written with test.support which is a
collection of helper functions.
Over the years, test.support got more and more functions, and
libregrtest got more and more features. The problem is that they
import more and more modules. For example, "import test.support"
imports 173 modules in Python 3.8!
$ python3.8
import sys, sys
before=set(sys.modules); import test.support; after=set(sys.modules)
len(after - before)
173
In these imports, you can find some "heavy" modules like asyncio and
multiprocessing. Moreover, some modules have "side effects" on import.
For example, "import logging" registers an "at fork" callback.
In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to
reduce the number of test.support imports from 171 to 25! test.support
was splitted into multiple sub-modules:
- bytecode_helper
- hashlib_helper
- import_helper
- logging_helper
- os_helper
- script_helper
- socket_helper
- threading_helper
- warnings_helper
I continued the work in bpo-41718 to reduce the number of libregrtest
imports, since libregrtest also imported many modules.
A dummy test which does nothing in the master branch now only has 131
modules in sys.modules, whereas in Python 3.9 it has 233 (+102)
modules! For example, asyncio, logging, multiprocessing and warnings
are no longer imported by default.
"import test.support" is now faster. master branch compared to Python
3.8 using ./python -c 'import test.support' command and Python built
in release mode:
[py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster
More realistic command running a single test method: ./python -m test
test_os -m test_access
[py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster
The only annoying point is that splitting test.support into
sub-modules was not backported to Python 3.8 and 3.9, and so it will
make backports a little bit more annoying. Sorry about that!
--
If you know that two modules should be tested together, please write a
dedicated test for that ;-)
Victor
--
Regards,
Ivan
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/3Q2VZVU3BEON36Z4JW6EUCMN4AZJYTF6/
Code of Conduct: http://python.org/psf/codeofconduct/
