v2: Forgot the Upstream-Status for the setuptools patch file in v1. v3: Correct subject line numbering and patch revision details in v2.
As explained in https://github.com/pypa/setuptools/issues/510, the pkg_resources module in setuptools is very slow to initialize scripts due to how it scans the Python path for each module that would import pkg_resources. My understanding of upstream Python's response is that as of 3.7 and later, importlib.resources is available for replacing this functionality in new scripts, but existing code that uses pkg_resources will still have to be re-written to use it (or something like python3-fastentrypoints. I've done some performance testing with this patch using the following, very simple steps: 1) Add "IMAGE_INSTALL_append = " python3-setuptools" to local.conf; 2) bitbake core-image-full-cmdline 3) mkdir -p test/minimal; 4) In test/minimal/__init__.py, add: def main(): print('hello world') if __name__ == '__main__': main() 5) In test/setup.py: from setuptools import setup, find_packages setup( name='minimal', version='0.1', python_requires='>=3.4', packages=['minimal'], entry_points={ 'console_scripts': [ 'minimal = minimal:main' ] }, ) 6) Run "python3 setup.py install" 7) Run "time /usr/bin/minimal" 8) Note the drastic differences in run time, e.g. with core-image-full-cmdline: core-image-full-cmdline, WITHOUT setuptools ScriptWriter patch: root@qemux86-64:~# time /usr/bin/minimal hello world real 0m0.198s user 0m0.174s sys 0m0.023s core-image-full-cmdline, WITH setuptools ScriptWriter patch: root@qemux86-64:~# time /usr/bin/minimal hello world real 0m0.034s user 0m0.024s sys 0m0.010s I've also tried to ensure that this has no adverse effects on the larger python ecosystem in Yocto, by running ptests on all modules in meta-python that have ptests available and use setuptools, with this patch installed. I saw no failures related to setuptools, although it is still possible that this could have unforeseen consequences on other modules. Finally, some (truncated) profiling comparisons to emphasize the gains in performance, using the same example module before and after the patch. There are three orders of magnitude difference in the number of calls required, including much more recursion. What is most apparent from these profiling results is how much more time the unpatched version spends parsing entry points when that effort isn't required to achieve the same end. As can be seen from the simple example used here, the time savings at initialization are significant enough even for tiny scripts that this fix should be incorporated in the project at the setuptools level, rather than requiring manual application, or modification of existing scripts on the user's end. WITHOUT PATCH: 134641 function calls (130543 primitive calls) in 0.723 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 88/1 0.002 0.000 0.723 0.723 {built-in method builtins.exec} 1 0.000 0.000 0.723 0.723 minimal:3(<module>) 100/2 0.002 0.000 0.704 0.352 <frozen importlib._bootstrap>:986(_find_and_load) 100/2 0.002 0.000 0.703 0.352 <frozen importlib._bootstrap>:956(_find_and_load_unlocked) 93/2 0.002 0.000 0.700 0.350 <frozen importlib._bootstrap>:650(_load_unlocked) 67/1 0.001 0.000 0.699 0.699 <frozen importlib._bootstrap_external>:777(exec_module) 115/1 0.000 0.000 0.698 0.698 <frozen importlib._bootstrap>:211(_call_with_frames_removed) 1 0.000 0.000 0.698 0.698 __init__.py:2(<module>) 30/19 0.000 0.000 0.445 0.023 {built-in method builtins.__import__} 82 0.002 0.000 0.372 0.005 re.py:289(_compile) 75 0.000 0.000 0.369 0.005 re.py:250(compile) 73 0.002 0.000 0.367 0.005 sre_compile.py:759(compile) 1 0.000 0.000 0.262 0.262 requirements.py:4(<module>) 73 0.001 0.000 0.259 0.004 sre_parse.py:937(parse) 308/73 0.007 0.000 0.255 0.003 sre_parse.py:435(_parse_sub) 457/77 0.072 0.000 0.253 0.003 sre_parse.py:493(_parse) 283/280 0.009 0.000 0.222 0.001 {built-in method builtins.__build_class__} 9/7 0.000 0.000 0.159 0.023 <frozen importlib._bootstrap>:613(_load_backward_compatible) 6 0.000 0.000 0.159 0.026 __init__.py:35(load_module) 1 0.001 0.001 0.145 0.145 pyparsing.py:26(<module>) 26 0.001 0.000 0.137 0.005 pyparsing.py:2779(__init__) 73 0.001 0.000 0.104 0.001 sre_compile.py:598(_code) 1 0.000 0.000 0.097 0.097 parser.py:5(<module>) 1 0.000 0.000 0.095 0.095 feedparser.py:5(<module>) 1 0.000 0.000 0.095 0.095 specifiers.py:4(<module>) 770/73 0.032 0.000 0.091 0.001 sre_compile.py:71(_compile) 1 0.000 0.000 0.086 0.086 _policybase.py:1(<module>) 11488 0.049 0.000 0.086 0.000 sre_parse.py:254(get) 1 0.000 0.000 0.083 0.083 specifiers.py:275(Specifier) 1 0.000 0.000 0.073 0.073 pyparsing.py:5399(pyparsing_common) 98 0.003 0.000 0.063 0.001 <frozen importlib._bootstrap>:890(_find_spec) 1 0.000 0.000 0.058 0.058 utils.py:5(<module>) WITH PATCH: 300 function calls (299 primitive calls) in 0.003 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 2/1 0.000 0.000 0.003 0.003 {built-in method builtins.exec} 1 0.000 0.000 0.003 0.003 minimal:2(<module>) 1 0.000 0.000 0.003 0.003 <frozen importlib._bootstrap>:986(_find_and_load) 1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:956(_find_and_load_unlocked) 1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:890(_find_spec) 1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap_external>:1334(find_spec) 1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap_external>:1302(_get_spec) 2 0.000 0.000 0.001 0.001 <frozen importlib._bootstrap_external>:1431(find_spec) 1 0.000 0.000 0.001 0.001 <frozen importlib._bootstrap>:650(_load_unlocked) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1479(_fill_cache) 18 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:62(_path_join) 1 0.000 0.000 0.000 0.000 {built-in method posix.listdir} 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:777(exec_module) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:849(get_code) 11 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:90(_path_is_mode_type) 9 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:99(_path_isfile) 15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:80(_path_stat) 18 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:64(<listcomp>) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:549(module_from_spec) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:477(_init_module_attrs) 2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1265(_path_importer_cache) 15 0.000 0.000 0.000 0.000 {built-in method posix.stat} 2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:294(cache_from_source) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1252(_path_hooks) 2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:376(cached) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:424(_get_cached) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1520(path_hook_for_FileFinder) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:969(get_data) 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:147(__enter__) 38 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects} 1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1394(__init__) 2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:104(_path_isdir) Trevor Gamblin (1): python3-setuptools: patch entrypoints for performance boost .../python/python-setuptools.inc | 4 +- ...nt-usr-bin-wrappers-signoff-included.patch | 60 +++++++++++++++++++ 2 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch -- 2.24.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139868): https://lists.openembedded.org/g/openembedded-core/message/139868 Mute This Topic: https://lists.openembedded.org/mt/75072130/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
