On Tue, Jun 23, 2020 at 4:20 PM Trevor Gamblin <[email protected]> wrote:
> v2 adds the missing Upstream-Status notice in the patch file. > This belongs below the triple-dash line, as it doesn’t make sense in permanent git history out of context > setuptools' pkg_resources module has major performance issues with how > it loads entry points (e.g. the console_script entry point, which sets > up a module as a command-line executable), leading even the simplest > "hello world" scripts to take on the order of 150ms to run if > pkg_resources is incorporated. This is prohibitive for code that needs > to run quickly, and so we patch setuptools to reduce this time. As of > Python 3.7, importlib.resources is available and intended to replace > much of the functionality that causes this sluggishness, but since > many projects still utilize the legacy setuptools modules, a patch is > still required. Note that python3-fastentrypoints (which is available > in the meta-virtualization layer) is also intended to help alleviate > the problem, but since it must be added to existing projects it has > the same disadvantage as resorting to importlib.resources, requiring > manual additions to existing code to see the performance gains. > > The intent here is to patch easy_install to load module entry points > directly with the installed setuptools, rather than importing > pkg_resources and having it search out the entry points itself. This > leads to a drastic performance improvement - the changes in this patch > have been shown to result in load time ~6-8x lower, depending on the > complexity of the code it is tested with. A simple "hello world" > example on core-image-full-cmdline gave these results with and without > the patch: > > core-image-full-cmdline, without setuptools ScriptWriter patch: > > root@qemux86-64:~# time /usr/bin/minimal > hello world > > real 0m0.198s > user 0m0.174s > sys 0m0.023s > > core-image-full-cmdline, with setuptools ScriptWriter patch: > > root@qemux86-64:~# time /usr/bin/minimal > hello world > > real 0m0.034s > user 0m0.024s > sys 0m0.010s > > More details on the pkg_resources issue are available at: > https://github.com/pypa/setuptools/issues/510 > > Signed-off-by: Trevor Gamblin <[email protected]> > --- Put v2 changes here. It allows for review, but keeps it out of permanent git history > > .../python/python-setuptools.inc | 4 +- > ...nt-usr-bin-wrappers-signoff-included.patch | 62 +++++++++++++++++++ > 2 files changed, 65 insertions(+), 1 deletion(-) > create mode 100644 > meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch > > diff --git a/meta/recipes-devtools/python/python-setuptools.inc > b/meta/recipes-devtools/python/python-setuptools.inc > index 9322ca9a05..3222de7079 100644 > --- a/meta/recipes-devtools/python/python-setuptools.inc > +++ b/meta/recipes-devtools/python/python-setuptools.inc > @@ -10,7 +10,9 @@ inherit pypi > > SRC_URI_append_class-native = " > file://0001-conditionally-do-not-fetch-code-by-easy_install.patch" > > -SRC_URI += "file://0001-change-shebang-to-python3.patch" > +SRC_URI += "file://0001-change-shebang-to-python3.patch \ > + > file://0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch > \ > + " > > SRC_URI[md5sum] = "6e9de90b242fdd60ef59f497424ce13a" > SRC_URI[sha256sum] = > "145fa62b9d7bb544fce16e9b5a9bf4ab2032d2f758b7cd674af09a92736aff74" > diff --git > a/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch > b/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch > new file mode 100644 > index 0000000000..7e931c845c > --- /dev/null > +++ > b/meta/recipes-devtools/python/python3-setuptools/0001-ScriptWriter-create-more-efficient-usr-bin-wrappers-signoff-included.patch > @@ -0,0 +1,62 @@ > +From aae8cd3de3f289cea3db01212579913c925191e8 Mon Sep 17 00:00:00 2001 > +From: Lauri Tirkkonen <[email protected]> > +Date: Thu, 26 Mar 2020 14:24:25 +0000 > +Subject: [PATCH] ScriptWriter: create more efficient /usr/bin wrappers > + > +Upstream setuptools writes scripts to /usr/bin that do insanely much > +stuff at runtime. https://github.com/pypa/setuptools/issues/510 > + > +Since the script entry points are already known at build time, we can > +just write those directly into the /usr/bin wrapper, avoiding the > +expensive 'pkg_resources' import at runtime. The idea is from > +https://github.com/ninjaaron/fast-entry_points but patched directly into > +the native build of setuptools here, so that all Python modules under > +bitbake automatically use it without needing additional build time > +dependencies. > + > +Upstream-Status: Pending > + > +Signed-off-by: Lauri Tirkkonen <[email protected]> > +Signed-off-by: Trevor Gamblin <[email protected]> > +--- > + setuptools/command/easy_install.py | 14 ++++++-------- > + 1 file changed, 6 insertions(+), 8 deletions(-) > + > +diff --git a/setuptools/command/easy_install.py > b/setuptools/command/easy_install.py > +index 8fba7b41..03a72714 100755 > +--- a/setuptools/command/easy_install.py > ++++ b/setuptools/command/easy_install.py > +@@ -2023,17 +2023,12 @@ class ScriptWriter(object): > + """ > + > + template = textwrap.dedent(r""" > +- # EASY-INSTALL-ENTRY-SCRIPT: %(spec)r,%(group)r,%(name)r > +- __requires__ = %(spec)r > +- import re > + import sys > +- from pkg_resources import load_entry_point > ++ > ++ from %(module)s import %(ep0)s > + > + if __name__ == '__main__': > +- sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', > sys.argv[0]) > +- sys.exit( > +- load_entry_point(%(spec)r, %(group)r, %(name)r)() > +- ) > ++ sys.exit(%(entrypoint)s()) > + """).lstrip() > + > + command_spec_class = CommandSpec > +@@ -2068,6 +2063,9 @@ class ScriptWriter(object): > + for type_ in 'console', 'gui': > + group = type_ + '_scripts' > + for name, ep in dist.get_entry_map(group).items(): > ++ module = ep.module_name > ++ ep0 = ep.attrs[0] > ++ entrypoint = '.'.join(ep.attrs) > + cls._ensure_safe_name(name) > + script_text = cls.template % locals() > + args = cls._get_script_args(type_, name, header, > script_text) > +-- > +2.24.1 > + > -- > 2.26.2 > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#139867): https://lists.openembedded.org/g/openembedded-core/message/139867 Mute This Topic: https://lists.openembedded.org/mt/75071815/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
