I have gone ahead and gathered some information using our standard development environment, devstack, I ran cProfile on our application, with the contents of it mocked out,
http://paste.openstack.org/show/40948/ When I try importing pkg_resources in our development environment it is very slow: vagrant@precise64:/opt/stack/nova$ time python -c "from pkg_resources import load_entry_point" real 0m0.185s user 0m0.136s sys 0m0.044s vagrant@precise64:/opt/stack/nova$ time python -c "print 'hi'" hi real 0m0.047s user 0m0.036s sys 0m0.008s I also ran cProfile on just the import line: http://paste.openstack.org/show/40949/ and $ python -vvvvvv -c "import pkg_resources" http://paste.openstack.org/show/40952/ As for python 3, we have to maintain python 2.6 and 2.7 compatibility so a Python 3 only fix isn't acceptable On Fri, Jul 19, 2013 at 5:58 AM, Daniel Holth <dho...@gmail.com> wrote: > On Fri, Jul 19, 2013 at 5:32 AM, Robert Collins > <robe...@robertcollins.net> wrote: > > On 19 July 2013 21:24, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote: > >> Robert Collins <robertc <at> robertcollins.net> writes: > >> > >>> So my question here would be - can we make it faster? We have just > >>> been diagnosing a performance problem in nova due to rootwrap being a > >>> pkg_resources scripts entry point : just getting to the first line of > >>> main() takes 200ms, and we make dozens of subprocess calls (has to be, > >>> we're escalating privileges) to the script in question : that time is > >>> nearly entirely doing introspection of metadata from disk. > >> > >> Is there more detailed information about where the time is being spent? > e.g. > >> os.stat(), file I/O, parsing of the actual metadata files, > load_entry_point() > >> etc. > > > > Not sure. Joe? > > > > -Rob > > You should at least time it against the simpler "import sys, x.main; > sys.exit(main())" style wrapper. > > As a pkg_resources optimization it might be worthwhile to try using > https://github.com/benhoyt/scandir/ or in Python 3, the undocumented > cache used by the importer system, to try to speed things up. > > It is a bit tricky to profile pkg_resources since it does a lot of > work at import. >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
