On Mon, 26 Jun 2017 at 01:07 Bhavishya <bhavishyagop...@gmail.com> wrote:
> Hi, > I'm having some trouble making the lazy_import "implicit", I tried > following approaches: > > 1)overriding "__import__ " using 'builtins.__import__ = > lazy_import.lazy_import_module' but this can't take more than 2 args > whereas "__import__" takes upto 5 args, so now I have to make changes to > "_bootstrap.py"(which I shouldn't probably make) and __init__.py to > accomadate everything new. > Don't override __import__ basically ever. > > 2)Using "meta_path" and "path_hooks" to register custom "Loader" and > "finder" but somehow it's breaking during 'make' ...probably due to > 'finder'(unable to import stuff like '.org*')...Working on this. > I don't know what you mean by this, e.g. what is ".org*"? > > 3)The Whitelist option(that Antoine suggested) ,somewhat hardcode, i.e. to > identify modules and in place of "import" add the lazy_import_module > function. > I actually don't think that's what Antoine meant. I think he was suggesting setting up a finder and loader as normal that would do lazy loading but controlling it with a whitelist/blacklist to then potentially fail accordingly so the normal finders/loaders would do the right thing. > > Any suggestion? > As I said in my other email, just don't bother with this right now. To quickly see if things will be faster you should just edit the modules that are imported at startup directly to use a lazy_import() function and measure the performance. -Brett > > Regards, > Bhavishya > > On Sun, Jun 25, 2017 at 12:52 PM, Bhavishya <bhavishyagop...@gmail.com> > wrote: > >> Also, the "fatoptimizer" project's compile fails due to missing >> "ma_version_tag" (and works on changing it to "ma_version"). >> >> Regards, >> Bhavishya >> >> On Sun, Jun 25, 2017 at 12:32 PM, Bhavishya <bhavishyagop...@gmail.com> >> wrote: >> >>> Hello, >>> I have added the "lazy_import" >>> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py> >>> function but still working on adding it implicitly(To ensure that at >>> startup it is actually used.) >>> >>> Thanks Antoine for the suggestion >>> >>> Regards, >>> Bhavishya >>> >>> On Sat, Jun 24, 2017 at 9:30 PM, <speed-requ...@python.org> wrote: >>> >>>> Send Speed mailing list submissions to >>>> speed@python.org >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> https://mail.python.org/mailman/listinfo/speed >>>> or, via email, send a message with subject or body 'help' to >>>> speed-requ...@python.org >>>> >>>> You can reach the person managing the list at >>>> speed-ow...@python.org >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> than "Re: Contents of Speed digest..." >>>> >>>> >>>> Today's Topics: >>>> >>>> 1. Re: Lazy-loading to decrease python_startup time (Brett Cannon) >>>> 2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou) >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> Message: 1 >>>> Date: Fri, 23 Jun 2017 23:03:57 +0000 >>>> From: Brett Cannon <br...@python.org> >>>> To: Bhavishya <bhavishyagop...@gmail.com>, speed@python.org, Ramya >>>> Meruva <meruvaramya...@gmail.com>, Victor Stinner >>>> <victor.stin...@gmail.com> >>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time >>>> Message-ID: >>>> <CAP1=2W41_xTDt7gkuUWnmYA=+ >>>> bz9ox-xt-pajtxkdgg4tdr...@mail.gmail.com> >>>> Content-Type: text/plain; charset="utf-8" >>>> >>>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagop...@gmail.com> >>>> wrote: >>>> >>>> > As suggested, I'd like to discuss if lazy-loading is an option for >>>> > improving python-startup time.And if could be done inside the scope >>>> of a >>>> > GSoc project. >>>> > >>>> >>>> It's a possibility and it could be done in the scope of a GSoC project >>>> easily. Basically what would be needed is an >>>> importlib.util.lazy_import() >>>> function which does mostly what is outlined in >>>> >>>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module >>>> but >>>> where the proper lazy loader is set on the spec object as an extra step. >>>> Then every module that is used during startup would use >>>> importlib.util.lazy_import() for importing instead of the normal import >>>> statement. What this would do is help guarantee that all modules that >>>> are >>>> identified as part of startup never import a module needlessly as the >>>> lazy >>>> loader would simply postpone the load until necessary. This would also >>>> allow for pulling out local imports that are currently done in modules >>>> that >>>> are a part of startup and make them global so they are more visible. >>>> >>>> But I have no idea if this will actually speed things up. :) At worst it >>>> would slow things down ever so slightly due to the extra overhead of >>>> lazy >>>> loading for things that are known to be needed already during startup. >>>> At >>>> best, though, is we accidentally discover modules that are being >>>> imported >>>> needlessly at startup as well as not having to hide imports in functions >>>> for performance. This fact that it may not be worth it is why I haven't >>>> bothered to try it out yet. >>>> -------------- next part -------------- >>>> An HTML attachment was scrubbed... >>>> URL: < >>>> http://mail.python.org/pipermail/speed/attachments/20170623/ed7416e2/attachment-0001.html >>>> > >>>> >>>> ------------------------------ >>>> >>>> Message: 2 >>>> Date: Sat, 24 Jun 2017 10:59:49 +0200 >>>> From: Antoine Pitrou <solip...@pitrou.net> >>>> To: speed@python.org >>>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time >>>> Message-ID: <20170624105949.62590bbf@fsol> >>>> Content-Type: text/plain; charset=US-ASCII >>>> >>>> On Fri, 23 Jun 2017 23:03:57 +0000 >>>> Brett Cannon <br...@python.org> wrote: >>>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagop...@gmail.com> >>>> wrote: >>>> > >>>> > > As suggested, I'd like to discuss if lazy-loading is an option for >>>> > > improving python-startup time.And if could be done inside the scope >>>> of a >>>> > > GSoc project. >>>> > > >>>> > >>>> > It's a possibility and it could be done in the scope of a GSoC project >>>> > easily. Basically what would be needed is an >>>> importlib.util.lazy_import() >>>> > function which does mostly what is outlined in >>>> > >>>> https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module >>>> > but >>>> > where the proper lazy loader is set on the spec object as an extra >>>> step. >>>> > Then every module that is used during startup would use >>>> > importlib.util.lazy_import() for importing instead of the normal >>>> import >>>> > statement. >>>> >>>> My experience is that: >>>> - you want lazy imports to be implicit, i.e. they should work using the >>>> "import" statement and not any special syntax or function invocation >>>> - you want a blacklist and/or whitelist mechanism to restrict lazy >>>> imports to a particular set of modules and packages, because some >>>> modules may not play well with lazy importing (e.g. anything that >>>> registers plugins, specializations -- think singledispatch --, etc.) >>>> >>>> For example, I may want to register the "tornado", "asyncio" and "numpy" >>>> namespaces / packages for lazy importing, but not the "numba" namespace >>>> as it uses registration mechanisms quite heavily. >>>> >>>> (and the stdlib could be part of the default lazy import whitelist) >>>> >>>> Regards >>>> >>>> Antoine. >>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> >>>> Subject: Digest Footer >>>> >>>> _______________________________________________ >>>> Speed mailing list >>>> Speed@python.org >>>> https://mail.python.org/mailman/listinfo/speed >>>> >>>> >>>> ------------------------------ >>>> >>>> End of Speed Digest, Vol 36, Issue 1 >>>> ************************************ >>>> >>> >>> >> >
_______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
