Hi, I'm having some trouble making the lazy_import "implicit", I tried following approaches:
1)overriding "__import__ " using 'builtins.__import__ = lazy_import.lazy_import_module' but this can't take more than 2 args whereas "__import__" takes upto 5 args, so now I have to make changes to "_bootstrap.py"(which I shouldn't probably make) and __init__.py to accomadate everything new. 2)Using "meta_path" and "path_hooks" to register custom "Loader" and "finder" but somehow it's breaking during 'make' ...probably due to 'finder'(unable to import stuff like '.org*')...Working on this. 3)The Whitelist option(that Antoine suggested) ,somewhat hardcode, i.e. to identify modules and in place of "import" add the lazy_import_module function. Any suggestion? Regards, Bhavishya On Sun, Jun 25, 2017 at 12:52 PM, Bhavishya <bhavishyagop...@gmail.com> wrote: > Also, the "fatoptimizer" project's compile fails due to missing > "ma_version_tag" (and works on changing it to "ma_version"). > > Regards, > Bhavishya > > On Sun, Jun 25, 2017 at 12:32 PM, Bhavishya <bhavishyagop...@gmail.com> > wrote: > >> Hello, >> I have added the "lazy_import" >> <https://github.com/bhavishyagopesh/gsoc_2017/blob/master/python_startup_time/lazy_loader.py> >> function but still working on adding it implicitly(To ensure that at >> startup it is actually used.) >> >> Thanks Antoine for the suggestion >> >> Regards, >> Bhavishya >> >> On Sat, Jun 24, 2017 at 9:30 PM, <speed-requ...@python.org> wrote: >> >>> Send Speed mailing list submissions to >>> speed@python.org >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> https://mail.python.org/mailman/listinfo/speed >>> or, via email, send a message with subject or body 'help' to >>> speed-requ...@python.org >>> >>> You can reach the person managing the list at >>> speed-ow...@python.org >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of Speed digest..." >>> >>> >>> Today's Topics: >>> >>> 1. Re: Lazy-loading to decrease python_startup time (Brett Cannon) >>> 2. Re: Lazy-loading to decrease python_startup time (Antoine Pitrou) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Fri, 23 Jun 2017 23:03:57 +0000 >>> From: Brett Cannon <br...@python.org> >>> To: Bhavishya <bhavishyagop...@gmail.com>, speed@python.org, Ramya >>> Meruva <meruvaramya...@gmail.com>, Victor Stinner >>> <victor.stin...@gmail.com> >>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time >>> Message-ID: >>> <CAP1=2W41_xTDt7gkuUWnmYA=+bz9ox-xt-pajtxkdgg4tdr...@mail.gm >>> ail.com> >>> Content-Type: text/plain; charset="utf-8" >>> >>> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagop...@gmail.com> >>> wrote: >>> >>> > As suggested, I'd like to discuss if lazy-loading is an option for >>> > improving python-startup time.And if could be done inside the scope of >>> a >>> > GSoc project. >>> > >>> >>> It's a possibility and it could be done in the scope of a GSoC project >>> easily. Basically what would be needed is an importlib.util.lazy_import() >>> function which does mostly what is outlined in >>> https://docs.python.org/3/library/importlib.html#approximati >>> ng-importlib-import-module >>> but >>> where the proper lazy loader is set on the spec object as an extra step. >>> Then every module that is used during startup would use >>> importlib.util.lazy_import() for importing instead of the normal import >>> statement. What this would do is help guarantee that all modules that are >>> identified as part of startup never import a module needlessly as the >>> lazy >>> loader would simply postpone the load until necessary. This would also >>> allow for pulling out local imports that are currently done in modules >>> that >>> are a part of startup and make them global so they are more visible. >>> >>> But I have no idea if this will actually speed things up. :) At worst it >>> would slow things down ever so slightly due to the extra overhead of lazy >>> loading for things that are known to be needed already during startup. At >>> best, though, is we accidentally discover modules that are being imported >>> needlessly at startup as well as not having to hide imports in functions >>> for performance. This fact that it may not be worth it is why I haven't >>> bothered to try it out yet. >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: <http://mail.python.org/pipermail/speed/attachments/20170623 >>> /ed7416e2/attachment-0001.html> >>> >>> ------------------------------ >>> >>> Message: 2 >>> Date: Sat, 24 Jun 2017 10:59:49 +0200 >>> From: Antoine Pitrou <solip...@pitrou.net> >>> To: speed@python.org >>> Subject: Re: [Speed] Lazy-loading to decrease python_startup time >>> Message-ID: <20170624105949.62590bbf@fsol> >>> Content-Type: text/plain; charset=US-ASCII >>> >>> On Fri, 23 Jun 2017 23:03:57 +0000 >>> Brett Cannon <br...@python.org> wrote: >>> > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagop...@gmail.com> >>> wrote: >>> > >>> > > As suggested, I'd like to discuss if lazy-loading is an option for >>> > > improving python-startup time.And if could be done inside the scope >>> of a >>> > > GSoc project. >>> > > >>> > >>> > It's a possibility and it could be done in the scope of a GSoC project >>> > easily. Basically what would be needed is an >>> importlib.util.lazy_import() >>> > function which does mostly what is outlined in >>> > https://docs.python.org/3/library/importlib.html#approximati >>> ng-importlib-import-module >>> > but >>> > where the proper lazy loader is set on the spec object as an extra >>> step. >>> > Then every module that is used during startup would use >>> > importlib.util.lazy_import() for importing instead of the normal import >>> > statement. >>> >>> My experience is that: >>> - you want lazy imports to be implicit, i.e. they should work using the >>> "import" statement and not any special syntax or function invocation >>> - you want a blacklist and/or whitelist mechanism to restrict lazy >>> imports to a particular set of modules and packages, because some >>> modules may not play well with lazy importing (e.g. anything that >>> registers plugins, specializations -- think singledispatch --, etc.) >>> >>> For example, I may want to register the "tornado", "asyncio" and "numpy" >>> namespaces / packages for lazy importing, but not the "numba" namespace >>> as it uses registration mechanisms quite heavily. >>> >>> (and the stdlib could be part of the default lazy import whitelist) >>> >>> Regards >>> >>> Antoine. >>> >>> >>> >>> >>> ------------------------------ >>> >>> Subject: Digest Footer >>> >>> _______________________________________________ >>> Speed mailing list >>> Speed@python.org >>> https://mail.python.org/mailman/listinfo/speed >>> >>> >>> ------------------------------ >>> >>> End of Speed Digest, Vol 36, Issue 1 >>> ************************************ >>> >> >> >
_______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
