On Sun, Jul 23, 2017 at 5:57 AM, Victor Stinner <victor.stin...@gmail.com> wrote: > We already did that. See _bootlocale for example. (Maybe also > _collecctions_abc?)
I was asking more in the context of recommended practices for third-party developers, as Nick mentioned earlier, because it's not a strategy I've ever seen mentioned (and common practice is to group only by functionality). It's good to know re: locale and collections though. Incidentally, from the issue thread it doesn't look like _bootlocale was motivated primarily by startup time, but _collections_abc was: locale: http://bugs.python.org/issue9548 collections.abc: http://bugs.python.org/issue19218 --Chris > > Victor > > Le 22 juil. 2017 07:20, "Chris Jerdonek" <chris.jerdo...@gmail.com> a écrit > : >> >> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <br...@python.org> wrote: >> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdo...@gmail.com> >> > wrote: >> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncogh...@gmail.com> >> >> wrote: >> >> > ... >> >> > * Lazy loading can have a significant impact on startup time, as it >> >> > means you don't have to pay for the cost of finding and loading >> >> > modules that you don't actually end up using on that particular run >> > >> > It should be mentioned that I have started designing an API to make >> > using >> > lazy loading much easier in Python 3.7 (i.e. "calling a single function" >> > easier), but I still have to write the tests and such before I propose a >> > patch and it will still be mainly for apps that know what they are doing >> > since lazy loading makes debugging import errors harder. >> > ... >> >> > However, if we're going to recommend them as good practices for 3rd >> >> > party developers looking to optimise the startup time of their Python >> >> > applications, then it makes sense for us to embrace them for the >> >> > standard library as well, rather than having our first reaction be to >> >> > write more hand-crafted C code. >> >> >> >> Are there any good write-ups of best practices and techniques in this >> >> area for applications (other than obvious things like avoiding >> >> unnecessary imports)? I'm thinking of things like how to structure >> >> your project, things to look for, developer tools that might help, and >> >> perhaps third-party runtime libraries? >> > >> > Nothing beyond "profile your application" and "don't do stuff during >> > import >> > as a side-effect" that I'm aware of. >> >> One "project structure" idea of the sort I had in mind is to move >> frequently used functions in a module into their own module. This way >> the most common paths of execution don't load unneeded functions. >> Following this line of reasoning could lead to grouping functions in >> an application by when they're needed instead of by what they do, >> which is different from what we normally see. I don't recall seeing >> advice like this anywhere, so maybe the trade-offs aren't worth it. >> Thoughts? >> >> --Chris >> >> >> > >> > -Brett >> > >> >> >> >> >> >> --Chris >> >> >> >> >> >> >> >> > >> >> > On that last point, it's also worth keeping in mind that we have a >> >> > much harder time finding new C-level contributors than we do new >> >> > Python-level ones, and have every reason to expect that problem to >> >> > get >> >> > worse over time rather than better (since writing and maintaining >> >> > handcrafted C code is likely to go the way of writing and maintaining >> >> > handcrafted assembly code as a skillset: while it will still be >> >> > genuinely necessary in some contexts, it will also be an increasingly >> >> > niche technical specialty). >> >> > >> >> > Starting to migrate to using Cython for our acceleration modules >> >> > instead of plain C should thus prove to be a win for everyone: >> >> > >> >> > - Cython structurally avoids a lot of typical bugs that arise in >> >> > hand-coded extensions (e.g. refcount bugs) >> >> > - by design, it's much easier to mentally switch between Python & >> >> > Cython than it is between Python & C >> >> > - Cython accelerated modules are easier to adapt to other interpeter >> >> > implementations than handcrafted C modules >> >> > - keeping Python modules and their C accelerated counterparts in sync >> >> > will be easier, as they'll mostly be using the same code >> >> > - we'd be able to start writing C API test cases in Cython rather >> >> > than >> >> > in handcrafted C (which currently mostly translates to only testing >> >> > them indirectly) >> >> > - CPython's own test suite would naturally help test Cython >> >> > compatibility with any C API updates >> >> > - we'd have an inherent incentive to help enhance Cython to take >> >> > advantage of new C API features >> >> > >> >> > The are some genuine downsides in increasing the complexity of >> >> > bootstrapping CPython when all you're starting with is a VCS clone >> >> > and >> >> > a C compiler, but those complications are ultimately no worse than >> >> > those we already have with Argument Clinic, and hence amenable to the >> >> > same solution: if we need to, we can check in the generated C files >> >> > in >> >> > order to make bootstrapping easier. >> >> > >> >> > Cheers, >> >> > Nick. >> >> > >> >> > -- >> >> > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> >> > _______________________________________________ >> >> > Python-Dev mailing list >> >> > Python-Dev@python.org >> >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> >> > >> >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com >> >> _______________________________________________ >> >> Python-Dev mailing list >> >> Python-Dev@python.org >> >> https://mail.python.org/mailman/listinfo/python-dev >> >> Unsubscribe: >> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> >> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <br...@python.org> wrote: >> > >> > >> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdo...@gmail.com> >> > wrote: >> >> >> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncogh...@gmail.com> >> >> wrote: >> >> > ... >> >> > * Lazy loading can have a significant impact on startup time, as it >> >> > means you don't have to pay for the cost of finding and loading >> >> > modules that you don't actually end up using on that particular run >> > >> > >> > It should be mentioned that I have started designing an API to make >> > using >> > lazy loading much easier in Python 3.7 (i.e. "calling a single function" >> > easier), but I still have to write the tests and such before I propose a >> > patch and it will still be mainly for apps that know what they are doing >> > since lazy loading makes debugging import errors harder. >> > >> >> >> >> > >> >> > We've historically resisted adopting these techniques for the >> >> > standard >> >> > library because they *do* make things more complicated *and* harder >> >> > to >> >> > debug relative to plain old eagerly imported dynamic Python code. >> >> > However, if we're going to recommend them as good practices for 3rd >> >> > party developers looking to optimise the startup time of their Python >> >> > applications, then it makes sense for us to embrace them for the >> >> > standard library as well, rather than having our first reaction be to >> >> > write more hand-crafted C code. >> >> >> >> Are there any good write-ups of best practices and techniques in this >> >> area for applications (other than obvious things like avoiding >> >> unnecessary imports)? I'm thinking of things like how to structure >> >> your project, things to look for, developer tools that might help, and >> >> perhaps third-party runtime libraries? >> > >> > >> > Nothing beyond "profile your application" and "don't do stuff during >> > import >> > as a side-effect" that I'm aware of. >> > >> > -Brett >> > >> >> >> >> >> >> --Chris >> >> >> >> >> >> >> >> > >> >> > On that last point, it's also worth keeping in mind that we have a >> >> > much harder time finding new C-level contributors than we do new >> >> > Python-level ones, and have every reason to expect that problem to >> >> > get >> >> > worse over time rather than better (since writing and maintaining >> >> > handcrafted C code is likely to go the way of writing and maintaining >> >> > handcrafted assembly code as a skillset: while it will still be >> >> > genuinely necessary in some contexts, it will also be an increasingly >> >> > niche technical specialty). >> >> > >> >> > Starting to migrate to using Cython for our acceleration modules >> >> > instead of plain C should thus prove to be a win for everyone: >> >> > >> >> > - Cython structurally avoids a lot of typical bugs that arise in >> >> > hand-coded extensions (e.g. refcount bugs) >> >> > - by design, it's much easier to mentally switch between Python & >> >> > Cython than it is between Python & C >> >> > - Cython accelerated modules are easier to adapt to other interpeter >> >> > implementations than handcrafted C modules >> >> > - keeping Python modules and their C accelerated counterparts in sync >> >> > will be easier, as they'll mostly be using the same code >> >> > - we'd be able to start writing C API test cases in Cython rather >> >> > than >> >> > in handcrafted C (which currently mostly translates to only testing >> >> > them indirectly) >> >> > - CPython's own test suite would naturally help test Cython >> >> > compatibility with any C API updates >> >> > - we'd have an inherent incentive to help enhance Cython to take >> >> > advantage of new C API features >> >> > >> >> > The are some genuine downsides in increasing the complexity of >> >> > bootstrapping CPython when all you're starting with is a VCS clone >> >> > and >> >> > a C compiler, but those complications are ultimately no worse than >> >> > those we already have with Argument Clinic, and hence amenable to the >> >> > same solution: if we need to, we can check in the generated C files >> >> > in >> >> > order to make bootstrapping easier. >> >> > >> >> > Cheers, >> >> > Nick. >> >> > >> >> > -- >> >> > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> >> > _______________________________________________ >> >> > Python-Dev mailing list >> >> > Python-Dev@python.org >> >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> >> > >> >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com >> >> _______________________________________________ >> >> Python-Dev mailing list >> >> Python-Dev@python.org >> >> https://mail.python.org/mailman/listinfo/python-dev >> >> Unsubscribe: >> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
