Re: [Python-Dev] More optimisation ideas

On Fri, Feb 5, 2016, at 10:33 AM, Emile van Sebille wrote: > On 2/5/2016 9:37 AM, Alexander Walters wrote: > > > > On 2/5/2016 12:27, Emile van Sebille wrote: > >> On 2/1/2016 9:20 AM, Ethan Furman wrote: > >>> On 02/01/2016 08:40 AM, R. David Murray wrote: > >> > On the other hand, if the

Re: [Python-Dev] More optimisation ideas

On 2/5/2016 12:27, Emile van Sebille wrote: On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of

Re: [Python-Dev] More optimisation ideas

On Fri, 5 Feb 2016 at 10:34 Emile van Sebille wrote: > On 2/5/2016 9:37 AM, Alexander Walters wrote: > > > > > > On 2/5/2016 12:27, Emile van Sebille wrote: > >> On 2/1/2016 9:20 AM, Ethan Furman wrote: > >>> On 02/01/2016 08:40 AM, R. David Murray wrote: > >> > On the

Re: [Python-Dev] More optimisation ideas

On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the

Re: [Python-Dev] More optimisation ideas

On 2/5/2016 9:37 AM, Alexander Walters wrote: On 2/5/2016 12:27, Emile van Sebille wrote: On 2/1/2016 9:20 AM, Ethan Furman wrote: On 02/01/2016 08:40 AM, R. David Murray wrote: On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system

Re: [Python-Dev] More optimisation ideas

On Friday, February 5, 2016 11:57 AM, Emile van Sebille wrote: > Aah, 'must' is less restrictive in this context than I expected. When > you combine the two halves the first part might be more accurately > phrased as 'The program must make source code available' rather than

Re: [Python-Dev] More optimisation ideas

On 2/5/2016 10:38 AM, Brett Cannon wrote: On Fri, 5 Feb 2016 at 10:34 Emile van Sebille > wrote: >> Except for that nasty licensing issue requiring source code. >> >> Emile > Licensing requires, in the GPL at least, that the

Re: [Python-Dev] More optimisation ideas

On 5 February 2016 at 15:05, Steven D'Aprano wrote: > (I'm not even sure if this suggestion makes sense, since I'm not really > sure what "freezing" the stdlib entails. Is it documented anywhere?) It's not particularly well documented - most of the docs you'll find are about

Re: [Python-Dev] More optimisation ideas

On 2/4/2016 12:18 PM, Sven R. Kunze wrote: On 04.02.2016 14:09, Nick Coghlan wrote: On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev wrote: On Feb 1, 2016, at 09:59,mike.romb...@comcast.net wrote: If the stdlib were to use implicit namespace packages

Re: [Python-Dev] More optimisation ideas

On 2 February 2016 at 02:40, R. David Murray wrote: > On the other hand, if the distros go the way Nick has (I think) been > advocating, and have a separate 'system python for system scripts' that > is independent of the one installed for user use, having the system-only >

Re: [Python-Dev] More optimisation ideas

On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev wrote: > On Feb 1, 2016, at 09:59, mike.romb...@comcast.net wrote: >> >> If the stdlib were to use implicit namespace packages >> ( https://www.python.org/dev/peps/pep-0420/ ) and the various >> loaders/importers

Re: [Python-Dev] More optimisation ideas

On Thu, Feb 04, 2016 at 07:58:30PM -0500, Terry Reedy wrote: > >>For folks that *do* know how to use the terminal: > >> > >>$ python3 -m inspect --details inspect > >>Target: inspect > >>Origin: /usr/lib64/python3.4/inspect.py > >>Cached: /usr/lib64/python3.4/__pycache__/inspect.cpython-34.pyc >

Re: [Python-Dev] More optimisation ideas

On 04.02.2016 14:09, Nick Coghlan wrote: On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev wrote: On Feb 1, 2016, at 09:59, mike.romb...@comcast.net wrote: If the stdlib were to use implicit namespace packages ( https://www.python.org/dev/peps/pep-0420/ ) and

Re: [Python-Dev] More optimisation ideas

On 02/01/2016 08:40 AM, R. David Murray wrote: On Mon, 01 Feb 2016 14:12:27 +1100, Steven D'Aprano wrote: I find that being able to easily open stdlib .py files in a text editor to read the source is extremely valuable. I've learned much more from reading the source than from (e.g.)

Re: [Python-Dev] More optimisation ideas

On Mon, 01 Feb 2016 14:12:27 +1100, Steven D'Aprano wrote: > On Sun, Jan 31, 2016 at 08:23:00PM +, Brett Cannon wrote: > > So freezing the stdlib helps on UNIX and not on OS X (if my old testing is > > still accurate). I guess the next question is what it does on Windows

Re: [Python-Dev] More optimisation ideas

Thanks, Brett. Wasn't aware of lazy imports as well. I think that one is even better reducing startup time as freezing stdlib. On 31.01.2016 18:57, Brett Cannon wrote: I have opened http://bugs.python.org/issue26252 to track writing the example (and before ppl go playing with the lazy loader,

Re: [Python-Dev] More optimisation ideas

On Feb 1, 2016, at 09:59, mike.romb...@comcast.net wrote: > > If the stdlib were to use implicit namespace packages > ( https://www.python.org/dev/peps/pep-0420/ ) and the various > loaders/importers as well, then python could do what I've done with an > embedded python application for years.

Re: [Python-Dev] More optimisation ideas

> " " == Barry Warsaw writes: >> On Feb 01, 2016, at 11:40 AM, R. David Murray wrote: >> I don't know about anyone else, but on my own development >> systems it is not that unusual for me to *edit* the stdlib >> files (to add debug prints) while

Re: [Python-Dev] More optimisation ideas

On Feb 01, 2016, at 11:40 AM, R. David Murray wrote: >Well, Brett said it would be optional, though perhaps the above >paragraph is asking about doing it in our Windows build. But the linux >distros might make also use the option if it exists, so the question is >very meaningful. However, you'd

Re: [Python-Dev] More optimisation ideas

On Feb 01 2016, mike.romb...@comcast.net wrote: " " == Barry Warsaw writes: >> On Feb 01, 2016, at 11:40 AM, R. David Murray wrote: >> I don't know about anyone else, but on my own development >> systems it is not that unusual for me to *edit* the >>

Re: [Python-Dev] More optimisation ideas

On Feb 1, 2016, at 19:44, Terry Reedy wrote: > >> On 2/1/2016 3:39 PM, Andrew Barnert via Python-Dev wrote: >> >> There are already multiple duplicate questions every month on >> StackOverflow from people asking "how do I find the source to stdlib >> module X". The canonical

Re: [Python-Dev] More optimisation ideas

On 2/1/2016 3:39 PM, Andrew Barnert via Python-Dev wrote: There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__,

Re: [Python-Dev] More optimisation ideas

On Mon, 1 Feb 2016 at 08:48 R. David Murray wrote: > On Mon, 01 Feb 2016 14:12:27 +1100, Steven D'Aprano > wrote: > > On Sun, Jan 31, 2016 at 08:23:00PM +, Brett Cannon wrote: > > > So freezing the stdlib helps on UNIX and not on OS X (if my old >

Re: [Python-Dev] More optimisation ideas

611 >http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > > > Top-posted from my Windows Phone > > > > -Original Message- > > From: "Serhiy Storchaka" <storch...@gmail.com>

Re: [Python-Dev] More optimisation ideas

On 1/31/2016 12:09 PM, Antoine Pitrou wrote: The following documentation leaves me absolutely clueless: """This class only works with loaders that define exec_module() as control over what module type is used for the module is required. No wonder. I cannot parse it as an English sentence.

Re: [Python-Dev] More optimisation ideas

I have opened http://bugs.python.org/issue26252 to track writing the example (and before ppl go playing with the lazy loader, be aware of http://bugs.python.org/issue26186). On Sun, 31 Jan 2016 at 09:26 Brett Cannon wrote: > There are no example docs for it yet, but enough

Re: [Python-Dev] More optimisation ideas

> On Jan 31, 2016, at 12:02 PM, Brett Cannon wrote: > > A lazy importer was added in Python 3.5 Is there any docs on how to actually use the LazyLoader in 3.5? I can’t seem to find any but I don’t really know the import system that well. - Donald Stufft

Re: [Python-Dev] More optimisation ideas

Brett Cannon python.org> writes: > > > A lazy importer was added in Python 3.5 and it was not possible > without the module spec refactoring. Wow... Thank you, I didn't know about that. Now for the next question: how am I supposed to use it? The following documentation leaves me absolutely

Re: [Python-Dev] More optimisation ideas

There are no example docs for it yet, but enough people have asked this week about how to set up a custom importer that I will write up a generic example case which will make sense for a lazy loader (need to file the issue before I forget). On Sun, 31 Jan 2016, 09:11 Donald Stufft

Re: [Python-Dev] More optimisation ideas

http://www.egenix.com/company/contact/ http://www.malemburg.com/ > Top-posted from my Windows Phone > > -Original Message- > From: "Serhiy Storchaka" <storch...@gmail.com> > Sent: ‎1/‎30/‎2016 10:22 > To: "python-dev@python.org"

Re: [Python-Dev] More optimisation ideas

Hi, If you want to make startup time faster for a broad range of applications, please consider adding a lazy import facility in the stdlib. I recently tried to write a lazy import mechanism using import hooks (to make it portable from 2.6 to 3.5), it seems nearly impossible to do so (or, at

Re: [Python-Dev] More optimisation ideas

A lazy importer was added in Python 3.5 and it was not possible without the module spec refactoring. On Sun, 31 Jan 2016, 08:57 Antoine Pitrou wrote: > > Hi, > > If you want to make startup time faster for a broad range of applications, > please consider adding a lazy import

Re: [Python-Dev] More optimisation ideas

On Sun, 31 Jan 2016, 15:36 Terry Reedy wrote: > On 1/31/2016 12:09 PM, Antoine Pitrou wrote: > > > The following documentation leaves me absolutely clueless: > > > > """This class only works with loaders that define exec_module() as > control > > over what module type is used

Re: [Python-Dev] More optimisation ideas

On Sun, Jan 31, 2016 at 08:23:00PM +, Brett Cannon wrote: > So freezing the stdlib helps on UNIX and not on OS X (if my old testing is > still accurate). I guess the next question is what it does on Windows and > if we would want to ever consider freezing the stdlib as part of the build >

Re: [Python-Dev] More optimisation ideas

On Sat, Jan 30, 2016, 12:30 Sven R. Kunze wrote: > On 30.01.2016 19:20, Serhiy Storchaka wrote: > > AFAIK the most time is spent in system calls like stat or open. > > Archiving the stdlib into the ZIP file and using zipimport can > > decrease Python startup time (perhaps there

Re: [Python-Dev] More optimisation ideas

-dev about twelve months ago, IIRC. Maybe a little longer. Top-posted from my Windows Phone -Original Message- From: "Serhiy Storchaka" <storch...@gmail.com> Sent: ‎1/‎30/‎2016 10:22 To: "python-dev@python.org" <python-dev@python.org> Subject: Re: [Pyt

Re: [Python-Dev] More optimisation ideas

On 30Jan2016 0645, Serhiy Storchaka wrote: $ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 10 loops, best of 3: 4.36 usec per loop Getting rid from charmap_build() would save you at most 4.4 microseconds per

Re: [Python-Dev] More optimisation ideas

On 30.01.16 18:31, Steve Dower wrote: On 30Jan2016 0645, Serhiy Storchaka wrote: $ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 10 loops, best of 3: 4.36 usec per loop Getting rid from charmap_build() would save

Re: [Python-Dev] More optimisation ideas

On Sat, 30 Jan 2016 at 10:21 Serhiy Storchaka wrote: > On 30.01.16 18:31, Steve Dower wrote: > > On 30Jan2016 0645, Serhiy Storchaka wrote: > >> $ ./python -m timeit -s "import codecs; from encodings.cp437 import > >> decoding_table" -- "codecs.charmap_build(decoding_table)"

Re: [Python-Dev] More optimisation ideas

On 30 January 2016 at 03:48, Steve Dower wrote: > > It doesn't currently end up on disk. Some tables are partially or completely > stored on disk as Python source code (some are partially generated from > simple rules), but others are generated by inverting those. That

Re: [Python-Dev] More optimisation ideas

On 29.01.16 19:05, Steve Dower wrote: This is probably the code snippet that bothered me the most: ### Encoding table encoding_table=codecs.charmap_build(decoding_table) It shows up in many of the encodings modules, and while it is not a bad function in itself, we are obviously

Re: [Python-Dev] More optimisation ideas

On 30.01.2016 19:20, Serhiy Storchaka wrote: AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this). Oh, please don't. One thing I love about Python

Re: [Python-Dev] More optimisation ideas

On 30.01.2016 21:32, Brett Cannon wrote: On Sat, Jan 30, 2016, 12:30 Sven R. Kunze > wrote: On 30.01.2016 19:20, Serhiy Storchaka wrote: > AFAIK the most time is spent in system calls like stat or open. > Archiving the stdlib into the ZIP

Re: [Python-Dev] More optimisation ideas

Hi, > > Storing these in static data is a tradeoff between > disk space and startup performance, and one I think it likely to be > worthwhile. it's really an important trade off? As far a I understand from your email those modules are always being loaded and the final data created. won't the

Re: [Python-Dev] More optimisation ideas

hon.org> Subject: Re: [Python-Dev] More optimisation ideas Hi, > > Storing these in static data is a tradeoff between > disk space and startup performance, and one I think it likely to be > worthwhile. it's really an important trade off? As far a I understand from your email those m