Re: [Python-Dev] Investigating time for `import requests`

2017-10-20 Thread Brett Cannon
On Fri, 20 Oct 2017 at 06:42 Ronald Oussoren wrote: > Op 10 okt. 2017 om 01:48 heeft Brett Cannon het > volgende geschreven: > > > > On Mon, Oct 2, 2017, 17:49 Ronald Oussoren, > wrote: > >> Op 3 okt. 2017 om 04:29 heeft Barry

Re: [Python-Dev] Investigating time for `import requests`

2017-10-20 Thread Ronald Oussoren
Op 10 okt. 2017 om 01:48 heeft Brett Cannon > het volgende geschreven: > > > On Mon, Oct 2, 2017, 17:49 Ronald Oussoren, > wrote: > Op 3 okt. 2017 om 04:29 heeft Barry Warsaw

Re: [Python-Dev] Investigating time for `import requests`

2017-10-09 Thread Brett Cannon
On Mon, Oct 2, 2017, 17:49 Ronald Oussoren, wrote: > Op 3 okt. 2017 om 04:29 heeft Barry Warsaw het > volgende geschreven: > > > On Oct 2, 2017, at 14:56, Brett Cannon wrote: > > > >> So Mercurial specifically is an odd duck because

Re: [Python-Dev] Investigating time for `import requests`

2017-10-09 Thread Brett Cannon
On Mon, Oct 2, 2017, 13:56 Antoine Pitrou, wrote: > On Mon, 02 Oct 2017 18:56:15 + > Brett Cannon wrote: > > > > So Mercurial specifically is an odd duck because they already do lazy > > importing (in fact they are using the lazy loading support from >

Re: [Python-Dev] Investigating time for `import requests`

2017-10-09 Thread Brett Cannon
On Mon, Oct 2, 2017, 12:30 Barry Warsaw, wrote: > On Oct 2, 2017, at 14:56, Brett Cannon wrote: > > > So Mercurial specifically is an odd duck because they already do lazy > importing (in fact they are using the lazy loading support from importlib). > In

Re: [Python-Dev] Investigating time for `import requests`

2017-10-08 Thread Eric V. Smith
The easiest workaround at the moment is still pretty clumsy: def import_SLLError():     from requests.exceptions import SLLError     return SLLError ...     except import_SLLError(): But what happens if that gives you an ImportError? You can't catch a requests exception unless requests

Re: [Python-Dev] Investigating time for `import requests`

2017-10-08 Thread Koos Zevenhoven
On Sun, Oct 8, 2017 at 2:44 PM, Chris Angelico wrote: > On Sun, Oct 8, 2017 at 7:02 PM, David Cournapeau > wrote: > > It is certainly true that for a CLI tool that actually makes any network > > I/O, especially SSL, import times will quickly be negligible.

Re: [Python-Dev] Investigating time for `import requests`

2017-10-08 Thread Koos Zevenhoven
On Sun, Oct 8, 2017 at 11:02 AM, David Cournapeau wrote: > > On Mon, Oct 2, 2017 at 6:42 PM, Raymond Hettinger < > raymond.hettin...@gmail.com> wrote: > >> >> > On Oct 2, 2017, at 12:39 AM, Nick Coghlan wrote: >> > >> > "What requests uses" can identify

Re: [Python-Dev] Investigating time for `import requests`

2017-10-08 Thread Chris Angelico
On Sun, Oct 8, 2017 at 7:02 PM, David Cournapeau wrote: > It is certainly true that for a CLI tool that actually makes any network > I/O, especially SSL, import times will quickly be negligible. It becomes > tricky for complex tools, because of error management. For example, a

Re: [Python-Dev] Investigating time for `import requests`

2017-10-08 Thread David Cournapeau
On Mon, Oct 2, 2017 at 6:42 PM, Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > > > On Oct 2, 2017, at 12:39 AM, Nick Coghlan wrote: > > > > "What requests uses" can identify a useful set of > > avoidable imports. A Flask "Hello world" app could likely provide > >

Re: [Python-Dev] Investigating time for `import requests`

2017-10-03 Thread Stéfane Fermigier
Hi, On Mon, Oct 2, 2017 at 11:42 AM, Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > > > I don't expect to find anything that would help users of Django, Flask, > and Bottle since those are typically long-running apps where we value > response time more than startup time. > Actually,

Re: [Python-Dev] Investigating time for `import requests`

2017-10-03 Thread Koos Zevenhoven
I've probably missed a lot of this discussion, but this lazy import discussion confuses me. We already have both eager import (import at the top of the file), and lazy import (import right before use). The former is good when you know you need the module, and the latter is good when you having

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Ronald Oussoren
Op 3 okt. 2017 om 04:29 heeft Barry Warsaw het volgende geschreven: > On Oct 2, 2017, at 14:56, Brett Cannon wrote: > >> So Mercurial specifically is an odd duck because they already do lazy >> importing (in fact they are using the lazy loading support

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Antoine Pitrou
On Mon, 2 Oct 2017 11:15:35 -0400 Barry Warsaw wrote: > > I think there are opportunities for an explicit API for lazy compilation of > regular expressions, but I’m skeptical of the adoption curve making it > worthwhile. But maybe I’m wrong! We already have two caching

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Antoine Pitrou
On Mon, 02 Oct 2017 18:56:15 + Brett Cannon wrote: > > So Mercurial specifically is an odd duck because they already do lazy > importing (in fact they are using the lazy loading support from importlib). Do they? I was under the impression they had their own home-baked,

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Barry Warsaw
On Oct 2, 2017, at 14:56, Brett Cannon wrote: > So Mercurial specifically is an odd duck because they already do lazy > importing (in fact they are using the lazy loading support from importlib). > In terms of all of this discussion of tweaking import to be lazy, I think the

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Brett Cannon
On Mon, 2 Oct 2017 at 02:43 Raymond Hettinger wrote: > > > On Oct 2, 2017, at 12:39 AM, Nick Coghlan wrote: > > > > "What requests uses" can identify a useful set of > > avoidable imports. A Flask "Hello world" app could likely provide > >

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Terry Reedy
On 10/2/2017 4:57 AM, Paul Moore wrote: In practice, I don't think the fact that re.search() et al cache the compiled expressions is that well known (it's mentioned in the re.compile docs, but not in the re.search docs) We could add redundant mentions in the functions ;-). and so people

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Barry Warsaw
On Oct 1, 2017, at 22:34, Nathaniel Smith wrote: > > In principle re.compile() itself could be made lazy -- return a > regular exception object that just holds the string, and then compiles > and caches it the first time it's used. Might be tricky to do in a > backwards

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Raymond Hettinger
> On Oct 2, 2017, at 12:39 AM, Nick Coghlan wrote: > > "What requests uses" can identify a useful set of > avoidable imports. A Flask "Hello world" app could likely provide > another such sample, as could some example data analysis notebooks). Right. It is probably

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Christian Heimes
On 2017-10-02 04:04, INADA Naoki wrote: > *3. ssl* > > import time:      2007 |       2007 |                     ipaddress > import time:      2386 |       2386 |                     textwrap > import time:      2723 |       2723 |                     _ssl > ... > import time:       306 |       

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Paul Moore
On 2 October 2017 at 06:13, Raymond Hettinger wrote: > >> On Oct 1, 2017, at 7:34 PM, Nathaniel Smith wrote: >> >> In principle re.compile() itself could be made lazy -- return a >> regular exception object that just holds the string, and then

Re: [Python-Dev] Investigating time for `import requests`

2017-10-02 Thread Nick Coghlan
On 2 October 2017 at 15:13, Raymond Hettinger wrote: > >> On Oct 1, 2017, at 7:34 PM, Nathaniel Smith wrote: >> >> In principle re.compile() itself could be made lazy -- return a >> regular exception object that just holds the string, and then

Re: [Python-Dev] Investigating time for `import requests`

2017-10-01 Thread Raymond Hettinger
> On Oct 1, 2017, at 7:34 PM, Nathaniel Smith wrote: > > In principle re.compile() itself could be made lazy -- return a > regular exception object that just holds the string, and then compiles > and caches it the first time it's used. Might be tricky to do in a > backwards

Re: [Python-Dev] Investigating time for `import requests`

2017-10-01 Thread Glenn Linderman
On 10/1/2017 7:34 PM, Nathaniel Smith wrote: Another major slowness comes from compiling regular expression. I think we can increase cache size of `re.compile` and use ondemand cached compiling (e.g. `re.match()`), instead of "compile at import time" in many modules. In principle re.compile()

Re: [Python-Dev] Investigating time for `import requests`

2017-10-01 Thread Nathaniel Smith
On Sun, Oct 1, 2017 at 7:04 PM, INADA Naoki wrote: > 4. http.client > > import time: 1376 | 2448 | email.header > ... > import time: 1469 | 7791 | email.utils > import time: 408 | 10646 |

[Python-Dev] Investigating time for `import requests`

2017-10-01 Thread INADA Naoki
See also https://github.com/requests/requests/issues/4315 I tried new `-X importtime` option to `import requests`. Full output is here: https://gist.github.com/methane/96d58a29e57e5be97769897462ee1c7e Currently, it took about 110ms. And major parts are from Python stdlib. Followings are root of