[issue14067] Avoid more stat() calls in importlib

[Joannah Nanjekye]

Joannah Nanjekye  added the comment:

Related : https://bugs.python.org/issue14067

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Joannah Nanjekye]

Joannah Nanjekye  added the comment:

I wonder if the circumstances changed since this is an old issue? the method 
looks to have a deprecation warning and changed a bit too, I stand to be 
corrected.

--
nosy: +nanjekyejoannah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Brett Cannon]

Brett Cannon br...@python.org added the comment:

Well, we already have a delay in detecting new files in importlib as the 
directory mtime check already adds a delay of the granularity of the mtime 
value for a directory. So if you manage to mutate a directory, import, write a 
file, and then import again all within the mtime of a directory, you will miss 
the new file (which has already triggered sporadic failures in various tests 
because they were not blowing out importlib's cache). IOW there is already a 
freshness problem that if we don't have importlib will never get bootstrapped 
because people will complain about performance too much.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Jim Jewett]

Jim Jewett jimjjew...@gmail.com added the comment:

As long as the interpreter knows about about files that *it* wrote, no repeat 
checks during startup seems utterly reasonable; sneaking in a new or changed 
file is inherently a race condition.

I think it would also be reasonable for general use, so long as there was also 
a way to say for this particular directory, always check.

--
nosy: +Jim.Jewett

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 While I don't think the delay in noticing filesystem changes is
 reasonable as the default behaviour, it might be interesting to see how 
 tricky it would be to create a custom importer (for either meta_path or 
 path_hooks) that did this.

Mmmh. Then I think it would be simpler to expose a global option to disable 
freshness checks. That said, it has to be activated as early as possible to be 
effective, which doesn't make it very practical as an opt-in setting.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Antoine Pitrou]

New submission from Antoine Pitrou pit...@free.fr:

This is an experimental patch that limits the frequency of stat() calls in 
_FileFinder.find_module(). It speeds up finding modules by 2x here, but 
unfortunately breaks some tests (which expect modules to appear immediately 
when created).

--
components: Library (Lib)
files: imptime.patch
keywords: patch
messages: 153809
nosy: brett.cannon, ncoghlan, pitrou
priority: low
severity: normal
status: open
title: Avoid more stat() calls in importlib
type: performance
versions: Python 3.3
Added file: http://bugs.python.org/file24583/imptime.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


Added file: http://bugs.python.org/file24584/imptime.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


Removed file: http://bugs.python.org/file24583/imptime.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Brett Cannon]

Brett Cannon br...@python.org added the comment:

The patch uses the time module which is not a built-in, so it breaks 
bootstrapping by directly importing a module, and an extension at that. At best 
you could switch to a modified _FileFinder after importlib is initially running 
and able to import extension modules.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 The patch uses the time module which is not a built-in, so it breaks
 bootstrapping by directly importing a module, and an extension at
 that. At best you could switch to a modified _FileFinder after
 importlib is initially running and able to import extension modules.

Yup, as I said it's more of a proof-of-concept.
(I was actually surprised that time is not a built-in module)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Brett Cannon]

Brett Cannon br...@python.org added the comment:

On Mon, Feb 20, 2012 at 18:06, Antoine Pitrou rep...@bugs.python.orgwrote:


 Antoine Pitrou pit...@free.fr added the comment:

  The patch uses the time module which is not a built-in, so it breaks
  bootstrapping by directly importing a module, and an extension at
  that. At best you could switch to a modified _FileFinder after
  importlib is initially running and able to import extension modules.

 Yup, as I said it's more of a proof-of-concept.
 (I was actually surprised that time is not a built-in module)

Yeah, I had to double-check myself.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14067] Avoid more stat() calls in importlib

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

FWIW, I doubt you'd get many objections if you ended up wanting to make time a 
builtin module and inject it into the bootstrapping namespace.

While I don't think the delay in noticing filesystem changes is reasonable as 
the default behaviour, it might be interesting to see how tricky it would be to 
create a custom importer (for either meta_path or path_hooks) that did this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14067
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com