Patches item #921466, was opened at 2004-03-22 16:10 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921466&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Martin v. Löwis (loewis) Assigned to: Nobody/Anonymous (nobody) Summary: Reduce number of open calls on startup Initial Comment: This patch uses sys.path_importer_cache to reduce the number of open calls, in the following way: - if the value in path_importer_cache is None, it stats the path to find out whether the file exists - it then puts True/False into path_importer_cache - if the value in path_importer_cache is False, the path entry is skipped on all imports - if the value is True, the stat call is skipped, and open calls for files in the directory are made. On Linux, this reduces the number of open calls for an empty script from 343 to 263. The startup-time (for 100 interpreter invocations) goes down by one percent (from 0.0819s to 0.08113s per invocation). ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2006-05-25 23:38 Message: Logged In: YES user_id=33168 Without looking at the patch impl, I'm +1 on the idea of reducing stat/open calls. On NFS this is a huge time sync. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-05-25 11:47 Message: Logged In: YES user_id=849994 I reviewed this patch, in in consequence discovered a problem with the sys.path_hooks machinery, described in http://mail.python.org/pipermail/python-dev/2006-May/065173.html This patch fixes the problem and corrects the original patch to not set any sys.path_importer_cache entry to True or False when no import hooks are enabled (the p_loader argument to find_module is NULL then). ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-02-25 11:55 Message: Logged In: YES user_id=849994 I'm very much for it. I haven't got too much RAM, and whenever I start a Python program (emerge being the most prominent example) after having worked heavily with e.g. graphics or VMware, I'm hit by the files Python's opening not being in the file cache anymore. ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-02-20 13:51 Message: Logged In: YES user_id=21627 Not sure. Anybody speaking in favour? against? ---------------------------------------------------------------------- Comment By: Georg Brandl (birkenfeld) Date: 2006-02-20 02:42 Message: Logged In: YES user_id=1188172 Can this go into 2.5? ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2004-03-23 07:43 Message: Logged In: YES user_id=21627 It's certainly the case that the system has cached all files needed for startup in memory, including the directory contents of all directories searched. OTOH, I assume that is the scenario in which people worry about startup time: high-frequency invocations of python. For a single invocation, it shouldn't matter much whether it takes 0.04s or 0.08s. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-22 23:30 Message: Logged In: YES user_id=80475 I am surprised that making 25% fewer open calls doesn't save more than 1% in startup time. One other thought, I wonder if the timing of these changes is affected by the OS keeping recently loaded files in buffers so that disk access time not included. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921466&group_id=5470 _______________________________________________ Patches mailing list [email protected] http://mail.python.org/mailman/listinfo/patches
