Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: a) the directory cache is out of date, and you should re-read the directory b) the module still isn't there, but is available in a later directory on sys.path (which hasn't yet been visited) c) the module isn't there at all, and the import will eventually fail. How can the interpreter determine which of these it is? It doesn't need to - if there is no file for the module in the cache, it assumes that the cache could be out of date and rebuilds it. If that turns up a file, then fine, else the module doesn't exist. BTW, I'm not thinking of cacheing individual directories, but scanning all the directories and building a single qualified_module_name - pathname mapping. If the cache gets invalidated, all the directories along the path are re-scanned, so a new module will be picked up wherever it is on the path. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: Martin v. Löwis wrote: a) the directory cache is out of date, and you should re-read the directory b) the module still isn't there, but is available in a later directory on sys.path (which hasn't yet been visited) c) the module isn't there at all, and the import will eventually fail. How can the interpreter determine which of these it is? It doesn't need to - if there is no file for the module in the cache, it assumes that the cache could be out of date and rebuilds it. If that turns up a file, then fine, else the module doesn't exist. I lost track. I thought we were talking about creating a cache of directory listings, not a stat cache? If you invalidate the cache when a file name is not listed, you will invalidate it on nearly every import, and multiple times, too: Python looks for foo.py, foo.pyc, foo.so, foomodule.so. At most one of them is found, the others aren't. So if foo.so would be found, are you invalidating the cache because foo.py isn't? BTW, I'm not thinking of cacheing individual directories, but scanning all the directories and building a single qualified_module_name - pathname mapping. If the cache gets invalidated, all the directories along the path are re-scanned, so a new module will be picked up wherever it is on the path. That won't work well with path import objects. You have to observe the order in which sys.path is scanned, for correct semantics. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 7Nov 2006, at 12:20 AM, Greg Ewing wrote: Also, if we do our own directory caching, the question is when to invalidate the cache. I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. Not only IDE's, also the interactive prompt. It is very convenient that you can currently install an additional module when an import fails and then try the import again (at the python prompt). Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo wrote: It would seem good practice to remove all .pycs after checking out a new version of the source, just in case there are other problems such as mismatched timestamps, which can cause the same trouble. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. That's a possibility. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Delaney, Timothy (Tim) wrote: Would it be reasonable to always do a stat() on the directory, reloading if there's been a change? Would this be reliable across platforms? To detect a new shadowing you'd have to stat all the directories along sys.path, not just the one you think the file is in. That might wipe out most of the advantage. It would be different on platforms which provide a way of watching a directory and getting notified of changes. I think MacOSX, Linux and Windows all provide some way of doing that nowadays, although I'm not familiar with the details. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Mon, 6 Nov 2006, Armin Rigo wrote: I know it's a discussion that comes up and dies out regularly. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. I like this approach. Bringing source code and program behaviour closer together makes debugging easier, and if someone wants to run Python programs without source code, then EIBTI. -- ?!ng ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: Currently, you can put a file on disk and import it immediately; that will stop working. One thing I should add is that if you try to import a module that wasn't there before, the interpreter will notice this and has the opportunity to update its idea of what's on the disk. Likewise, if you delete a module, the interpreter will notice when it tries to open a file that no longer exists. The only change would be if you added a module that shadowed something formerly visible further along sys.path -- in between starting the program and attempting to import it for the first time. So I don't think there would be any visible change as far as most people could tell. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: One thing I should add is that if you try to import a module that wasn't there before, the interpreter will notice this and has the opportunity to update its idea of what's on the disk. How will it notice that it wasn't there before? The interpreter will see that it hasn't imported the module; it can't know whether it was there before while trying to resolve the import: when looking at a directory in sys.path, it needs to decide whether to use the directory cache or not. If the directory is not in the cache, it might be one of three things: a) the directory cache is out of date, and you should re-read the directory b) the module still isn't there, but is available in a later directory on sys.path (which hasn't yet been visited) c) the module isn't there at all, and the import will eventually fail. How can the interpreter determine which of these it is? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. -- bye by Wolfgang ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Hi Martin, On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. It caused many headaches in several projects I worked on. Additionally trying to importing .pyo files looks like a complete semantic non-sense, but I can't really argue from experience, as I never run python -O. Typical example: someone in the project removes a .py file, and checks in this change; someone else does an 'svn up', which kills the .py in his working copy, but not the .pyc. These stale .pyc's cause pain, e.g. by shadowing the real module (further down sys.path), or simply by preventing the project's developers from realizing that they forgot to fix some imports. We regularly had obscure problems that went away as soon as we deleted all .pyc files around, but I cannot comment more on that because we never really investigated. I know it's a discussion that comes up and dies out regularly. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. This could e.g. be done with different file extensions (then you just rename the files if you want to ship them as bytecode without source), or with a temporary cache directory (from where you can fish bytecode files if you want to ship them). Experience suggests I should not be holding my breath until something is decided about this, though. If I were asked to come up with a patch I'd simply propose one that removes importing of stale .pyc files (I'm always running a version of Python with such a patch, to avoid the above-mentioned troubles). A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo [EMAIL PROTECTED] wrote: Hi Martin, On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. It caused many headaches in several projects I worked on. Typical example: someone in the project removes a .py file, and checks in this change; someone else does an 'svn up', which kills the .py in his working copy, but not the .pyc. These stale .pyc's cause pain, e.g. by shadowing the real module (further down sys.path), or simply by preventing the project's developers from realizing that they forgot to fix some imports. We regularly had obscure problems that went away as soon as we deleted all .pyc files around, but I cannot comment more on that because we never really investigated. I had a very similar problem the other week when mucking about with a patch to ntpath . I had it in a somewhat small temporary projects folder and needed to run another project. It picked up the local ntpath.py when importing path.py, but then failed because I was working on a 2.5 derived ntpath, but I was using 2.3 to run the other project. After renaming the local ntpath, I continued to get the error until I realized damn pyc and was halfway through a filsystem wide search for the problem code (10 minutes elapsed). About the only place where I have found the need for pyc-without-py importing is for zipimports, specifically as used by py2exe and other freezing applications. I don't know if we want to add a new command line option, or a __future__ import, or something, but I think there should be some method of warning people that an import was performed without source code. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo schrieb: My strong opinion on the matter is that importing a .pyc file if the .py file is not present is wrong in the first place. There is, of course, an important use case (which you are addressing with a different approach): people want to ship only byte code, not source code, because they feel it protects their IP better, and also for space reasons. So outright ignoring pyc files is not really an option. I know it's a discussion that comes up and dies out regularly. My two cents is that it would be saner to have two separate concepts: cache files used internally by the interpreter for speed reasons only, and bytecode files that can be shipped and imported. There once was a PEP to better control byte code file generation; it died because it wasn't implemented. I don't think there is a strong opposition to changing the status quo - it's just that you need a well-designed specification before you start, a serious, all-singing-all-dancing implementation, and a lot of test cases. I believe it is these constraints which have prevented any progress here. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Wolfgang Langner schrieb: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Armin Rigo wrote:
Typical example: someone in the project removes a .py file, and checks
in this change; someone else does an 'svn up', which kills the .py in
his working copy, but not the .pyc. These stale .pyc's cause pain,
e.g.
by shadowing the real module (further down sys.path), or simply by
preventing the project's developers from realizing that they forgot to
fix some imports. We regularly had obscure problems that went away as
soon as we deleted all .pyc files around, but I cannot comment more on
that because we never really investigated.
This is exactly why I always use this module:
== nobarepyc.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
import ihooks
import os
class _NoBarePycHooks(ihooks.Hooks):
def load_compiled(self, name, filename, *args, **kwargs):
sourcefn = os.path.splitext(filename)[0] + .py
if not os.path.isfile(sourcefn):
raise ImportError('forbidden import of bare .pyc file: %r' %
filename)
return ihooks.Hooks.load_compiled(name, filename, *args, **kwargs)
ihooks.ModuleImporter(ihooks.ModuleLoader(_NoBarePycHooks())).install()
== /nobarepyc.py
Just import it before importing anything else (or in site.py if you prefer)
and you'll be done.
Ah, it doesn't work with zipimports...
--
Giovanni Bajo
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter? I just had an idea: we could have only pyc files, and *no* way to identify whether specific optimizations (-O, -OO --only-strip-docstrings, whatever) were performed on them or not. So, if you regularly run different python applications with different optimization settings, you'll end up with .pyc files containing bytecode that was generated with mixed optimization settings. It doesn't really matter in most cases, after all. Then, we add a single command line option (eg: -I) which is: ignore *every* .pyc file out there, and regenerate them as needed. So, the few times that you really care that a certain application is run with a specific setting, you can use python -I -OO app.py. And that's all. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/6/06, Giovanni Bajo [EMAIL PROTECTED] wrote: Martin v. Löwis wrote: Why not only import *.pyc files and no longer use *.pyo files. It is simpler to have one compiled python file extension. PYC files can contain optimized python byte code and normal byte code. So what would you do with the -O option of the interpreter?I just had an idea: we could have only pyc files, and *no* way to identifywhether specific optimizations (-O, -OO --only-strip-docstrings, whatever) were performed on them or not. So, if you regularly run different pythonapplications with different optimization settings, you'll end up with .pycfiles containing bytecode that was generated with mixed optimization settings. It doesn't really matter in most cases, after all.I don't know about that. If you suspected that a failure could be because of some bytecode optimization you were trying wouldn't you like to be able to tell easily that fact? Granted our situation is not as bad as gcc in terms the impact of having to regenerate a compiled version, but it still would be nice to be able to make sure that every .pyc file is the same. We would need to make it easy to blast out every .pyc file found if we did allow mixing of optimizations (as you suggest below). Then, we add a single command line option (eg: -I) which is: ignore *every* .pyc file out there, and regenerate them as needed. So, the fewtimes that you really care that a certain application is run with a specificsetting, you can use python -I -OO app.py. That might work.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: A stat call will not only look at the directory entry, but also look at the inode. This will require another disk access, as the inode is at a different location of the disk. That should be in favour of the directory-reading approach, since e.g. to find out which if any of x.py/x.pyc/x.pyo exists, you only need to look for the names. It depends on the file system you are using. An NTFS directory lookup is a B-Tree search; ... Yes, I know that some file systems are smarter; MacOS HFS is another one that uses b-trees. However it still seems to me that looking up a path in a file system is a much heavier operation than looking up a Python dict, even if everything is in memory. You have to parse the path, and look up each component separately in a different directory tree or whatever. The way I envisage it, you would read all the directories and build a single dictionary mapping fully-qualified module names to pathnames. Any given import then takes at most one dict lookup and one access of a known-to-exist file. For a large directory, the cost of reading in the entire directory might be higher than the savings gained from not having to search it. Possibly. I guess we'd need some timings to assess the meaning of large. Also, if we do our own directory caching, the question is when to invalidate the cache. I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Tue, 07 Nov 2006 12:20:00 +1300, Greg Ewing [EMAIL PROTECTED] wrote: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. Another questionable optimization which changes application- level semantics. No, please? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. That would be a change in behavior, of course. Currently, you can put a file on disk and import it immediately; that will stop working. I'm pretty sure that there are a number of applications that rely on this specific detail of the current implementation (and not only IDEs). It still might be worthwhile to make such a change, but I'd like to see practical advantages demonstrated first. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: Greg Ewing schrieb: I think I'd be happy with having to do that explicitly. I expect the vast majority of Python programs don't need to track changes to the set of importable modules during execution. The exceptions would be things like IDEs, and they could do a cache flush before reloading a module, etc. That would be a change in behavior, of course. Currently, you can put a file on disk and import it immediately; that will stop working. I'm pretty sure that there are a number of applications that rely on this specific detail of the current implementation (and not only IDEs). Would it be reasonable to always do a stat() on the directory, reloading if there's been a change? Would this be reliable across platforms? Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
[Off-list] Brett Cannon wrote: [...] Hopefully my import rewrite is flexible enough that people will be able to plug in their own importer/loader for the filesystem so that they can tune how things like this are handled (e.g., caching what files are in a directory, skipping bytecode files, etc.). I just wondered whether you plan to support other importers of the PEP 302 style? I have been experimenting with import from database, and would like to see that work migrate to your rewrite if possible. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Sun, Nov 05, 2006, Martin v. L?wis wrote: Greg Ewing schrieb: Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? That should never be better: the system will cache the directory blocks, also, and it will do a better job than Python will. Maybe so, but I recently dealt with a painful bottleneck in Python code caused by excessive stat() calls on a directory with thousands of files, while the os.listdir() function was bogging things down hardly at all. Granted, Python bytecode was almost certainly the cause of much of the overhead, but I still suspect that a simple listing will be faster in C code because of fewer system calls. It should be a matter of profiling before this suggestion is rejected rather than making assertions about what should be happening. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ In many ways, it's a dull language, borrowing solid old concepts from many other languages styles: boring syntax, unsurprising semantics, few automatic coercions, etc etc. But that's one of the things I like about it. --Tim Peters on Python, 16 Sep 1993 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/5/06, Steve Holden [EMAIL PROTECTED] wrote: [Off-list]Brett Cannon wrote:[...] Hopefully my import rewrite is flexible enough that people will be able to plug in their own importer/loader for the filesystem so that they can tune how things like this are handled ( e.g., caching what files are in a directory, skipping bytecode files, etc.).I just wondered whether you plan to support other importers of the PEP302 style? I have been experimenting with import from database, and would like to see that work migrate to your rewrite if possible.Yep. The main point of this rewrite is to refactor the built-in importers to be PEP 302 importers so that they can easily be left out to protect imports. Plus I have made sure that doing something like .ptl files off the filesystem is simple (a subclass with a single method overloaded) or introducing a DB as a back-end store (should only require the importer/loader part; can even use an existing class to handle whether bytecode should be recreated or not). Since a DB back-end is a specific use-case I even have notes in the module docstring stating how I would go about doing it.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Aahz schrieb: Maybe so, but I recently dealt with a painful bottleneck in Python code caused by excessive stat() calls on a directory with thousands of files, while the os.listdir() function was bogging things down hardly at all. Granted, Python bytecode was almost certainly the cause of much of the overhead, but I still suspect that a simple listing will be faster in C code because of fewer system calls. It should be a matter of profiling before this suggestion is rejected rather than making assertions about what should be happening. That works both ways, of course: whoever implements such a patch should also provide profiling information. Last time I changed the importing code to reduce the number of stat calls, I could hardly demonstrate a speedup. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Martin v. Löwis wrote: That should never be better: the system will cache the directory blocks, also, and it will do a better job than Python will. If that's really the case, then why do discussions of how improve Python startup speeds seem to focus on the number of stat calls made? Also, cacheing isn't the only thing to consider. Last time I looked at the implementation of unix file systems, they mostly seemed to do directory lookups by linear search. Unless that's changed a lot, I have a hard time seeing how that's going to beat Python's highly-tuned dictionaries. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: That should never be better: the system will cache the directory blocks, also, and it will do a better job than Python will. If that's really the case, then why do discussions of how improve Python startup speeds seem to focus on the number of stat calls made? A stat call will not only look at the directory entry, but also look at the inode. This will require another disk access, as the inode is at a different location of the disk. Also, cacheing isn't the only thing to consider. Last time I looked at the implementation of unix file systems, they mostly seemed to do directory lookups by linear search. Unless that's changed a lot, I have a hard time seeing how that's going to beat Python's highly-tuned dictionaries. It depends on the file system you are using. An NTFS directory lookup is a B-Tree search; NT has not been doing linear search since its introduction 15 years ago. Linux only recently started doing tree-based directories with the introduction of ext4. However, Linux' in-memory directory cache (the dcache) doesn't need to scan over the directory block structure; not sure whether it uses linear search still. For a small directory, the difference is likely negligible. For a large directory, the cost of reading in the entire directory might be higher than the savings gained from not having to search it. Also, if we do our own directory caching, the question is when to invalidate the cache. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Importing .pyc in -O mode and vice versa
Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. This reasoning is somewhat flawed, of course: to achieve consistency, one could also change the zipimporter instead. However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? What do you think? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Sat, Nov 04, 2006 at 04:47:37PM +0100, Martin v. L?wis wrote: Patch #1346572 proposes to also search for .pyc when OptimizeFlag is set, and for .pyo when it is not set. The author argues this is for consistency, as the zipimporter already does that. This reasoning is somewhat flawed, of course: to achieve consistency, one could also change the zipimporter instead. However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? What do you think? +1 from me. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true. (wouldn't transparent import of PYO files mean that you end up with a program where some assertions apply, and others don't? could be con- fusing...) That's also true, however, it might still be better to do that instead of raising an ImportError. I'm not sure whether a scenario were you have only .pyo files for some modules and only .pyc files for others is really likely, though, and the performance hit of another system call doesn't sound attractive. So I guess that zipimport should stop importing .pyo files if OptimizeFlag is false, then? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Hi, I'm the author of this patch and we are already using it in Python port for Maemo platform. We are using .pyo modules mainly to remove docstrings from the modules. We've discussed about this patch here[1] before. Now, I agree that the zipimport behaviour is incorrect but I don't have other option to remove docstrings of a .pyc file. I'm planning to send a patch that adds a --remove-docs to the Python interpreter to replace the -OO option that create only .pyo files. [1] http://mail.python.org/pipermail/python-dev/2005-November/057959.html On 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true. [cut] -- Osvaldo Santana Neto (aCiDBaSe) http://www.pythonologia.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things.That's true. (wouldn't transparent import of PYO files mean that you end up with a program where some assertions apply, and others don't?could be con- fusing...)That's also true, however, it might still be better to do that insteadof raising an ImportError.I'm not sure whether a scenario were you have only .pyo files for some modules and only .pyc files for others is really likely, though,and the performance hit of another system call doesn't sound attractive.So I guess that zipimport should stop importing .pyo files if OptimizeFlag is false, then?Yes, I think it should. When I get around to rewriting zipimport for my import rewrite it will do this by default.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Osvaldo Santana [EMAIL PROTECTED] wrote: Hi,I'm the author of this patch and we are already using it in Pythonport for Maemo platform.We are using .pyo modules mainly to remove docstrings from themodules. We've discussed about this patch here[1] before. Now, I agree that the zipimport behaviour is incorrect but I don'thave other option to remove docstrings of a .pyc file.I'm planning to send a patch that adds a --remove-docs to the Python interpreter to replace the -OO option that create only .pyo files.[1] http://mail.python.org/pipermail/python-dev/2005-November/057959.html The other option is to do away with .pyo files: http://www.python.org/dev/summary/2005-11-01_2005-11-15/#importing-pyc-and-pyo-files Guido has said he wouldn't mind it, but then .pyc files need to grow a field or so to be able to store what optimizations were used. While this would lead to more bytecode regeneration, it would help deal with this case and allow for more optimizations on the bytecode. -BrettOn 11/4/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Fredrik Lundh schrieb: However, I find the proposed behaviour reasonable: Python already automatically imports the .pyc file if .py is not given and vice versa. So why not look for .pyo if the .pyc file is not present? well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. That's true.[cut]--Osvaldo Santana Neto (aCiDBaSe)http://www.pythonologia.org___Python-Dev mailing list Python-Dev@python.orghttp://mail.python.org/mailman/listinfo/python-devUnsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On Sun, 05 Nov 2006 14:21:34 +1300, Greg Ewing [EMAIL PROTECTED] wrote: Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? Bad for large directories. There's a cross-over at some number of entries. Maybe Python should have a runtime-tuned heuristic for selecting a filesystem traversal mechanism. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
Greg Ewing schrieb: Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things. Instead of searching for things by doing a stat call for each possible file name, would it perhaps be faster to read the contents of all the directories along sys.path into memory and then go searching through that? That should never be better: the system will cache the directory blocks, also, and it will do a better job than Python will. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Importing .pyc in -O mode and vice versa
On 11/4/06, Jean-Paul Calderone [EMAIL PROTECTED] wrote: On Sun, 05 Nov 2006 14:21:34 +1300, Greg Ewing [EMAIL PROTECTED] wrote:Fredrik Lundh wrote: well, from a performance perspective, it would be nice if Python looked for *fewer* things, not more things.Instead of searching for things by doing a stat callfor each possible file name, would it perhaps befaster to read the contents of all the directories along sys.path into memory and then go searchingthrough that?Bad for large directories.There's a cross-over at some numberof entries.Maybe Python should have a runtime-tuned heuristicfor selecting a filesystem traversal mechanism. Hopefully my import rewrite is flexible enough that people will be able to plug in their own importer/loader for the filesystem so that they can tune how things like this are handled (e.g., caching what files are in a directory, skipping bytecode files, etc.). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
