Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-09 Thread Nick Coghlan
Terry Reedy wrote: Definitely. I have even wondered whether it would be possible to cache not just the bytecode for initializing a module, but also the initialized module itself (perhaps minus the name bindings for other imported modules). Not easily, since running the module may have other

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Nick Coghlan
Ron Adam wrote: To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution time. Hmmm, I wonder if there's a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Terry Reedy
On 2/8/2010 7:54 AM, Nick Coghlan wrote: Ron Adam wrote: To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 02:20 PM, Guido van Rossum wrote: Upon further reflection, I agree.  __file__ also points to the source in Python 2.7. Not in the 2.7 svn repo I have access to. It still points to the .pyc file if it was used. Ah, I was fooled by a missing pyc file. Run it a second time

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 04, 2010, at 03:00 PM, Glenn Linderman wrote: When a PEP 3147 (if modified by my suggestion) version of Python runs, and the directory doesn't exist, and it wants to create a .pyc, it would create the directory, and put the .pyc there. Sort of just like how it creates .pyc files, now,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers. One important reason for wanting to keep the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Michael Foord
On 07/02/2010 17:48, Barry Warsaw wrote: [snip...] And I propose not to disturb this in 2.7, at least not by default. I'm fine though with a flag or distro-overridable config setting to change this behavior. Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll call

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: Martin v. Löwis schrieb: There is also the issue of race conditions with multiple simultaneous accesses. The original format for the PEP had race conditions for multiple simultaneous writers; ZIP will also have race conditions for

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: I must admit I quite like the __pyr__ directory approach as well. Since the interpreter knows the suffix it is looking for, names shouldn't conflict. Using a single directory allows the name to be less cryptic, too (e.g. __pycache__). Something

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: The pyc/pyo files are just an optimisation detail, and are essentially temporary. Given that, if they were to live in a single directory, to me it seems obvious that the default location for that should be in the system temporary directory. I an

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:02 PM, Guido van Rossum wrote: On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw ba...@python.org wrote: On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: It's an interesting challenge to write the file in such a way that it's safe for a reader and writer to co-exist. Like

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is proposing this out of fear that the new behavior might upset

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 07, 2010, at 05:59 PM, Michael Foord wrote: On 07/02/2010 17:48, Barry Warsaw wrote: [snip...] And I propose not to disturb this in 2.7, at least not by default. I'm fine though with a flag or distro-overridable config setting to change this behavior. Cool. I'm not sure this

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread M.-A. Lemburg
Barry Warsaw wrote: On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: How about using an optionally relative cache dir setting to let the user decide ? Why do we need that level of flexibility? It's very easy to implement (see the code I posted) and gives you a lot of control with a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 10:17 AM, Barry Warsaw ba...@python.org wrote: On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: I must admit I quite like the __pyr__ directory approach as well. Since the interpreter knows the suffix it is looking for, names shouldn't conflict. Using a single directory

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Brett Cannon
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw ba...@python.org wrote: On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 12:23 PM, Brett Cannon br...@python.org wrote: On Sun, Feb 7, 2010 at 10:44, Barry Warsaw ba...@python.org wrote: On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Ron Adam
Barry Warsaw wrote: On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers. One important reason

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed to a .pyc file any more (even if one existed) that would be just fine. So

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 05, 2010, at 07:37 PM, Nick Coghlan wrote: Brett Cannon wrote: Does code exist out there where people are constructing bytecode from multiple files for a single module? I'm quite prepared to call YAGNI on that idea and just return a 2-tuple of source filename and compiled filename. Me

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: How about using an optionally relative cache dir setting to let the user decide ? Why do we need that level of flexibility? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: It's also the case that having to run Python to manage my own filesystem would very annoying. If a dev has a broken .pyc that prevents the affected Python build from even starting how are they meant to use the nonfunctioning interpreter to find

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 09:26 AM, Floris Bruynooghe wrote: On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: I don't understand the distinction you're making between those two options. Can you explain what you mean by each of “siblings” and “folder-per-folder”? sibilings: the original

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread exarkun
On 08:21 pm, ba...@python.org wrote: On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed to a .pyc file any more (even if one

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 12:21 PM, Barry Warsaw ba...@python.org wrote: On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed to a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote: It does this by allowing many different byte compilation files (.pyc files) to be co-located with the Python source file (.py file). It would be nice if all the compilation files could be tucked into one single zipfile per directory to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: It's an interesting challenge to write the file in such a way that it's safe for a reader and writer to co-exist. Like Brett, I considered an append-only scheme, but one needs to handle the case where the bytecode for a particular magic number

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote: So what would you do for concurrent writers, then? The current implementation relies on creat(O_EXCL) to be atomic, so a second writer would just fail. This is but the only IO operation that is guaranteed to be atomic (along with mkdir(2)), so

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw ba...@python.org wrote: On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: It's an interesting challenge to write the file in such a way that it's safe for a reader and writer to co-exist. Like Brett, I considered an append-only scheme, but one needs

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Barry Warsaw ba...@python.org writes: On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: It's also the case that having to run Python to manage my own filesystem would very annoying. […] Files that are problematic wouldn't need Python to manage any more than currently. The suggestion was just

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 4:27 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Barry Warsaw ba...@python.org writes: I agree. I'd prefer to have a predictable place for the cached files, independent of having to run Python to tell you where that is. Right; I don't see who would disagree with

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
Ben Finney wrote: Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled bytecode files”. The more decoupled they are, the harder it is to manually

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 5:10 PM, Nick Coghlan ncogh...@gmail.com wrote: Ben Finney wrote: Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Nick Coghlan ncogh...@gmail.com writes: The more decoupled they are, the harder it is to manually find the bytecode file. Okay. So it's not so much about “predictable”, but rather about “predictable by a human without too much cognitive effort”. I can see value in that, though it's best to be

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
exar...@twistedmatrix.com wrote: On 08:21 pm, ba...@python.org wrote: On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-05 Thread Nick Coghlan
Brett Cannon wrote: If we add a new method like get_filenames(), I would suggest going with Antoine's suggestion of a tuple for __compiled__ (allowing loaders to indicate that they actually constructed the runtime bytecode from multiple cached files on-disk). Does code exist out there

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Brett Cannon
On Wed, Feb 3, 2010 at 13:33, Martin v. Löwis mar...@v.loewis.de wrote: Guido van Rossum wrote: On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On the issue of __file__, I'd suggesting not being too hasty in deprecating that in favour of __source__. While I can see

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Glenn Linderman
On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` directory and write a `pyc` file to that directory with the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Nick Coghlan
Brett Cannon wrote: My thinking is we deprecate get_filename() and introduce some new method that returns a two-item tuple (get_paths?). First item is where the source should be, and the second is where the bytecode is if it exists (else it's None). Putting both calculations into a single

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Nick Coghlan
Glenn Linderman wrote: Alt. C... source-file-dir/__pyr_version__, each Python version with different bytecode would have some sort of version string or magic number that identifies it, and would look only in that directory for its .pyc/.pyo files. I prefer C for 4 reasons: 1) easier to blow

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Brett Cannon
On Thu, Feb 4, 2010 at 13:51, Nick Coghlan ncogh...@gmail.com wrote: Brett Cannon wrote: My thinking is we deprecate get_filename() and introduce some new method that returns a two-item tuple (get_paths?). First item is where the source should be, and the second is where the bytecode is if

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Eric Smith
Glenn Linderman wrote: On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` directory and write a `pyc` file to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Glenn Linderman
On approximately 2/4/2010 2:28 PM, came the following characters from the keyboard of Eric Smith: Glenn Linderman wrote: On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Floris Bruynooghe
On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: Barry Warsaw ba...@python.org writes: I suppose this is going to be very subjective, but in skimming the thread it seems like most people like putting the byte code cache artifacts in subdirectories (be they siblings or

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Glenn Linderman
On approximately 2/2/2010 7:05 PM, came the following characters from the keyboard of Guido van Rossum: On Tue, Feb 2, 2010 at 5:41 PM, Glenn Lindermanv+pyt...@g.nevcal.com wrote: On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum:

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Michael Foord
On 03/02/2010 06:50, Barry Warsaw wrote: I have to say up front that I'm somewhat shocked at how quickly this thread has exploded! Since I'm sprinting this week, I haven't thoroughly read every message and won't have time tonight to answer every question, but I'll try to pick out some common

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread M.-A. Lemburg
On 03/02/2010 06:50, Barry Warsaw wrote: As to the question of sibling directories or folder-per-folder I think performance issues should be the deciding factor. There are file system limitations to consider (but also a wide variety of file systems in use). Do the number of stat calls

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Antoine Pitrou
Barry Warsaw barry at python.org writes: As to the question of sibling directories or folder-per-folder I think performance issues should be the deciding factor. There are file system limitations to consider (but also a wide variety of file systems in use). Do the number of stat calls

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Ben Finney wrote: I don't think keeping the cache files in a mass of intertwingled extra subdirectories is the way to solve that problem. That speaks, rather, to the need for Python to be able to find the file on behalf of the user and blow it away on request, so the user doesn't need to go

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Floris Bruynooghe wrote: Personally I'm +1 on the folder-per-folder option. Of all the proposed options, I also dislike the SVN/CVS style folder structure the least ;) Cheers, Nick. P.S. Translation of the double negative: I don't find any of the solutions, even the current .pyc/.pyo approach,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Bob Ippolito wrote: I like this option as well, but why not just name the directory .pyc instead of __pyr__ or .pyr? That way people probably won't even have to reconfigure their tools to ignore it :) This actually came up in another part of the thread. The conclusion was that, since the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Glenn Linderman wrote: On approximately 2/2/2010 7:05 PM, came the following characters from the keyboard of Guido van Rossum: On Tue, Feb 2, 2010 at 5:41 PM, Glenn Lindermanv+pyt...@g.nevcal.com wrote: Agreed. But in reading that, it somehow triggered a question: does zipimport only

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Barry Warsaw wrote: Encoding the magic number in the file name under .pyr would I thought make the look up scheme more efficient since the import machinery can craft the file name directly. I agree it's not very human friendly because nobody really knows which magic numbers are associated

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Barry Warsaw
On Feb 03, 2010, at 11:31 PM, Nick Coghlan wrote: Having a lookup dictionary from Python version + C API magic numbers to the magic strings used in cache filenames in the import engine shouldn't be too tricky. I'll admit it wasn't until the thread had already been going for a while that I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Barry Warsaw
On Feb 03, 2010, at 12:57 PM, Antoine Pitrou wrote: How about doing measurements /with the current implementation/? Everyone seems to worry about stat() calls but there doesn't seem to be any figures to evaluate their significance. Yes, very good idea, if for no other reason than to give us a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Barry Warsaw wrote: On Feb 03, 2010, at 11:31 PM, Nick Coghlan wrote: Having a lookup dictionary from Python version + C API magic numbers to the magic strings used in cache filenames in the import engine shouldn't be too tricky. I'll admit it wasn't until the thread had already been going

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Brett Cannon
On Wed, Feb 3, 2010 at 05:27, Nick Coghlan ncogh...@gmail.com wrote: Glenn Linderman wrote: On approximately 2/2/2010 7:05 PM, came the following characters from  the keyboard of Guido van Rossum: On Tue, Feb 2, 2010 at 5:41 PM, Glenn Lindermanv+pyt...@g.nevcal.com  wrote: Agreed.  But in

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Ben Finney
Nick Coghlan ncogh...@gmail.com writes: P.S. Translation of the double negative: I don't find any of the solutions, even the current .pyc/.pyo approach, to be particularly elegant, so I can't really say I like any of them in an absolute sense. However, having a single cache folder inside each

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Guido van Rossum
On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On the issue of __file__, I'd suggesting not being too hasty in deprecating that in favour of __source__. While I can see a lot of value in having it point to the source file more often with a different attribute that

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Martin v. Löwis
Guido van Rossum wrote: On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On the issue of __file__, I'd suggesting not being too hasty in deprecating that in favour of __source__. While I can see a lot of value in having it point to the source file more often with a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Guido van Rossum wrote: On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan ncogh...@gmail.com wrote: On the issue of __file__, I'd suggesting not being too hasty in deprecating that in favour of __source__. While I can see a lot of value in having it point to the source file more often with a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Ben Finney wrote: Nick Coghlan ncogh...@gmail.com writes: P.S. Translation of the double negative: I don't find any of the solutions, even the current .pyc/.pyo approach, to be particularly elegant, so I can't really say I like any of them in an absolute sense. However, having a single

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Ben Finney
Thanks for the explanation. Nick Coghlan ncogh...@gmail.com writes: Being able to get rid of the existing .pyc/.pyo clutter at the same time is just a bonus. Okay. I maintain (unsurprisingly) that replacing it with subdirectory clutter is a poor bargain. But I have nothing new to add on that

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Larry Hastings
On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called __pyr__ rather than .pyr. Guido van Rossum wrote: Exactly what I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Sebastian Rittau
On Sun, Jan 31, 2010 at 12:44:33PM +0100, Georg Brandl wrote: At least to me, this does not explain why an unwanted (why unwanted? If it's unwanted, set PYTHONDONTWRITEBYTECODE=1) directory is worse than an unwanted file. A directory feels different than. For example, typing ls in my shell

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Neil Schemenauer
Nick Coghlan ncogh...@gmail.com wrote: Henning von Bargen wrote: The solution is so obvious: Why not use a .pyr file that is internally a zip file? I think a Zip file might be the right approach too. Either you could have directories in the zip file for each version, e.g. 2.7/foo.pyc

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Glenn Linderman
On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are way to complex to be writing. Agreed. But in reading that, it somehow triggered a question: does zipimport only work for zipfiles, or does it work for any archive format

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Guido van Rossum
On Tue, Feb 2, 2010 at 5:41 PM, Glenn Linderman v+pyt...@g.nevcal.com wrote: On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are way to complex to be writing. Agreed.  But in reading that, it somehow triggered a question:

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Bob Ippolito
On Sun, Jan 31, 2010 at 11:16 AM, Guido van Rossum gu...@python.org wrote: Whoa. This thread already exploded. I'm picking this message to respond to because it reflects my own view after reading the PEP. On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
I have to say up front that I'm somewhat shocked at how quickly this thread has exploded! Since I'm sprinting this week, I haven't thoroughly read every message and won't have time tonight to answer every question, but I'll try to pick out some common ideas. I really appreciate everyone's input

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 30, 2010, at 11:21 PM, Vitor Bosshard wrote: Why not: foo.py foo.pyc # 2.7 or 3.2 foo.27.pyc foo.32.pyc etc. Because this clutters the module's directory more than it does today, which I considered to be a negative factor. And as others have pointed out, there isn't a one-to-one

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 03:07 PM, Ben Finney wrote: In other words, my understanding is that the current PEP would have the following tree for an example project:: foo/ __init__.py __init__.pyr/ deadbeef.pyc decafbad.pyc lorem.py lorem.pyr/

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 12:36 PM, Georg Brandl wrote: Not really -- much of the code I've seen that tries to guess the source file name from a __file__ value just does something like this: if fname.lower().endswith(('.pyc', '.pyo')): fname = fname[:-1] That's not compatible with using .pyr,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 09:30 PM, Martin v. Löwis wrote: If a single pyc folder is used, I think an additional __source__ attribute would be needed to indicate what source file time stamp had been checked (if any) to determine that the byte code file is current. This is a good point. __file__ is

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw ba...@python.org writes: I suppose this is going to be very subjective, but in skimming the thread it seems like most people like putting the byte code cache artifacts in subdirectories (be they siblings or folder-per-folder). I don't understand the distinction you're making

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw ba...@python.org writes: If you have to manually blow away a particular pyc file, folder-per-folder makes it much easier to find exactly what you want to blow away without have to search up the file system, and then back down again to find the pyc file to delete. How many ..'s

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
3. In each top level directory on sys.path, shadow file heirarchy Major Pro: trivial to separate out all cached files Major Con: ??? (I got nuthin') The major con of this option (and option 2) is an ambiguity of where to look for in case of packages. In particular for namespace packages

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Gertjan Klein
Hanno Schlichting wrote: +1 for a single strategy that is used in all cases. The current solution could be phased out across multiple releases, but in the end there should be a single approach and no flag. Otherwise some code and tools will only support one of the approaches, especially if this

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
Would you still be a -1 on making it the new scheme the default if it used a single cache directory instead? That would actually be cleaner than the current solution rather than messier. Well, I guess no, although additional directories are always more intrusive than additional files

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread M.-A. Lemburg
Raymond Hettinger wrote: On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: Abstract This PEP describes an extension to Python's import mechanism which improves sharing of Python source code files among multiple installed different versions of the Python interpreter. +1 +1

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:16, Guido van Rossum gu...@python.org wrote: Whoa. This thread already exploded. I'm picking this message to respond to because it reflects my own view after reading the PEP. On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan 31,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:04, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: Abstract This PEP describes an extension to Python's import mechanism which improves sharing of Python source code files among multiple installed

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
Le Mon, 01 Feb 2010 11:35:19 -0800, Brett Cannon a écrit : As others have said, an uncompressed zip file could work here. Or even a file format where the first 4 bytes is the timestamp and then after that are chunks of length-of-bytecode|magic|bytecode. That allows for opening a file in

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
And I disagree this would be difficult as the PEP suggests given the proper file format. For zip files zipimport already has the read code in C; it just would require the code to write to a zip file. And as for the format I mentioned above, that's dead-simple to implement. How do you write to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Mon, Feb 1, 2010 at 13:19, Martin v. Löwis mar...@v.loewis.de wrote: And I disagree this would be difficult as the PEP suggests given the proper file format. For zip files zipimport already has the read code in C; it just would require the code to write to a zip file. And as for the format

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
On Mon, Feb 1, 2010 at 13:19, Martin v. Löwis mar...@v.loewis.de wrote: How do you write to a zipfile while others are reading it? On Mon, Feb 1, 2010 at 1:23 PM, Brett Cannon br...@python.org wrote: By hating concurrency (i.e. I don't have an answer which kills my idea). The python I use

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
The python I use (win32 2.6.2) does not complain if it cannot read from or write to a .pyc; and thus it handles multiple python processes trying to create .pyc files at the same time. Is the .zip case really any different? Since .pyc files are an optimization, it seems natural and correct

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
The python I use (win32 2.6.2) does not complain if it cannot read from or write to a .pyc; and thus it handles multiple python processes trying to create .pyc files at the same time. Is the .zip case really any different? [ snip discussion of difficulty of writing a sharing-safe update ] On

[Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Pablo Mouzo
On Mon, Feb 1, 2010 at 11:56 PM, Pablo Mouzo pablomo...@gmail.com wrote: On Mon, Feb 1, 2010 at 10:23 PM, Paul Du Bois paul.dub...@gmail.com wrote: [...] Sorry, I'm guilty of having assumed that the POSIX API has an operation analogous to win32 CreateFile(GENERIC_WRITE, 0 /* ie,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Henning von Bargen wrote: I like the idea of the PEP. On the other hand, I dislike using directories for it. Others have explained enough reasons for why creating many directories is a bad idea; and there may be other reasons (file-system limits for number of directories, problems when the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. In order to ease the burden on operating system

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Ben Finney
Nick Coghlan ncogh...@gmail.com writes: Actually, this is the first post I've seen noting objective problems with the use of a subdirectory. The others were just a subjective difference in perspective that saw subdirectory clutter as somehow being worse than file clutter. Here's another one,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Agreed this should be discussed in the PEP, but one obvious problem is the speed impact. Picking up a file from a subdirectory is going to introduce less overhead than unpacking it from a zipfile. There is also the issue of race conditions with multiple simultaneous accesses. The original

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Antoine Pitrou
Barry Warsaw barry at python.org writes: Putting Python version numbers in the packages would be a maintenance nightmare, since all the packages - *and their dependencies* - would have to be updated every time a new Python release was added or removed from the distribution. Because of the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 07:29, schrieb Nick Coghlan: Vitor Bosshard wrote: There is no one-to-one correspondence between Python version and pyc magic numbers. Different runtime options may change the magic number and different versions may reuse a magic number Good point. Runtime options would need

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 05:18, schrieb Ben Finney: Nick Coghlan ncogh...@gmail.com writes: It won't be cluttered with subfolders - you will have at most one .pyr per source .py file. If that doesn't meet your threshold of “cluttered with subfolders”, I'm at a loss for words to think where that

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 10:21, schrieb Ben Finney: Nick Coghlan ncogh...@gmail.com writes: Actually, this is the first post I've seen noting objective problems with the use of a subdirectory. The others were just a subjective difference in perspective that saw subdirectory clutter as somehow being

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Simon Cross
On Sun, Jan 31, 2010 at 1:54 PM, Hanno Schlichting ha...@hannosch.eu wrote: I'd be a big +1 to using a single .pyr directory per source directory. I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Hanno Schlichting
On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called __pyr__ rather than .pyr. Do you have any specific reason for

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Simon Cross
On Sun, Jan 31, 2010 at 2:13 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to

  1   2   >