Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Henning von Bargen wrote: I like the idea of the PEP. On the other hand, I dislike using directories for it. Others have explained enough reasons for why creating many directories is a bad idea; and there may be other reasons (file-system limits for number of directories, problems when the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. In order to ease the burden on operating system

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Ben Finney
Nick Coghlan ncogh...@gmail.com writes: Actually, this is the first post I've seen noting objective problems with the use of a subdirectory. The others were just a subjective difference in perspective that saw subdirectory clutter as somehow being worse than file clutter. Here's another one,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Agreed this should be discussed in the PEP, but one obvious problem is the speed impact. Picking up a file from a subdirectory is going to introduce less overhead than unpacking it from a zipfile. There is also the issue of race conditions with multiple simultaneous accesses. The original

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Antoine Pitrou
Barry Warsaw barry at python.org writes: Putting Python version numbers in the packages would be a maintenance nightmare, since all the packages - *and their dependencies* - would have to be updated every time a new Python release was added or removed from the distribution. Because of the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 07:29, schrieb Nick Coghlan: Vitor Bosshard wrote: There is no one-to-one correspondence between Python version and pyc magic numbers. Different runtime options may change the magic number and different versions may reuse a magic number Good point. Runtime options would need

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 05:18, schrieb Ben Finney: Nick Coghlan ncogh...@gmail.com writes: It won't be cluttered with subfolders - you will have at most one .pyr per source .py file. If that doesn't meet your threshold of “cluttered with subfolders”, I'm at a loss for words to think where that

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 10:21, schrieb Ben Finney: Nick Coghlan ncogh...@gmail.com writes: Actually, this is the first post I've seen noting objective problems with the use of a subdirectory. The others were just a subjective difference in perspective that saw subdirectory clutter as somehow being

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Simon Cross
On Sun, Jan 31, 2010 at 1:54 PM, Hanno Schlichting ha...@hannosch.eu wrote: I'd be a big +1 to using a single .pyr directory per source directory. I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Hanno Schlichting
On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called __pyr__ rather than .pyr. Do you have any specific reason for

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Simon Cross
On Sun, Jan 31, 2010 at 2:13 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Georg Brandl wrote: Am 31.01.2010 07:18, schrieb Nick Coghlan: Ben Finney wrote: Could we instead have a single subdirectory for each tree of module packages, keeping them tidily out of the way of the source files, while making them located just as deterministically:: Not easily. With the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Georg Brandl wrote: Then why did Subversion choose to follow the CVS way and create a subdirectory in each versioned directory? IMO, this is much more annoying given the alternative of a single .hg/.bzr/whatever directory. For .pyc vs .pyr, you didn't have the alternative of putting all that

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Vitor Bosshard
2010/1/31 Georg Brandl g.bra...@gmx.net: foo.py foo.pyr/   cpython-25.pyc   cpython-25U.pyc   cpython-27.pyc   cpython-27U.pyc   cpython-32.pyc   unladen-011.pyc   wpython-11.pyc +1.  It should be quite easy to assign a new name every time the magic number is updated. If we don't

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Georg Brandl wrote: +1. Having a single (visible) __pyr__ directory is much less clutter than multiple .pyc files anyway. Also, don't forget Windows users, for whom the dot convention doesn't mean anything. I must admit I quite like the __pyr__ directory approach as well. Since the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Vitor Bosshard wrote: Optimizing disk space (and marginal compile time) is not worth the mental overhead this would introduce. Better keep it as clear and simple as possible, i.e. create different .pyc files even if the bytecode doesn't change between releases. Yeah, makes sense. Given the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Vitor Bosshard
2010/1/31 Nick Coghlan ncogh...@gmail.com: Georg Brandl wrote: Then why did Subversion choose to follow the CVS way and create a subdirectory in each versioned directory?  IMO, this is much more annoying given the alternative of a single .hg/.bzr/whatever directory. For .pyc vs .pyr, you

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Georg Brandl
Am 31.01.2010 14:02, schrieb Nick Coghlan: Georg Brandl wrote: Then why did Subversion choose to follow the CVS way and create a subdirectory in each versioned directory? IMO, this is much more annoying given the alternative of a single .hg/.bzr/whatever directory. For .pyc vs .pyr, you

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread R. David Murray
On Sun, 31 Jan 2010 13:13:24 +0100, Georg Brandl g.bra...@gmx.net wrote: Am 31.01.2010 13:03, schrieb Simon Cross: On Sun, Jan 31, 2010 at 1:54 PM, Hanno Schlichting ha...@hannosch.eu wrote: I'd be a big +1 to using a single .pyr directory per source directory. I don't know whether I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread R. David Murray
On Sun, 31 Jan 2010 09:50:16 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= mar...@v.loewis.de wrote: Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala can install Python

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Antoine Pitrou
Le Sun, 31 Jan 2010 18:46:54 +1000, Nick Coghlan a écrit : Actually, this is the first post I've seen noting objective problems with the use of a subdirectory. The others were just a subjective difference in perspective that saw subdirectory clutter as somehow being worse than file clutter.

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Antoine Pitrou
Le Sat, 30 Jan 2010 21:04:14 -0800, Jeffrey Yasskin a écrit : I have a couple bikesheddy or why didn't you do this comments. I'll be perfectly satisfied with an answer or a line in the pep. 1. Why the -R flag? It seems like this is a uniform improvement, so it should be the default. Have

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Michael Crute
On Sun, Jan 31, 2010 at 5:00 AM, Martin v. Löwis mar...@v.loewis.de wrote: Agreed this should be discussed in the PEP, but one obvious problem is the speed impact. Picking up a file from a subdirectory is going to introduce less overhead than unpacking it from a zipfile. There is also the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread MRAB
Vitor Bosshard wrote: 2010/1/31 Georg Brandl g.bra...@gmx.net: foo.py foo.pyr/ cpython-25.pyc cpython-25U.pyc cpython-27.pyc cpython-27U.pyc cpython-32.pyc unladen-011.pyc wpython-11.pyc +1. It should be quite easy to assign a new name every time the magic number is updated.

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Reid Kleckner
On Sun, Jan 31, 2010 at 8:34 AM, Nick Coghlan ncogh...@gmail.com wrote: That still leaves the question of what to do with __file__ (for which even the solution in the PEP isn't particularly clean). Perhaps the thing to do there is to have __file__ always point to the source file and introduce

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
By the way, the part that caused me the most confusion in the language in the PEP was the emphasized *and their dependencies*, as if a package having dependencies somehow turned the problem into a factorial explosion. But there seems to be nothing special, according to your explanation, about

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Hanno Schlichting
On Sun, Jan 31, 2010 at 6:04 AM, Jeffrey Yasskin jyass...@gmail.com wrote: 1. Why the -R flag? It seems like this is a uniform improvement, so it should be the default. Have faith in your design! ;-) +1 for a single strategy that is used in all cases. The current solution could be phased out

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
This would eliminate the read-time race condition but still potentially allow for a write-time race condition if locking isn't used. The benefit of this approach is that it is no less clear than pyc is today and doesn't result in n * versions_of_python pyc files. There is still the overhead

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread MRAB
Reid Kleckner wrote: On Sun, Jan 31, 2010 at 8:34 AM, Nick Coghlan ncogh...@gmail.com wrote: That still leaves the question of what to do with __file__ (for which even the solution in the PEP isn't particularly clean). Perhaps the thing to do there is to have __file__ always point to the source

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Raymond Hettinger
On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: Abstract This PEP describes an extension to Python's import mechanism which improves sharing of Python source code files among multiple installed different versions of the Python interpreter. +1 It does this by allowing many

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Ron Adam
Sense this is something new, I believe it is helpful to look at all the possibilities so it doesn't become something we regret we did later. This is something that once it gets put in place may be real hard to get rid of. So here are a few questions that I think haven't seen asked yet.

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Silke von Bargen
Martin v. Löwis schrieb: There is also the issue of race conditions with multiple simultaneous accesses. The original format for the PEP had race conditions for multiple simultaneous writers; ZIP will also have race conditions for concurrent readers/writers (as any new writer will have to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Terry Reedy
On 1/31/2010 8:58 AM, Vitor Bosshard wrote: Having one single pyr (or__pycache__ or whatever it's called) subfolder per folder is an easy to understand, solid solution. As a user who browses directories to see what is there and to find files to open and look at, I would like this. The

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Guido van Rossum
Whoa. This thread already exploded. I'm picking this message to respond to because it reflects my own view after reading the PEP. On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Terry Reedy
On 1/31/2010 8:34 AM, Nick Coghlan wrote: Georg Brandl wrote: +1. Having a single (visible) __pyr__ directory is much less clutter than multiple .pyc files anyway. Also, don't forget Windows users, for whom the dot convention doesn't mean anything. I must admit I quite like the __pyr__

[Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Silke von Bargen
That still leaves the question of what to do with __file__ (for which even the solution in the PEP isn't particularly clean). Perhaps the thing to do there is to have __file__ always point to the source file and introduce a __file_cached__ that points to the bytecompiled file on disk (set to

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread MRAB
Silke von Bargen wrote: That still leaves the question of what to do with __file__ (for which even the solution in the PEP isn't particularly clean). Perhaps the thing to do there is to have __file__ always point to the source file and introduce a __file_cached__ that points to the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
foo.pyr/ cpython-25.pyc cpython-25U.pyc cpython-27.pyc cpython-27U.pyc cpython-32.pyc unladen-011.pyc wpython-11.pyc +1. It should be quite easy to assign a new name every time the magic number is updated. It would actually be possible to drop the magic numbers

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Exactly. How would you define where the pyr folder goes? At the root of a package? What if I delete the __init__.py file there? Will the existing pyr folder be orphaned and a new one created in each subfolder? Unlike VCS working copies, the package / module / script hierarchy is not formally

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
Not really -- much of the code I've seen that tries to guess the source file name from a __file__ value just does something like this: if fname.lower().endswith(('.pyc', '.pyo')): fname = fname[:-1] That's not compatible with using .pyr, either. If a single pyc folder is used, I think

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Tim Delaney
On 1 February 2010 00:34, Nick Coghlan ncogh...@gmail.com wrote: __file__ would always point to the source files __file_cached__ would always point to the relevant compiled file (either pre-existing or newly created) I like this solution combined with having a single cache directory and a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Neil Hodgson
Tim Delaney: I like this solution combined with having a single cache directory and a few other things I've added below. ... 2. /tmp is often on non-volatile memory. If it is (e.g. my Windows system temp dir is on a RAMdisk) then it seems wise to respect the obvious desire to throw away

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
I can also see advantages to allowing out of tree compiled cache directories. For example, you could have a locked down .py tree with .pycs going into per-user trees. This prevents another user from spoofing a .pyc I use as well as allowing users to install arbitrary versions of Python

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Scott Dial
On 1/31/2010 2:04 PM, Raymond Hettinger wrote: On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: It does this by allowing many different byte compilation files (.pyc files) to be co-located with the Python source file (.py file). It would be nice if all the compilation files could be

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Curt Hagenlocher
On Sun, Jan 31, 2010 at 11:16 AM, Terry Reedy tjre...@udel.edu wrote: 'pycache' would be pretty clear. Heh -- without the underscores, I read this as pyc ache. Seems appropriate. -- Curt Hagenlocher c...@hagenlocher.org ___ Python-Dev mailing list

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Terry Reedy
On 1/31/2010 4:26 PM, Tim Delaney wrote: The pyc/pyo files are just an optimisation detail, and are essentially temporary. The .pycs for /Lib and similar are*not* temporarily in the sense you are using. They are effectively permanent for as long as the version is installed. They should

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread R. David Murray
On Sun, 31 Jan 2010 19:48:19 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= mar...@v.loewis.de wrote: By the way, the part that caused me the most confusion in the language in the PEP was the emphasized *and their dependencies*, as if a package having dependencies somehow turned the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Antoine Pitrou wrote: Le Sat, 30 Jan 2010 21:04:14 -0800, Jeffrey Yasskin a écrit : I have a couple bikesheddy or why didn't you do this comments. I'll be perfectly satisfied with an answer or a line in the pep. 1. Why the -R flag? It seems like this is a uniform improvement, so it should be

[Python-Dev] subprocess docs patch

2010-01-31 Thread Chris Rebert
Hello mighty Python developers, I was wondering if someone could take a gander at, and hopefully act upon, a patch I submitted a while ago for the subprocess module's docs. It's been languishing in the bug tracker: http://bugs.python.org/issue6760 Any help you could provide would be

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Martin v. Löwis wrote: Exactly. How would you define where the pyr folder goes? At the root of a package? What if I delete the __init__.py file there? Will the existing pyr folder be orphaned and a new one created in each subfolder? Unlike VCS working copies, the package / module / script

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Silke von Bargen wrote: That still leaves the question of what to do with __file__ (for which even the solution in the PEP isn't particularly clean). Perhaps the thing to do there is to have __file__ always point to the source file and introduce a __file_cached__ that points to the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
But I don't understand how this answers the question. If the python26-zope.sendmail package doesn't run setup.py, then a python-zope.sendmail package where you specify at install time which directory to install the files to isn't going to run setup.py, either. If the only difference between

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Ben Finney
Nick Coghlan ncogh...@gmail.com writes: Would you still be a -1 on making it the new scheme the default if it used a single cache directory instead? That would actually be cleaner than the current solution rather than messier. +0 on a default of “store compiled bytecode files in a single