Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread M.-A. Lemburg
Collin Winter wrote: On Mon, Feb 1, 2010 at 11:17 AM, M.-A. Lemburg m...@egenix.com wrote: Collin Winter wrote: I think this idea underestimates a) how deeply the current CPython VM is intertwined with the rest of the implementation, and b) the nature of the changes required by these separate

Re: [Python-Dev] Reworking the GIL

2010-02-02 Thread Eric Hopper
I recently saw the video of David Beazley's presentation on how poorly the old GIL implementation handled certain cases and thought I can fix that!. Unfortunately for me, someone else has beaten me to it, and done a somewhat better job than I would've because I wasn't thinking of doing anything

[Python-Dev] Mail Archives

2010-02-02 Thread Ben.Young
Hi, Currently the Python-Dev archives at http://mail.python.org/pipermail/python-dev/ doesn't appear to have been updated for the last week or so. Is that a known issue? Thanks, Ben Ben Young - Senior Software Engineer SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR Tel +44

Re: [Python-Dev] Mail Archives

2010-02-02 Thread Chris Rebert
On Tue, Feb 2, 2010 at 1:50 AM, ben.yo...@sungard.com wrote: Hi, Currently the Python-Dev archives at http://mail.python.org/pipermail/python-dev/ doesn't appear to have been updated for the last week or so. Is that a known issue? I think so:

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Pascal Chambon
Although I would be in favor of an atfork callback registration system (similar to atexit), it seems there is no way to solve the fork() problem automatically with this. Any attempt to acquire/release locks automatically will lead to deadlocks, as it is necessary to know the exact program

Re: [Python-Dev] Reworking the GIL

2010-02-02 Thread Neil Hodgson
Eric Hopper: I don't suppose it will ever be ported back to Python 2.x?  It doesn't look like the whole GIL concept has changed much between Python 2.x and 3.x so I expect back-porting it would be pretty easy. There was a patch but it has been rejected. http://bugs.python.org/issue7753

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Nick Coghlan
M.-A. Lemburg wrote: BTW: I also doubt that Mercurial will make any of this easier. It makes creating branches easier for non-committers, but the problem of having to maintain the branches remains. It greatly simplifies the process of syncing the branch with the main line of development so

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Nick Coghlan
Pascal Chambon wrote: I guess spawnl semantic (i.e, like win32's CreateProcess()) can't become the default multiprocessing behaviour, as too many programs implicitly rely on the whole sharing of data under unix (and py3k itself is maybe becoming a little too mature for new compatility breaks)

Re: [Python-Dev] Mail Archives

2010-02-02 Thread Nick Coghlan
Chris Rebert wrote: On Tue, Feb 2, 2010 at 1:50 AM, ben.yo...@sungard.com wrote: Hi, Currently the Python-Dev archives at http://mail.python.org/pipermail/python-dev/ doesn't appear to have been updated for the last week or so. Is that a known issue? I think so:

[Python-Dev] Absolute imports in Python 2.x?

2010-02-02 Thread Mark Dickinson
What are the current plans for PEP 328 (the absolute imports PEP) in Python 2.x? The PEP says: In Python 2.6, any import statement that results in an intra-package import will raise DeprecationWarning (this also applies to from import that fails to use the relative import syntax). In Python

Re: [Python-Dev] Reworking the GIL

2010-02-02 Thread Antoine Pitrou
Hello Eric, I notice that Guido vetoed this idea, but just in case it should come up again, I have some thoughts that likely have already occurred to people, but I didn't notice on the list. Yes, I hindsight I think Guido was right. (no, I'm not trying to make myself agreeable :-)) Also,

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread skip
I guess spawnl semantic (i.e, like win32's CreateProcess()) can't become the default multiprocessing behaviour... Nick It would also make it much easier to write cross-platform Nick multiprocessing code (by always using the non-forking semantics Nick even on fork-capable

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Antoine Pitrou
skip at pobox.com writes: I don't understand. On Unix-y systems isn't spawn* layered on top of fork/exec? The point is that exec() relinquishes all the existing resources, so the initial fork() becomes an implementation detail (IIUC). If you were going to combine both threading and

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Antoine Pitrou
Tres Seaver tseaver at palladion.com writes: Note that the we in your sentence is not anything like the quod semper quod ubique quod ab omnibus criterion for accepting dogma: mutliprocessing is a tool, and needs to be used according to its nature, just as with threading. I don't remember

Re: [Python-Dev] Reworking the GIL

2010-02-02 Thread Larry Hastings
Antoine Pitrou wrote: Also, the MSDN doc (*) says timeBeginPeriod() can have a detrimental effect on system performance; I don't know how much of it is true. (*) http://msdn.microsoft.com/en-us/library/dd757624(VS.85).aspx Indeed it does. This is ancient, dusty wisdom, from the days of 50mhz

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Antoine Pitrou wrote: Tres Seaver tseaver at palladion.com writes: Note that the we in your sentence is not anything like the quod semper quod ubique quod ab omnibus criterion for accepting dogma: mutliprocessing is a tool, and needs to be used

Re: [Python-Dev] Absolute imports in Python 2.x?

2010-02-02 Thread Eric Smith
Mark Dickinson wrote: What are the current plans for PEP 328 (the absolute imports PEP) in Python 2.x? The PEP says: In Python 2.6, any import statement that results in an intra-package import will raise DeprecationWarning (this also applies to from import that fails to use the relative

Re: [Python-Dev] Reworking the GIL

2010-02-02 Thread Stephen J. Turnbull
Antoine Pitrou writes: Yes, I hindsight I think Guido was right. Guido does too. One of the benefits of having a time machine is getting to turn your hindsight into foresight.wink ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Larry Hastings
On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross hodgestar+python...@gmail.com wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called __pyr__ rather than .pyr. Guido van Rossum wrote: Exactly what I

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread skip
Tres Some applications may seem to work when violating this rule, but Tres their developers are doomed to hair loss over time. Then for us bald guys it should be okay, right? ;-) Skip ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 s...@pobox.com wrote: Tres Some applications may seem to work when violating this rule, but Tres their developers are doomed to hair loss over time. Then for us bald guys it should be okay, right? ;-) Sure: you might even grow hair

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Pascal Chambon
The word dogma is a good one in this context however. We ( ;-)) have accepted and promoted the dogma that multiprocessing is the solution to parallelism in the face of the GIL. While it needn't be applicable in any and every situation, we should make it so that it is applicable often enough.

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Jesse Noller
On Tue, Feb 2, 2010 at 10:34 AM, Pascal Chambon chambon.pas...@gmail.com wrote: The word dogma is a good one in this context however. We ( ;-)) have accepted and promoted the dogma that multiprocessing is the solution to parallelism in the face of the GIL. While it needn't be applicable in

Re: [Python-Dev] Mecurial (was: PEP 3146: Merge Unladen Swallow into CPython)

2010-02-02 Thread M.-A. Lemburg
Nick Coghlan wrote: M.-A. Lemburg wrote: BTW: I also doubt that Mercurial will make any of this easier. It makes creating branches easier for non-committers, but the problem of having to maintain the branches remains. It greatly simplifies the process of syncing the branch with the main

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Nir Aides
Seems the problem under discussion is already taken care of in Python. Possibly remains to verify that the logic described below does not possibly generate deadlocks. From the Python docs: http://docs.python.org/c-api/init.html Another important thing to note about threads is their behaviour in

[Python-Dev] Python 2.6.5

2010-02-02 Thread Barry Warsaw
I'm thinking about doing a Python 2.6.5 release soon. I've added the following dates to the Python release schedule Google calendar: 2009-03-01 Python 2.6.5 rc 1 2009-03-15 Python 2.6.5 final This allows us to spend some time on 2.6.5 at Pycon if we want. Please let me know if you have any

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Martin v. Löwis
Le lundi 01 février 2010 à 23:58 +0100, Martin v. Löwis a écrit : Interestingly, the POSIX pthread_atfork documentation defines how you are supposed to do that: create an atfork handler set, and acquire all mutexes in the prepare handler. Then fork, and have the parent and child handlers

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Martin v. Löwis
Although I would be in favor of an atfork callback registration system (similar to atexit), it seems there is no way to solve the fork() problem automatically with this. Any attempt to acquire/release locks automatically will lead to deadlocks, as it is necessary to know the exact program

[Python-Dev] API for VM branches (was: PEP 3146)

2010-02-02 Thread Collin Winter
[Moving to python-ideas; python-dev to bcc] On Tue, Feb 2, 2010 at 2:02 AM, M.-A. Lemburg m...@egenix.com wrote: Collin Winter wrote: [snip] If such a restrictive plugin-based scheme had been available when we began Unladen Swallow, I do not doubt that we would have ignored it entirely. I do

Re: [Python-Dev] Absolute imports in Python 2.x?

2010-02-02 Thread Brett Cannon
On Tue, Feb 2, 2010 at 05:20, Eric Smith e...@trueblade.com wrote: Mark Dickinson wrote: What are the current plans for PEP 328 (the absolute imports PEP) in Python 2.x? The PEP says: In Python 2.6, any import statement that results in an intra-package import will raise DeprecationWarning

Re: [Python-Dev] Python 2.6.5

2010-02-02 Thread Raymond Hettinger
+1 On Feb 2, 2010, at 10:08 AM, Barry Warsaw wrote: I'm thinking about doing a Python 2.6.5 release soon. I've added the following dates to the Python release schedule Google calendar: 2009-03-01 Python 2.6.5 rc 1 2009-03-15 Python 2.6.5 final This allows us to spend some time on 2.6.5

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Collin Winter
Hey Dirkjan, [Circling back to this part of the thread] On Thu, Jan 21, 2010 at 1:37 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Thu, Jan 21, 2010 at 21:14, Collin Winter collinwin...@google.com wrote: [snip] My quick take on Cython and Shedskin is that they are useful-but-limited

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Sebastian Rittau
On Sun, Jan 31, 2010 at 12:44:33PM +0100, Georg Brandl wrote: At least to me, this does not explain why an unwanted (why unwanted? If it's unwanted, set PYTHONDONTWRITEBYTECODE=1) directory is worse than an unwanted file. A directory feels different than. For example, typing ls in my shell

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Neil Schemenauer
Nick Coghlan ncogh...@gmail.com wrote: Henning von Bargen wrote: The solution is so obvious: Why not use a .pyr file that is internally a zip file? I think a Zip file might be the right approach too. Either you could have directories in the zip file for each version, e.g. 2.7/foo.pyc

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Craig Citro
Done. The diff is at http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed Cython, Shedskin and a bunch of other alternatives to pure CPython. Some of that information is based on conversations I've had with the respective developers, and I'd appreciate corrections if I'm out of

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Glenn Linderman
On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are way to complex to be writing. Agreed. But in reading that, it somehow triggered a question: does zipimport only work for zipfiles, or does it work for any archive format

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Collin Winter
Hey MA, On Fri, Jan 29, 2010 at 11:14 AM, M.-A. Lemburg m...@egenix.com wrote: Collin Winter wrote: I added startup benchmarks for Mercurial and Bazaar yesterday (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we can use them as more macro-ish benchmarks, rather than merely

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Reid Kleckner
On Tue, Feb 2, 2010 at 8:57 PM, Collin Winter collinwin...@google.com wrote: Wouldn't it be possible to have the compiler approach work in three phases in order to reduce the memory footprint and startup time hit, ie.  1. run an instrumented Python interpreter to collect all    the needed

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Guido van Rossum
On Tue, Feb 2, 2010 at 5:41 PM, Glenn Linderman v+pyt...@g.nevcal.com wrote: On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are way to complex to be writing. Agreed.  But in reading that, it somehow triggered a question:

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Bob Ippolito
On Sun, Jan 31, 2010 at 11:16 AM, Guido van Rossum gu...@python.org wrote: Whoa. This thread already exploded. I'm picking this message to respond to because it reflects my own view after reading the PEP. On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
I have to say up front that I'm somewhat shocked at how quickly this thread has exploded! Since I'm sprinting this week, I haven't thoroughly read every message and won't have time tonight to answer every question, but I'll try to pick out some common ideas. I really appreciate everyone's input

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 30, 2010, at 11:21 PM, Vitor Bosshard wrote: Why not: foo.py foo.pyc # 2.7 or 3.2 foo.27.pyc foo.32.pyc etc. Because this clutters the module's directory more than it does today, which I considered to be a negative factor. And as others have pointed out, there isn't a one-to-one

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 03:07 PM, Ben Finney wrote: In other words, my understanding is that the current PEP would have the following tree for an example project:: foo/ __init__.py __init__.pyr/ deadbeef.pyc decafbad.pyc lorem.py lorem.pyr/

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 12:36 PM, Georg Brandl wrote: Not really -- much of the code I've seen that tries to guess the source file name from a __file__ value just does something like this: if fname.lower().endswith(('.pyc', '.pyo')): fname = fname[:-1] That's not compatible with using .pyr,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 09:30 PM, Martin v. Löwis wrote: If a single pyc folder is used, I think an additional __source__ attribute would be needed to indicate what source file time stamp had been checked (if any) to determine that the byte code file is current. This is a good point. __file__ is

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw ba...@python.org writes: I suppose this is going to be very subjective, but in skimming the thread it seems like most people like putting the byte code cache artifacts in subdirectories (be they siblings or folder-per-folder). I don't understand the distinction you're making

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw ba...@python.org writes: If you have to manually blow away a particular pyc file, folder-per-folder makes it much easier to find exactly what you want to blow away without have to search up the file system, and then back down again to find the pyc file to delete. How many ..'s

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-02 Thread Dirkjan Ochtman
On Tue, Feb 2, 2010 at 23:54, Collin Winter collinwin...@google.com wrote: Done. The diff is at http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed Cython, Shedskin and a bunch of other alternatives to pure CPython. Some of that information is based on conversations I've had