Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley 
> 
> Message: 1
> Date: Tue, 9 Aug 2011 15:31:47 +0600
> From:  ? 
> To: [email protected]
> Subject: [Python-Dev] GIL removal question
> Message-ID:
>   
> Content-Type: text/plain; charset=UTF-8
> 
> Probably I want to re-invent a bicycle. I want developers to say me
> why we can not remove GIL in that way:
> 
> 1. Remove GIL completely with all current logick.
> 2. Add it's own RW-locking to all mutable objects (like list or dict)
> 3. Add RW-locks to every context instance
> 4. use RW-locks when accessing members of object instances

You're forgetting step 5.

5. Put fine-grain locks around all reference counting operations (or rewrite 
all of Python's memory management and garbage collection from scratch).

> Only one reason, I see, not do that -- is performance of
> singlethreaded applications.

After implementing the aforementioned step 5, you will find that the 
performance of everything, including the threaded code, will be quite a bit 
worse.  Frankly, this is probably the most significant obstacle to have any 
kind of GIL-less Python with reasonable performance.

Just as an aside, I recently did some experiments with the fabled patch to 
remove the GIL from Python 1.4 (mainly for my own historical curiosity).   On 
Linux, the performance isn't just slightly worse, it makes single-threaded code 
run about 6-7 times slower and threaded code runs even worse.  So, basically 
everything runs like a dog.  No GIL though.

Cheers,
Dave

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Nick Coghlan 
On Wed, Aug 10, 2011 at 9:09 PM, David Beazley  wrote:
> You're forgetting step 5.
>
> 5. Put fine-grain locks around all reference counting operations (or rewrite 
> all of Python's memory management and garbage collection from scratch).
...
> After implementing the aforementioned step 5, you will find that the 
> performance of everything, including the threaded code, will be quite a bit 
> worse.  Frankly, this is probably the most significant obstacle to have any 
> kind of GIL-less Python with reasonable performance.

PyPy would actually make a significantly better basis for this kind of
experimentation, since they *don't* use reference counting for their
memory management.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread David Beazley 

On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote:

> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley  wrote:
>> You're forgetting step 5.
>> 
>> 5. Put fine-grain locks around all reference counting operations (or rewrite 
>> all of Python's memory management and garbage collection from scratch).
> ...
>> After implementing the aforementioned step 5, you will find that the 
>> performance of everything, including the threaded code, will be quite a bit 
>> worse.  Frankly, this is probably the most significant obstacle to have any 
>> kind of GIL-less Python with reasonable performance.
> 
> PyPy would actually make a significantly better basis for this kind of
> experimentation, since they *don't* use reference counting for their
> memory management.
> 

That's an experiment that would pretty interesting.  I think the real question 
would boil down to what *else* do they have to lock to make everything work.   
Reference counting is a huge bottleneck for CPython to be sure, but it's 
definitely not the only issue that has to be addressed in making a 
free-threaded Python.

Cheers,
Dave
  
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Nick Coghlan 
On Wed, Aug 10, 2011 at 9:32 PM, David Beazley  wrote:
> On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote:
>> PyPy would actually make a significantly better basis for this kind of
>> experimentation, since they *don't* use reference counting for their
>> memory management.
>
> That's an experiment that would pretty interesting.  I think the real 
> question would boil down to what *else* do they have to lock to make 
> everything work.   Reference counting is a huge bottleneck for CPython to be 
> sure, but it's definitely not the only issue that has to be addressed in 
> making a free-threaded Python.
>

Yeah, the problem reduces back to the 4 steps in the original post.
Still not trivial, since there's quite a bit of internal interpreter
state to protect, but significantly more feasible than dealing with
CPython's reference counting. However, you do get additional
complexities like the JIT compiler coming into play, so it is really a
question that would need to be raised directly with the PyPy dev team.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Guido van Rossum 
On Wed, Aug 10, 2011 at 7:32 AM, David Beazley  wrote:
>
> On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote:
>
>> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley  wrote:
>>> You're forgetting step 5.
>>>
>>> 5. Put fine-grain locks around all reference counting operations (or 
>>> rewrite all of Python's memory management and garbage collection from 
>>> scratch).
>> ...
>>> After implementing the aforementioned step 5, you will find that the 
>>> performance of everything, including the threaded code, will be quite a bit 
>>> worse.  Frankly, this is probably the most significant obstacle to have any 
>>> kind of GIL-less Python with reasonable performance.
>>
>> PyPy would actually make a significantly better basis for this kind of
>> experimentation, since they *don't* use reference counting for their
>> memory management.
>>
>
> That's an experiment that would pretty interesting.  I think the real 
> question would boil down to what *else* do they have to lock to make 
> everything work.   Reference counting is a huge bottleneck for CPython to be 
> sure, but it's definitely not the only issue that has to be addressed in 
> making a free-threaded Python.

They have a specific plan, based on Software Transactional Memory:
http://morepypy.blogspot.com/2011/06/global-interpreter-lock-or-how-to-kill.html

Personally, I'm not holding my breath, because STM in other areas has
so far captured many imaginations without bringing practical results
(I keep hearing about it as this promising theory that needs more work
to implement, sort-of like String Theory in theoretical physics).

But I'm also not denying that Armin Rigo has a brain the size of the
planet, and PyPy *has* already made much real, practical progress.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski 
On Wed, Aug 10, 2011 at 1:43 PM, Guido van Rossum  wrote:
> On Wed, Aug 10, 2011 at 7:32 AM, David Beazley  wrote:
>>
>> On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote:
>>
>>> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley  wrote:
 You're forgetting step 5.

 5. Put fine-grain locks around all reference counting operations (or 
 rewrite all of Python's memory management and garbage collection from 
 scratch).
>>> ...
 After implementing the aforementioned step 5, you will find that the 
 performance of everything, including the threaded code, will be quite a 
 bit worse.  Frankly, this is probably the most significant obstacle to 
 have any kind of GIL-less Python with reasonable performance.
>>>
>>> PyPy would actually make a significantly better basis for this kind of
>>> experimentation, since they *don't* use reference counting for their
>>> memory management.
>>>
>>
>> That's an experiment that would pretty interesting.  I think the real 
>> question would boil down to what *else* do they have to lock to make 
>> everything work.   Reference counting is a huge bottleneck for CPython to be 
>> sure, but it's definitely not the only issue that has to be addressed in 
>> making a free-threaded Python.
>
> They have a specific plan, based on Software Transactional Memory:
> http://morepypy.blogspot.com/2011/06/global-interpreter-lock-or-how-to-kill.html
>
> Personally, I'm not holding my breath, because STM in other areas has
> so far captured many imaginations without bringing practical results
> (I keep hearing about it as this promising theory that needs more work
> to implement, sort-of like String Theory in theoretical physics).

Note that the PyPy's plan does *not* assume the end result will be
comparable in the single-threaded case. The goal is to be able to
compile two *different* pypy's, one fast single-threaded, one
gil-less, but with a significant overhead. The trick is to get this
working in a way that does not increase maintenance burden. It's also
research, so among other things it might not work.

Cheers,
fijal
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Vlad Riscutia 
Removing GIL is interesting work and probably multiple people are willing to
contribute. Threading and synchronization is a deep topic and it might be
that if just one person toys around with removing GIL he might not see
performance improvement (not meaning to offend anyone who tried this,
honestly) but what about forking a branch for this work, with some good
benchmarks in place and have community contribute? Let's say first step
would be just replacing GIL with some fine grained locks with expected
performance degradation but afterwards we can try to incrementally improve
on this.

Thank you,
Vlad

On Wed, Aug 10, 2011 at 8:20 AM, Maciej Fijalkowski wrote:

> On Wed, Aug 10, 2011 at 1:43 PM, Guido van Rossum 
> wrote:
> > On Wed, Aug 10, 2011 at 7:32 AM, David Beazley  wrote:
> >>
> >> On Aug 10, 2011, at 6:15 AM, Nick Coghlan wrote:
> >>
> >>> On Wed, Aug 10, 2011 at 9:09 PM, David Beazley 
> wrote:
>  You're forgetting step 5.
> 
>  5. Put fine-grain locks around all reference counting operations (or
> rewrite all of Python's memory management and garbage collection from
> scratch).
> >>> ...
>  After implementing the aforementioned step 5, you will find that the
> performance of everything, including the threaded code, will be quite a bit
> worse.  Frankly, this is probably the most significant obstacle to have any
> kind of GIL-less Python with reasonable performance.
> >>>
> >>> PyPy would actually make a significantly better basis for this kind of
> >>> experimentation, since they *don't* use reference counting for their
> >>> memory management.
> >>>
> >>
> >> That's an experiment that would pretty interesting.  I think the real
> question would boil down to what *else* do they have to lock to make
> everything work.   Reference counting is a huge bottleneck for CPython to be
> sure, but it's definitely not the only issue that has to be addressed in
> making a free-threaded Python.
> >
> > They have a specific plan, based on Software Transactional Memory:
> >
> http://morepypy.blogspot.com/2011/06/global-interpreter-lock-or-how-to-kill.html
> >
> > Personally, I'm not holding my breath, because STM in other areas has
> > so far captured many imaginations without bringing practical results
> > (I keep hearing about it as this promising theory that needs more work
> > to implement, sort-of like String Theory in theoretical physics).
>
> Note that the PyPy's plan does *not* assume the end result will be
> comparable in the single-threaded case. The goal is to be able to
> compile two *different* pypy's, one fast single-threaded, one
> gil-less, but with a significant overhead. The trick is to get this
> working in a way that does not increase maintenance burden. It's also
> research, so among other things it might not work.
>
> Cheers,
> fijal
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/riscutiavlad%40gmail.com
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Brian Curtin 
On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia  wrote:

> Removing GIL is interesting work and probably multiple people are willing
> to contribute. Threading and synchronization is a deep topic and it might be
> that if just one person toys around with removing GIL he might not see
> performance improvement (not meaning to offend anyone who tried this,
> honestly) but what about forking a branch for this work, with some good
> benchmarks in place and have community contribute? Let's say first step
> would be just replacing GIL with some fine grained locks with expected
> performance degradation but afterwards we can try to incrementally improve
> on this.
>
> Thank you,
> Vlad
>

Feel free to start this: http://hg.python.org/cpython
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Eric Snow 
On Wed, Aug 10, 2011 at 10:19 AM, Brian Curtin  wrote:
> On Wed, Aug 10, 2011 at 11:14, Vlad Riscutia  wrote:
>>
>> Removing GIL is interesting work and probably multiple people are willing
>> to contribute. Threading and synchronization is a deep topic and it might be
>> that if just one person toys around with removing GIL he might not see
>> performance improvement (not meaning to offend anyone who tried this,
>> honestly) but what about forking a branch for this work, with some good
>> benchmarks in place and have community contribute? Let's say first step
>> would be just replacing GIL with some fine grained locks with expected
>> performance degradation but afterwards we can try to incrementally improve
>> on this.
>> Thank you,
>> Vlad
>
> Feel free to start this: http://hg.python.org/cpython

+1 on not waiting for someone else to do it if you have an idea. :)

Bitbucket makes it really easy for anyone to fork a repo into a new
project and they keep an up to date mirror of the CPython repo:

https://bitbucket.org/mirror/cpython/overview

-eric

> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Raymond Hettinger 

On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote:

>> After implementing the aforementioned step 5, you will find that the 
>> performance of everything, including the threaded code, will be quite a bit 
>> worse.  Frankly, this is probably the most significant obstacle to have any 
>> kind of GIL-less Python with reasonable performance.
> 
> PyPy would actually make a significantly better basis for this kind of
> experimentation, since they *don't* use reference counting for their
> memory management.

Jython may be a better choice.  It is all about concurrency.  Its dicts are 
built on top of Java's ConcurrentHashMap for example.



Raymond___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Brian Curtin 
Now that we have concurrent.futures, is there any plan for multiprocessing
to follow suit? PEP 3148 mentions a hope to add or move things in the future
[0], which would be now.

[0] http://www.python.org/dev/peps/pep-3148/#naming
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Benjamin Peterson 
2011/8/10 Brian Curtin :
> Now that we have concurrent.futures, is there any plan for multiprocessing
> to follow suit? PEP 3148 mentions a hope to add or move things in the future

Is there some sort of concrete proposal? The PEP just seems to mention
it as an idea.

In general, -1. I think we don't need to be moving things around more
to little advantage.


-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

2011-08-10 Thread Maciej Fijalkowski 
On Wed, Aug 10, 2011 at 7:19 PM, Raymond Hettinger
 wrote:
>
> On Aug 10, 2011, at 4:15 AM, Nick Coghlan wrote:
>
> After implementing the aforementioned step 5, you will find that the
> performance of everything, including the threaded code, will be quite a bit
> worse.  Frankly, this is probably the most significant obstacle to have any
> kind of GIL-less Python with reasonable performance.
>
> PyPy would actually make a significantly better basis for this kind of
> experimentation, since they *don't* use reference counting for their
> memory management.
>
> Jython may be a better choice.  It is all about concurrency.  Its dicts are
> built on top of Java's ConcurrentHashMap for example.
>

Jython is kind of boring choice because it does not have a GIL at all
(same as IronPython). It might *work* for what you're trying to
achieve but GIL-removal is not really that interesting.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Antoine Pitrou 
Le Wed, 10 Aug 2011 14:54:33 -0500,
Benjamin Peterson  a écrit :
> 2011/8/10 Brian Curtin :
> > Now that we have concurrent.futures, is there any plan for
> > multiprocessing to follow suit? PEP 3148 mentions a hope to add or move
> > things in the future
> 
> Is there some sort of concrete proposal? The PEP just seems to mention
> it as an idea.
> 
> In general, -1. I think we don't need to be moving things around more
> to little advantage.

Agreed. Also, flat is better than nested. Whoever wants to populate the
concurrent package should work on new features to be added to it, rather
than plans to rename things around.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Brian Curtin 
On Wed, Aug 10, 2011 at 15:36, Antoine Pitrou  wrote:

> Le Wed, 10 Aug 2011 14:54:33 -0500,
> Benjamin Peterson  a écrit :
> > 2011/8/10 Brian Curtin :
> > > Now that we have concurrent.futures, is there any plan for
> > > multiprocessing to follow suit? PEP 3148 mentions a hope to add or move
> > > things in the future
> >
> > Is there some sort of concrete proposal? The PEP just seems to mention
> > it as an idea.
> >
> > In general, -1. I think we don't need to be moving things around more
> > to little advantage.
>
> Agreed. Also, flat is better than nested. Whoever wants to populate the
> concurrent package should work on new features to be added to it, rather
> than plans to rename things around.


I agree with flat being better than nested and won't be pushing to move
things around, but the creation of the concurrent package seemed like a
place to put those things. I just found myself typing
"concurrent.multiprocessing" a minute ago, so I figured I'd put it out
there.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Raymond Hettinger 

On Aug 10, 2011, at 1:36 PM, Antoine Pitrou wrote:

> Le Wed, 10 Aug 2011 14:54:33 -0500,
> Benjamin Peterson  a écrit :
>> 2011/8/10 Brian Curtin :
>>> Now that we have concurrent.futures, is there any plan for
>>> multiprocessing to follow suit? PEP 3148 mentions a hope to add or move
>>> things in the future
>> 
>> Is there some sort of concrete proposal? The PEP just seems to mention
>> it as an idea.
>> 
>> In general, -1. I think we don't need to be moving things around more
>> to little advantage.
> 
> Agreed. Also, flat is better than nested. Whoever wants to populate the
> concurrent package should work on new features to be added to it, rather
> than plans to rename things around.

I concur.


Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the

2011-08-10 Thread Sandro Tosi 
On Wed, Aug 10, 2011 at 21:55, Terry Reedy  wrote:
>
>>
>>     Latest version of the `heapq Python source code
>>
>> -`_
>>
>> +`_
>
> Should links be to the hg repository instead of svn?
> Is svn updated from hg?
> I thought is was (mostly) historical read-only.

I made the same remark to Senthil on IRC, and came out that web
frontend for hg.p.o doesn't allow for a nice way to specify a branch
(different than 'default'), it's something like
hg.python.org/cpython//path/to/file.py
which is almost always outdated :)

What do we use to provide the web part of hg.p.o? maybe we can just
ask the developers of this tool to provide (or advertize) a proper way
to select a branch. If some can provide me some info, I can do the
"ask the devs" part.

Cheers,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the

2011-08-10 Thread Ezio Melotti 

On 11/08/2011 0.02, Sandro Tosi wrote:

On Wed, Aug 10, 2011 at 21:55, Terry Reedy  wrote:

 Latest version of the `heapq Python source code

-`_

+`_

Should links be to the hg repository instead of svn?
Is svn updated from hg?
I thought is was (mostly) historical read-only.

I made the same remark to Senthil on IRC, and came out that web
frontend for hg.p.o doesn't allow for a nice way to specify a branch
(different than 'default'), it's something like
hg.python.org/cpython//path/to/file.py
which is almost always outdated :)


hg.python.org/cpython/2.7/path/to/file.py should work just fine.

IIRC the reason why we don't do it on 2.x is because we don't have the 
'source' directive available in Sphinx and therefore we would have to 
update all the links manually to link to h.p.o instead of s.p.o.


Best Regards,
Ezio Melotti



What do we use to provide the web part of hg.p.o? maybe we can just
ask the developers of this tool to provide (or advertize) a proper way
to select a branch. If some can provide me some info, I can do the
"ask the devs" part.

Cheers,


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Jesse Noller 
On Wed, Aug 10, 2011 at 4:45 PM, Brian Curtin  wrote:
> On Wed, Aug 10, 2011 at 15:36, Antoine Pitrou  wrote:
>>
>> Le Wed, 10 Aug 2011 14:54:33 -0500,
>> Benjamin Peterson  a écrit :
>> > 2011/8/10 Brian Curtin :
>> > > Now that we have concurrent.futures, is there any plan for
>> > > multiprocessing to follow suit? PEP 3148 mentions a hope to add or
>> > > move
>> > > things in the future
>> >
>> > Is there some sort of concrete proposal? The PEP just seems to mention
>> > it as an idea.
>> >
>> > In general, -1. I think we don't need to be moving things around more
>> > to little advantage.
>>
>> Agreed. Also, flat is better than nested. Whoever wants to populate the
>> concurrent package should work on new features to be added to it, rather
>> than plans to rename things around.
>
> I agree with flat being better than nested and won't be pushing to move
> things around, but the creation of the concurrent package seemed like a
> place to put those things. I just found myself typing
> "concurrent.multiprocessing" a minute ago, so I figured I'd put it out
> there.

I would like to move certain *features* of multiprocessing into that
namespace - some things like map and others don't belong in the
multiprocessing namespace, and should have been put into concurrent.*
a long time ago.

As for my plans: I had intended on making multiprocessing a closer
corollary to threading, and moving the bigger features that should
have been broken out into a different package (such as
http://bugs.python.org/issue12708) and the managers.

Those plans are obviously stalled as my time is being spent elsewhere.
I disagree on the "flat is better than nested" point -
multiprocessing's namespace is flat - but bloated, and many of it's
features could work just as well in a threaded context (e.g, they are
generally useful outside of multiprocessing alone).

Regardless; currently I can't lead this, and multiprocessing-sig is silent.

Jesse
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Benjamin Peterson 
2011/8/10 Raymond Hettinger :
>
> On Aug 10, 2011, at 1:36 PM, Antoine Pitrou wrote:
>
>> Le Wed, 10 Aug 2011 14:54:33 -0500,
>> Benjamin Peterson  a écrit :
>>> 2011/8/10 Brian Curtin :
 Now that we have concurrent.futures, is there any plan for
 multiprocessing to follow suit? PEP 3148 mentions a hope to add or move
 things in the future
>>>
>>> Is there some sort of concrete proposal? The PEP just seems to mention
>>> it as an idea.
>>>
>>> In general, -1. I think we don't need to be moving things around more
>>> to little advantage.
>>
>> Agreed. Also, flat is better than nested. Whoever wants to populate the
>> concurrent package should work on new features to be added to it, rather
>> than plans to rename things around.
>
> I concur.

So we could put yourself, Antoine, and me in the concurrent package. :)

Sorry,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Nick Coghlan 
On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin  wrote:
> Now that we have concurrent.futures, is there any plan for multiprocessing
> to follow suit? PEP 3148 mentions a hope to add or move things in the future
> [0], which would be now.

As Jesse said, moving multiprocessing or threading wholesale was never
part of the plan. The main motivator of that comment in PEP 3148 was
the idea of creating 'concurrent.pool', which would provide a
concurrent worker pool API modelled on multiprocessing.Pool that
supported either threads or processes as the back end, just like the
executor model in concurrent.futures.

The basic approach is to look at a feature in threading or
multiprocessing that is only available in one of them and ask the
question: Does it make sense to allow a project to switch easily
between a threading strategy and a multiprocessing strategy when using
this feature?

If the answer to that question is yes (as it was for
concurrent.futures itself, and as I believe it to be for
multiprocessing.Pool), then a feature request (and probably a PEP)
proposing the definition of a common API in the concurrent namespace
would be appropriate.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Moving forward with the concurrent package

2011-08-10 Thread Jesse Noller 
On Wed, Aug 10, 2011 at 7:03 PM, Nick Coghlan  wrote:
> On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin  wrote:
>> Now that we have concurrent.futures, is there any plan for multiprocessing
>> to follow suit? PEP 3148 mentions a hope to add or move things in the future
>> [0], which would be now.
>
> As Jesse said, moving multiprocessing or threading wholesale was never
> part of the plan. The main motivator of that comment in PEP 3148 was
> the idea of creating 'concurrent.pool', which would provide a
> concurrent worker pool API modelled on multiprocessing.Pool that
> supported either threads or processes as the back end, just like the
> executor model in concurrent.futures.
>
> The basic approach is to look at a feature in threading or
> multiprocessing that is only available in one of them and ask the
> question: Does it make sense to allow a project to switch easily
> between a threading strategy and a multiprocessing strategy when using
> this feature?
>
> If the answer to that question is yes (as it was for
> concurrent.futures itself, and as I believe it to be for
> multiprocessing.Pool), then a feature request (and probably a PEP)
> proposing the definition of a common API in the concurrent namespace
> would be appropriate.
>

Precisely. Thank you Nick, want a job working for PyCon? ;)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the

2011-08-10 Thread Senthil Kumaran 
On Wed, Aug 10, 2011 at 11:58:24PM +0300, Ezio Melotti wrote:
> 
> hg.python.org/cpython/2.7/path/to/file.py should work just fine.

The correct path seems to be:

http://hg.python.org/cpython/file/2.7/Lib/
> 
> IIRC the reason why we don't do it on 2.x is because we don't have
> the 'source' directive available in Sphinx and therefore we would
> have to update all the links manually to link to h.p.o instead of
> s.p.o.

I see.  Does sphinx have any such directive already? How is it
supposed to behave?

-- 
Senthil
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the

2011-08-10 Thread Raymond Hettinger 

On Aug 10, 2011, at 1:58 PM, Ezio Melotti wrote:

> we would have to update all the links manually to link to h.p.o instead of 
> s.p.o.

sed is your friend.


Raymond

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: News item for #12724

2011-08-10 Thread Georg Brandl 
Am 11.08.2011 03:34, schrieb brian.curtin:
> http://hg.python.org/cpython/rev/3a6782f2a4a8
> changeset:   71811:3a6782f2a4a8
> user:Brian Curtin 
> date:Wed Aug 10 20:32:10 2011 -0500
> summary:
>   News item for #12724
> 
> files:
>   Misc/NEWS |  2 ++
>   1 files changed, 2 insertions(+), 0 deletions(-)

If it gets a NEWS entry, why isn't it documented?

Georg

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com