Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On Mon, Jul 12, 2010 at 02:13, Jesse Noller jnol...@gmail.com wrote: On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum gu...@python.org wrote: [snip] Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) So, after some cool down - and the last rounds of discussion which triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148 futures - execute computations asynchronously. I feel that it's a good addition, and a good start for something bigger down the road. Brian - you'll need to provide someone such as Martin or Georg your public key for ssh access into SVN, and you'll need developer access to the bug tracker. Brian, did you ever get your keys to Martin or Georg? If not please do (or send it to me) and let us know what your bugs.python.org username is to get Developer privileges. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
Hi I learned about the futures PEP only today. I saw the example on
http://code.google.com/p/pythonfutures/
One thing that worries me is that this approach seems to bypass the
usual exception handling mechanism of Python. In particular I'm
wondering why you have to do things like:
if future.exception() is not None:
...
This reminds me a lot of how things are done in C but it's not very
pythonic. Wouldn't it be possible and nicer to raise the exception --
if there was one inside the asynchronous job -- when the result of the
future is accessed?
try:
print('%r page is %d bytes' % (url, len(future.result(
except FutureError:
print('%r generated an exception: %s' % (url, future.exception()))
Best,
Titus
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On Tue, Jul 13, 2010 at 12:19 AM, Titus von der Malsburg malsb...@gmail.com wrote: This reminds me a lot of how things are done in C but it's not very pythonic. Wouldn't it be possible and nicer to raise the exception -- if there was one inside the asynchronous job -- when the result of the future is accessed? That's what actually happens, so you can code it either way (either just calling result() and dealing with any exception that was raised, or else checking for a non-None exception value directly). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On Tue, Jul 13, 2010 at 12:48:35AM +1000, Nick Coghlan wrote: On Tue, Jul 13, 2010 at 12:19 AM, Titus von der Malsburg That's what actually happens, so you can code it either way That's great! None of the examples I found used the pythonic exception style, that's why I assumed that checking the return value is the only possibility. Reading the PEP carefully would have helped. :-) Titus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On 13 Jul 2010, at 00:59, Titus von der Malsburg wrote:
On Tue, Jul 13, 2010 at 12:48:35AM +1000, Nick Coghlan wrote:
On Tue, Jul 13, 2010 at 12:19 AM, Titus von der Malsburg
That's what actually happens, so you can code it either way
That's great! None of the examples I found used the pythonic
exception style, that's why I assumed that checking the return value
is the only possibility. Reading the PEP carefully would have helped.
:-)
I'd add that it would feel more natural to me to write:
try:
print('%r page is %d bytes' % (url, len(future.result(
- except FutureError:
- print('%r generated an exception: %s' % (url, future.exception()))
+ except FutureError as e:
+ print('%r generated an exception: %s' % (url, e))
Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
Titus von der Malsburg wrote: None of the examples I found used the pythonic exception style, that's why I assumed that checking the return value is the only possibility. Reading the PEP carefully would have helped. :-) I had to read the pep fairly carefully before I noticed this too, so perhaps it could be made more prominent. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum gu...@python.org wrote: [snip] Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) So, after some cool down - and the last rounds of discussion which triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148 futures - execute computations asynchronously. I feel that it's a good addition, and a good start for something bigger down the road. Brian - you'll need to provide someone such as Martin or Georg your public key for ssh access into SVN, and you'll need developer access to the bug tracker. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
2010/7/11 Jesse Noller jnol...@gmail.com: On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum gu...@python.org wrote: [snip] Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) So, after some cool down - and the last rounds of discussion which triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148 futures - execute computations asynchronously. I feel that it's a good addition, and a good start for something bigger down the road. Brian - you'll need to provide someone such as Martin or Georg your public key for ssh access into SVN, and you'll need developer access to the bug tracker. Oh, don't worry. He's already been hacking happily on Windows stuff. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
On Sun, Jul 11, 2010 at 10:07 PM, Benjamin Peterson benja...@python.org wrote: 2010/7/11 Jesse Noller jnol...@gmail.com: On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum gu...@python.org wrote: [snip] Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) So, after some cool down - and the last rounds of discussion which triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148 futures - execute computations asynchronously. I feel that it's a good addition, and a good start for something bigger down the road. Brian - you'll need to provide someone such as Martin or Georg your public key for ssh access into SVN, and you'll need developer access to the bug tracker. Oh, don't worry. He's already been hacking happily on Windows stuff. -- Regards, Benjamin Wrong Brian - that's Brian Curtin, this is Brian Quinlan - I double checked the committer's list (http://www.python.org/dev/committers). We now have two Brians. I say we name them PresentBrian and FutureBrian. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement [ACCEPTED]
2010/7/11 Jesse Noller jnol...@gmail.com: On Sun, Jul 11, 2010 at 10:07 PM, Benjamin Peterson benja...@python.org wrote: 2010/7/11 Jesse Noller jnol...@gmail.com: On Sat, May 22, 2010 at 11:38 AM, Guido van Rossum gu...@python.org wrote: [snip] Great points Jesse! Since I really don't have the time or expertise to make a judgment on this PEP, I hereby appoint you chair of the approval process for this PEP. That basically means that when you think it's ready to be approved, you say so, and it's a done deal. The remaining feedback cycle is up to you now -- it sounds like you're ready for closure, which sounds good to me (again, without having read the PEP or tried to write something using the proposed code). You can do it however you like: you can declare it approved now, or read it over once more yourself and suggest some final changes, or set a period (e.g. 48 hours) during which final comments have to be received, which you then will judge by merit or by your whim, or you can flip a coin or say a prayer... (I've tried most of those myself in the past and haven't done too badly if I say so myself. :-) You're the boss now. I know you will do the right thing for this PEP. -- --Guido van Rossum (python.org/~guido) So, after some cool down - and the last rounds of discussion which triggered some jiggery-pokery on Brian's part, I'm accepting PEP 3148 futures - execute computations asynchronously. I feel that it's a good addition, and a good start for something bigger down the road. Brian - you'll need to provide someone such as Martin or Georg your public key for ssh access into SVN, and you'll need developer access to the bug tracker. Oh, don't worry. He's already been hacking happily on Windows stuff. -- Regards, Benjamin Wrong Brian - that's Brian Curtin, this is Brian Quinlan - I double checked the committer's list (http://www.python.org/dev/committers). We now have two Brians. I say we name them PresentBrian and FutureBrian. My apologies, Brians! -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 28, 2010, at 11:31 PM, Nick Coghlan ncogh...@gmail.com wrote: On 29/05/10 10:19, Jesse Noller wrote: In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! Yes, your suggestion for how to move things is the way we would want to do it, and in the back of my head, what we should do long term - just not right now. Yep, this is what I have been saying as well. 1. Using concurrency.futures rather than a top level futures module resolves the potential confusion with __future__ and stock market futures without inventing our own name for a well established computer science concept. 2. With the concurrency package in place following PEP 3148, we can separately consider the question of if/when/how to move other concurrency related modules (e.g. threading, multiprocessing, Queue) into that package at a later date. Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. I'll type something up this weekend and shoot it to Brian for inclusion. I was hoping to be able to keep it out of the futures pep itself, but it seems that won't work :) Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 29/05/10 22:46, Jesse Noller wrote: On May 28, 2010, at 11:31 PM, Nick Coghlan ncogh...@gmail.com wrote: Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. I'll type something up this weekend and shoot it to Brian for inclusion. I was hoping to be able to keep it out of the futures pep itself, but it seems that won't work :) Well, punting on whether or not we actually *do* part 2 is still fine. As Eric pointed out, there are issues with unpickling that make the wisdom of following through with renaming any existing modules fairly questionable. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Fri, May 28, 2010 at 17:12, Steven D'Aprano st...@pearwood.info wrote: On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: I've not seen this mentioned, but on such a long thread I might have missed it: we already have a __future__ module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term future. [...] I'm not sure of the benefit of a concurrent namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to threading, multiprocessing). I have suggested a way to move the existing concurrency stuff without breaking backwards compatibility, and Terry Reedy asked if it would work. I haven't seen any responses, either positive or negative. For the record, my suggestion was: for each concurrency modules: move it into the concurrency package add a top level module with the same name containing: # e.g. for threading from concurrency.threading import * Then in some future Python version, each top level module gets a PendingDeprecation warning, followed by a Deprecation warning some time later, and eventually in the indefinite future removal of the top level module. This was the procedure we used for about a month for Python 2.6 in order to help renamed modules migrate to their new names in Python 3. The issue that came up (and forced use to revert this approach and fully rely on 2to3) was anything pickled by the older interpreters is not going to be happy with that shift. Luckily the stuff being moved most likely does not contain things that have been pickled and stored to disk for ages and thus would break in a transition. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 28, 2010, at 1:39 PM, Scott Dial wrote: On 5/27/2010 4:13 AM, Brian Quinlan wrote: On 27 May 2010, at 17:53, Floris Bruynooghe wrote: On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Given your rationale, I don't understand from the PEP: shutdown(wait=True) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to Executor.submit and Executor.map and made after shutdown will raise RuntimeError. If wait is True then the executor will not return until all the pending futures are done executing and the resources associated with the executor have been freed. Can you tell me what is the expected execution time of the following: executor = ThreadPoolExecutor(max_workers=1) executor.submit(lambda: time.sleep(1000)) executor.shutdown(wait=False) sys.exit(0) I believe it's 1000 seconds, which seems to defy my request of shutdown(wait=False) because secretly the Python exit is going to wait anyways. It would take 1000 seconds. ...then the executor will not return... should read ...then the method will not return ISTM, it is much easier to get behavior #2 if you have behavior #1, and it would also seem rather trivial to make ThreadPoolExecutor take an optional argument specifying which behavior you want. Adding a daemon option would be reasonable. If you don't shutdown your executors you are pretty much guaranteed to get random traceback output on exit through. Your reference implementation does not actually implement the specification given in the PEP, so it's quite impossible to check this myself. There is no wait=True option for shutdown() in the reference implementation, so I can only guess what that implementation might look like. Look at around line 129 in: http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/thread.py Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Brian Quinlan brian at sweetapp.com writes: future is a computing science term of art, like thread. Anyway, this has been discussed in the past and Guido was happy with the name. I've not seen this mentioned, but on such a long thread I might have missed it: we already have a __future__ module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term future. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. You could have general thread pools that aren't related to executors (actually, it would be great if Python had a good built-in thread pool implementation) and I'd like to avoid using an overly generic name. Aren't executors and pools both housekeepers around a bunch of threads which execute code as a service for other threads? A thread is useless unless it executes code, isn't it? I'm not sure I understand the fine distinction between an executor and a pool. Having Executor as a suffix will give a point of familiarity to those who already know java.util.concurrent. And ProcessPool might cause confusion with multiprocessing.Pool because at first glance they seem to be the same thing. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. I think that Jesse was planning to add some functionality to this namespace. I don't really have an opinion on this. I'm not sure of the benefit of a concurrent namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to threading, multiprocessing). Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: I've not seen this mentioned, but on such a long thread I might have missed it: we already have a __future__ module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term future. [...] I'm not sure of the benefit of a concurrent namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to threading, multiprocessing). I have suggested a way to move the existing concurrency stuff without breaking backwards compatibility, and Terry Reedy asked if it would work. I haven't seen any responses, either positive or negative. For the record, my suggestion was: for each concurrency modules: move it into the concurrency package add a top level module with the same name containing: # e.g. for threading from concurrency.threading import * Then in some future Python version, each top level module gets a PendingDeprecation warning, followed by a Deprecation warning some time later, and eventually in the indefinite future removal of the top level module. Leaving the futures module in the top level of the std lib is far more likely to confuse users than adding it to its own namespace. Compare: import __future__ import futures with: import __future__ import concurrency.futures In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Vinay Sajip wrote: Brian Quinlan brian at sweetapp.com writes: future is a computing science term of art, like thread. Anyway, this has been discussed in the past and Guido was happy with the name. I've not seen this mentioned, but on such a long thread I might have missed it: we already have a __future__ module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term future. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. You could have general thread pools that aren't related to executors (actually, it would be great if Python had a good built-in thread pool implementation) and I'd like to avoid using an overly generic name. Aren't executors and pools both housekeepers around a bunch of threads which execute code as a service for other threads? A thread is useless unless it executes code, isn't it? I'm not sure I understand the fine distinction between an executor and a pool. Having Executor as a suffix will give a point of familiarity to those who already know java.util.concurrent. And ProcessPool might cause confusion with multiprocessing.Pool because at first glance they seem to be the same thing. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. I think that Jesse was planning to add some functionality to this namespace. I don't really have an opinion on this. I'm not sure of the benefit of a concurrent namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to threading, multiprocessing). Unless there's some way of having the two namespaces (existing and concurrent-oriented) simultaneously coexist. A single implementation with two separate namespace mappings? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ All I want for my birthday is another birthday - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 28, 2010, at 8:12 PM, Steven D'Aprano st...@pearwood.info wrote: On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: I've not seen this mentioned, but on such a long thread I might have missed it: we already have a __future__ module, as in from __future__ import with_statement and to my mind, this is a potential point of confusion with the term future. [...] I'm not sure of the benefit of a concurrent namespace, since it wouldn't contain the existing concurrency stuff. I think it's something to consider only for a big reorg which would break backward compatibility. IMO it would make more sense to leave this module as a top-level module for now (a sibling to threading, multiprocessing). I have suggested a way to move the existing concurrency stuff without breaking backwards compatibility, and Terry Reedy asked if it would work. I haven't seen any responses, either positive or negative. For the record, my suggestion was: for each concurrency modules: move it into the concurrency package add a top level module with the same name containing: # e.g. for threading from concurrency.threading import * Then in some future Python version, each top level module gets a PendingDeprecation warning, followed by a Deprecation warning some time later, and eventually in the indefinite future removal of the top level module. Leaving the futures module in the top level of the std lib is far more likely to confuse users than adding it to its own namespace. Compare: import __future__ import futures with: import __future__ import concurrency.futures In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! Yes, your suggestion for how to move things is the way we would want to do it, and in the back of my head, what we should do long term - just not right now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 29/05/10 10:19, Jesse Noller wrote: In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! Yes, your suggestion for how to move things is the way we would want to do it, and in the back of my head, what we should do long term - just not right now. Yep, this is what I have been saying as well. 1. Using concurrency.futures rather than a top level futures module resolves the potential confusion with __future__ and stock market futures without inventing our own name for a well established computer science concept. 2. With the concurrency package in place following PEP 3148, we can separately consider the question of if/when/how to move other concurrency related modules (e.g. threading, multiprocessing, Queue) into that package at a later date. Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Steven D'Aprano wrote: I have suggested a way to move the existing concurrency stuff without breaking backwards compatibility, and Terry Reedy asked if it would work. I haven't seen any responses, either positive or negative. For the record, my suggestion was: for each concurrency modules: move it into the concurrency package add a top level module with the same name containing: # e.g. for threading from concurrency.threading import * In the past the problem identified with this approach has been that pickles produced with new pythons would not be readable by older pythons. I think this was the main reason that Brett's 3.0 library reorganization wasn't more radical. Theres a discussion if this here: http://mail.python.org/pipermail/python-dev/2008-May/079535.html http://mail.python.org/pipermail/stdlib-sig/2008-May/000303.html and a little more here: http://bugs.python.org/issue2775 Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 27, 2010, at 1:21 PM, Greg Ewing wrote: On 27/05/10 12:04, Jesse Noller wrote: Namespaces are only a honking great idea if you actually let them do the job they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. I was referring to the issue of ThreadPool vs. ThreadPoolExecutor etc. By your own argument above, concurrent.futures.ThreadPool is quite descriptive enough of what it provides. It's not a problem if some other module also provides something called a ThreadPool. I think that the Executor suffix is a good indicator of the interface being provided. Pool is not because you can could have Executor implementations that don't involve pools. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. And yes, if you shut the interpreter down while threads are running they sometimes wake up at the wrong time to find the world around them destroyed. But that's part of programming with threads so it's not like the futures lib suddenly makes things behave differently. I'm glad I'm not alone in preferring (1) tough. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27 May 2010, at 17:53, Floris Bruynooghe wrote: On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. And yes, if you shut the interpreter down while threads are running they sometimes wake up at the wrong time to find the world around them destroyed. But that's part of programming with threads so it's not like the futures lib suddenly makes things behave differently. I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, 27 May 2010 14:29:28 +1200 Greg Ewing greg.ew...@canterbury.ac.nz wrote: On 27/05/10 01:48, Nick Coghlan wrote: I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. Wouldn't this be better addressed by adding the relevant options to the futures pools, rather than adding another module that provides almost exactly the same thing with different options? +1. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 12:48, Scott Dial wrote: On 5/26/2010 8:03 PM, Nick Coghlan wrote: On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a concurrent namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it futures, without putting it in a concurrent namespace. We can accept PEP 3148 by saying that we're happy to add the extra namespace level purely for disambiguation purposes, even if we never follow through on adding anything else to the package (although I consider such an outcome to be highly unlikely). Any future additions or renames to move things into the concurrent package would then be handled as their own PEPs. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 12:29, Greg Ewing wrote: On 27/05/10 01:48, Nick Coghlan wrote: I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. Wouldn't this be better addressed by adding the relevant options to the futures pools, rather than adding another module that provides almost exactly the same thing with different options? It would depend on the details, but my instinct says no (instead, the futures pools would be refactored to be task specific tailorings of the general purpose pools). However, this is all very hypothetical at this point and not really relevant to the PEP review. We may never even bother creating these more general purpose threading pools - it was just an example of the kind of thing that may go alongside the futures module. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 18:13, Brian Quinlan wrote: On 27 May 2010, at 17:53, Floris Bruynooghe wrote: I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Along those lines, an Executor.daemon option may be a good idea. That way the default behaviour is to wait until things are done (just like threading itself), but it is easy for someone to turn that behaviour off for a specific executor. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 10:29, Antoine Pitrou wrote: On Thu, 27 May 2010 10:19:50 +1000 Nick Coghlanncogh...@gmail.com wrote: futures.ThreadPoolExecutor would likely be refactored to inherit from the mooted pool.ThreadPool. There still doesn't seem to be reason to have two different thread pool APIs, though. Shouldn't there be one obvious way to do it? Executors and thread pools are not the same thing. I might create a thread pool for *anything*. An executor will always have a specific execution model associated with it (whether it be called futures, as in this case, or runnables or something else). This confusion is making me think that dropping the Pool from the names might even be beneficial (since, to my mind, it currently emphasises a largely irrelevant implementation detail). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 28/05/10 02:16, Antoine Pitrou wrote: On Fri, 28 May 2010 02:05:14 +1000 Nick Coghlanncogh...@gmail.com wrote: Executors and thread pools are not the same thing. I might create a thread pool for *anything*. An executor will always have a specific execution model associated with it (whether it be called futures, as in this case, or runnables or something else). I'm sorry, but what is the specific execution model you are talking about, and how is it different from what you usually do with a thread pool? Why would you do something other with a thread pool than running tasks and (optionally) collecting their results? Both the execution and communications models may be different. The components may be long-lived state machines, they may be active objects, they may communicate by message passing or by manipulating shared state, who knows. Executors are designed around a model of go do this and let me know when you're done. A general purpose pool needs to support other execution models, and hence will look different (and is harder to design and write, since it needs to be more flexible). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Brian Quinlan wrote: I think that the Executor suffix is a good indicator of the interface being provided. It's not usually considered necessary for the name of a type to indicate its interface. We don't have 'listsequence' and 'dictmapping' for example. I think what bothers me most about these names is their longwindedness. Two parts to a name is okay, but three or more starts to sound pedantic. And for me, Pool is a more important piece of information than Executor. The fact that it manages a pool is the main reason I'd use such a module rather than just spawning a thread myself for each task. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 28 May 2010, at 09:18, Greg Ewing wrote: Brian Quinlan wrote: I think that the Executor suffix is a good indicator of the interface being provided. It's not usually considered necessary for the name of a type to indicate its interface. We don't have 'listsequence' and 'dictmapping' for example. I think what bothers me most about these names is their longwindedness. Two parts to a name is okay, but three or more starts to sound pedantic. And for me, Pool is a more important piece of information than Executor. The fact that it manages a pool is the main reason I'd use such a module rather than just spawning a thread myself for each task. Actually, an executor implementation that created a new thread per task would still be useful - it would save you the hassle of developing a mechanism to wait for the thread to finish and to collect the results. We actually have such an implementation at Google and it is quite popular. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan br...@sweetapp.com wrote: Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Is there a compelling to make the threads daemon threads? If not, perhaps they can just be normal threads, and you can rely on the threading module to wait for them to finish. Unrelatedly, I feel like this behavior of waiting for the thread to terminate usually manifests as deadlocks when the main thread throws an uncaught exception. The application then no longer responds properly to interrupts, since it's stuck waiting on a semaphore. I guess it's better than the alternative of random crashes when daemon threads wake up during interpreter shutdown, though. Reid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 28, 2010, at 11:57 AM, Reid Kleckner wrote: On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan br...@sweetapp.com wrote: Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Is there a compelling to make the threads daemon threads? If not, perhaps they can just be normal threads, and you can rely on the threading module to wait for them to finish. Did you read my explanation of the reasoning behind my approach? Cheers, Brian Unrelatedly, I feel like this behavior of waiting for the thread to terminate usually manifests as deadlocks when the main thread throws an uncaught exception. The application then no longer responds properly to interrupts, since it's stuck waiting on a semaphore. I guess it's better than the alternative of random crashes when daemon threads wake up during interpreter shutdown, though. Reid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, May 27, 2010 at 8:06 PM, Brian Quinlan br...@sweetapp.com wrote: On May 28, 2010, at 11:57 AM, Reid Kleckner wrote: On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan br...@sweetapp.com wrote: Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Is there a compelling to make the threads daemon threads? If not, perhaps they can just be normal threads, and you can rely on the threading module to wait for them to finish. Did you read my explanation of the reasoning behind my approach? You should try to link to explanations. These have been long threads, and it's often hard to find the previous message where a subject was addressed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/27/2010 4:13 AM, Brian Quinlan wrote: On 27 May 2010, at 17:53, Floris Bruynooghe wrote: On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Given your rationale, I don't understand from the PEP: shutdown(wait=True) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to Executor.submit and Executor.map and made after shutdown will raise RuntimeError. If wait is True then the executor will not return until all the pending futures are done executing and the resources associated with the executor have been freed. Can you tell me what is the expected execution time of the following: executor = ThreadPoolExecutor(max_workers=1) executor.submit(lambda: time.sleep(1000)) executor.shutdown(wait=False) sys.exit(0) I believe it's 1000 seconds, which seems to defy my request of shutdown(wait=False) because secretly the Python exit is going to wait anyways. ISTM, it is much easier to get behavior #2 if you have behavior #1, and it would also seem rather trivial to make ThreadPoolExecutor take an optional argument specifying which behavior you want. Your reference implementation does not actually implement the specification given in the PEP, so it's quite impossible to check this myself. There is no wait=True option for shutdown() in the reference implementation, so I can only guess what that implementation might look like. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 28/05/10 09:52, Greg Ewing wrote: Nick Coghlan wrote: We can accept PEP 3148 by saying that we're happy to add the extra namespace level purely for disambiguation purposes, If that's the only rationale for the namespace, it makes it sound like a kludge to work around a poor choice of name. It's the right name though (it really is a futures implementation - I don't know what else you would even consider calling it). The problem is that the same word is used to mean different things in other programming domains (most obviously finance). Resolving that kind of ambiguity is an *excellent* use of a package namespace - you remove the ambiguity without imposing any significant long term cognitive overhead. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 06:22, Nick Coghlan ncogh...@gmail.com wrote: - download a futures module from PyPI and live with the additional dependency Why would that be a problem? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010 08:11, Lennart Regebro rege...@gmail.com wrote: On Wed, May 26, 2010 at 06:22, Nick Coghlan ncogh...@gmail.com wrote: - download a futures module from PyPI and live with the additional dependency Why would that be a problem? That has been hashed out repeatedly on this and other lists. Can it please be stipulated that for *some* people, in *some* cases, it is a problem? It seems to me that if you've experienced the sort of culture that makes it a problem, you understand the point immediately, but if you haven't, you never will (that's not disparaging anyone, the idiosyncracies of corporate culture are widespread and bizarre - if it helps, just remember that Dilbert is a documentary :-)) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 09:54:13 +1000 Nick Coghlan ncogh...@gmail.com wrote: What I would question here is what other things will be part of the concurrent package, and who will implement them. Are there plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like concurrent.pool to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Ha, I'm a bit surprised. Isn't it what futures already provides? (except that for some reason it insists on the SomeExecutor naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 24, 2010, at 5:36 AM, Brian Quinlan wrote: On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed. Linking to the pickle documentation might be a good start. Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. I suppose so. I guess I'm just looking for more precise usage of terminology. (This is a PEP, after all. It's a specification that multiple VMs may have to follow, not just some user documentation for a package, even if they'll *probably* be using your code in all cases.) I'd be happier if there were a clearer term than calls for the things being scheduled (submissions?), since the done callbacks aren't called in the subprocess for ProcessPoolExecutor, as we just discussed. Sure. Really, almost any contract would work, it just needs to be spelled out. It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary. Your concerns is that the thread will be killed when the interpreter exits? It won't be. Good to know. Tell it to the PEP though, not me ;). No reaction on [invoker vs. future]? I think you'll wish you did this in a couple of years when you start bumping into application code that calls set_result :). My reactions are mixed ;-) Well, you are not obliged to take my advice, as long as I am not obliged to refrain from mocking you mercilessly if it happens that I was right in a couple of years ;-). Your proposal is to add a level of indirection to make it harder for people to call implementation methods. The downside is that it makes it a bit harder to write tests and Executors. Both tests and executors will still create and invoke methods directly on one object; the only additional difficulty seems to be the need to type '.future' every so often on the executor/testing side of things, and that seems a cost well worth paying to avoid confusion over who is allowed to call those methods and when. I also can't see a big problem in letting people call set_result in client code though it is documented as being only for Executor implementations and tests. On the implementation side, I don't see why an Invoker needs a reference to the future. Well, uh... class Invoker(object): def __init__(self): Should only be called by Executor implementations. self.future = Future() ^ this is what I'd call a reference to the future ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 09:37, Paul Moore p.f.mo...@gmail.com wrote: It seems to me that if you've experienced the sort of culture that makes it a problem, Ah, it's a culture problem. In a heterogenous world, every action will benefit some and hurt some. Another arbitrary corporate ruleset could also mean you might be stuck on ancient python versions, and might not see a new module added to stdlib in 3.2 until 2015 or so. Some corporations go through a lot of trouble to prevent their employees from doing their job. Pythons core developers can not and should not let that hinder *them* from doing what is best for Python. Decisions on inclusion in stdlib must be made on what benefits Python and it's users in general. Since even small mistakes in a stdlib module will hurt far more people than having to having the module mature on PyPI until the worst API issues and bugs are ironed out, it's clear to me that letting a module mature on PyPI before inclusion is the better policy here, although how long obviously must be decided on a case by case basis. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 3:37 AM, Paul Moore wrote: On 26 May 2010 08:11, Lennart Regebro rege...@gmail.com wrote: On Wed, May 26, 2010 at 06:22, Nick Coghlan ncogh...@gmail.com wrote: - download a futures module from PyPI and live with the additional dependency Why would that be a problem? That has been hashed out repeatedly on this and other lists. Can it please be stipulated that for *some* people, in *some* cases, it is a problem? Sure, but I for one fully support Lennart asking the question, because while in the short term this *is* a problem with packaging tools in the Python ecosystem, in the long term (as you do note) it's an organizational dysfunction that can be addressed with better tools. I think it would be bad to ever concede the point that sane factoring of dependencies and code re-use aren't worth it because some jerk in Accounting or System Operations wants you to fill out a requisition form for a software component that's free and liberally licensed anyway. To support the unfortunate reality that such jerks in such departments really do in fact exist, there should be simple tools to glom a set of small, nicely factored dependencies into a giant monolithic ball of crud that installs all at once, and slap a sticker on the side of it that says I am only filling out your stupid form once, okay. This should be as distant as possible from the actual decision to package things in sensibly-sized chunks. In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything.___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 18:44, Stephen J. Turnbull wrote: Nick Coghlan writes: On 26/05/10 13:51, Stephen J. Turnbull wrote: People have been asking what's special about this module, to violate the BCP principle? There's nothing special about the fact that several people would use a robust and debugged futures module if it were in the stdlib. That's true of *every* module that is worth a PEP. The trick with futures and executor pools is that they're a *better* way of programming with threads in many cases. and However, given the choices of [...]. I'll choose the first option every time, and my programs will be the worse for it. Again, nothing all that special about those; lots of proposed changes satisfy similar conditions. I don't think anyone denies the truth or applicability of those arguments. But are they enough? Really, what you're arguing is now is better than never. Indeed, that is so. But you shouldn't forget that is immediately followed by although never is often better than *right* now. I've been trying to stay out of the meta-discussions but *right* now would be 6 months if it applies in this context. If that is what *right* now means to you then I hope that I never have a heart attack in your presence and need an ambulance *right* now :-) Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 18:09, Glyph Lefkowitz wrote: On May 24, 2010, at 5:36 AM, Brian Quinlan wrote: On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed. Linking to the pickle documentation might be a good start. Will do. Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. I suppose so. I guess I'm just looking for more precise usage of terminology. (This is a PEP, after all. It's a specification that multiple VMs may have to follow, not just some user documentation for a package, even if they'll *probably* be using your code in all cases.) I'd be happier if there were a clearer term than calls for the things being scheduled (submissions?), since the done callbacks aren't called in the subprocess for ProcessPoolExecutor, as we just discussed. Sure. Really, almost any contract would work, it just needs to be spelled out. It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary. Your concerns is that the thread will be killed when the interpreter exits? It won't be. Good to know. Tell it to the PEP though, not me ;). Will do. No reaction on [invoker vs. future]? I think you'll wish you did this in a couple of years when you start bumping into application code that calls set_result :). My reactions are mixed ;-) Well, you are not obliged to take my advice, as long as I am not obliged to refrain from mocking you mercilessly if it happens that I was right in a couple of years ;-). I was looking for your reasoning rather than trying to negotiate the circumstances under which you would mock me. Your proposal is to add a level of indirection to make it harder for people to call implementation methods. The downside is that it makes it a bit harder to write tests and Executors. Both tests and executors will still create and invoke methods directly on one object; the only additional difficulty seems to be the need to type '.future' every so often on the executor/testing side of things, and that seems a cost well worth paying to avoid confusion over who is allowed to call those methods and when. I also can't see a big problem in letting people call set_result in client code though it is documented as being only for Executor implementations and tests. On the implementation side, I don't see why an Invoker needs a reference to the future. Well, uh... class Invoker(object): def __init__(self): Should only be called by Executor implementations. self.future = Future() ^ this is what I'd call a reference to the future I said exactly the opposite of what I meant: futures don't need a reference to the invoker. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 4:55 AM, Brian Quinlan wrote: I said exactly the opposite of what I meant: futures don't need a reference to the invoker. Indeed they don't, and they really shouldn't have one. If I wrote that they did, then it was an error. ... and that appears to be it! Thank you for your very gracious handling of a pretty huge pile of criticism :). Good luck with the PEP, -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 04:25:18 -0400 Glyph Lefkowitz gl...@twistedmatrix.com wrote: In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything. +1. Why don't the castrated-by-the-corporation people offer to maintain a Sumo distribution of Python on python.org instead? The rest of the world shouldn't have to be impacted by their corporate culture woes. cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 07:39:15 pm Antoine Pitrou wrote: On Wed, 26 May 2010 04:25:18 -0400 Glyph Lefkowitz gl...@twistedmatrix.com wrote: In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything. +1. Why don't the castrated-by-the-corporation people offer to maintain a Sumo distribution of Python on python.org instead? The rest of the world shouldn't have to be impacted by their corporate culture woes. It's not just the corporate culture. For many people, the standard library is the first introduction to even the existence of a particular technique or technology. You can't go looking for something on PyPI if you don't know that there's a something to look for. And for many beginners and not-so-beginners, the idea and practice of installing additional packages is simply problematic. I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. This is a Python implementation of a stable Java API, Brian has said the futures package has been on PyPI for about a year, and it's been flagged as a production/stable release since October last year. http://pypi.python.org/pypi/futures3 Given that there does seem to be a general agreement that futures should go into the std lib at some point, is this not sufficient exposure? -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Having read through the PEP again, here are my thoughts. * I'm bothered by the term future. To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word future is not something that would spring readily to mind. The stated purpose of the module is to execute computations asynchronously, so perhaps a name such as asyntask would be appropriate, following the pattern of existing modules dealing with ansynchronous matters, ansyncore and asynchat. For the Future object itself, I'd suggest something like Task or Job. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Concerning the structure of the PEP: * A section titled 'Specification' should *not* start with a bunch of examples. It may be appropriate to include short examples *following* items in the specification in order to illustrate the features concerned. Extended examples such as these belong in a section of their own. * I found the examples included to be rather difficult to follow, and they served more to confuse than elucidate. I think this is partly because they are written in a fairly compressed style, burying important things being illustrated inside complicated expressions. Rewriting them in a more straightforward style might help. Concerning details of the specification: * Is it possible to have more than one Executor active at a time? The fact that as_completed() is a module-level function rather than an Executor method suggests that it is, but it would be good to have this spelled out one way or the other in the PEP. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano st...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) This is a Python implementation of a stable Java API, Brian has said the futures package has been on PyPI for about a year, and it's been flagged as a production/stable release since October last year. I'm not against futures being in the stdlib, I was just pointing out that I don't agree with the corporate culture issues should be accomodated by including more modules in the stdlib argument. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 8:57 PM, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * I'm bothered by the term future. To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word future is not something that would spring readily to mind. The stated purpose of the module is to execute computations asynchronously, so perhaps a name such as asyntask would be appropriate, following the pattern of existing modules dealing with ansynchronous matters, ansyncore and asynchat. For the Future object itself, I'd suggest something like Task or Job. future is a computing science term of art, like thread. Anyway, this has been discussed in the past and Guido was happy with the name. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. You could have general thread pools that aren't related to executors (actually, it would be great if Python had a good built-in thread pool implementation) and I'd like to avoid using an overly generic name. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. I think that Jesse was planning to add some functionality to this namespace. I don't really have an opinion on this. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Concerning the structure of the PEP: * A section titled 'Specification' should *not* start with a bunch of examples. It may be appropriate to include short examples *following* items in the specification in order to illustrate the features concerned. Extended examples such as these belong in a section of their own. I thought that the specification would be difficult to follow without examples to pave the way. Anyone else have an opinion on this? * I found the examples included to be rather difficult to follow, and they served more to confuse than elucidate. I think this is partly because they are written in a fairly compressed style, burying important things being illustrated inside complicated expressions. Rewriting them in a more straightforward style might help. Do you think starting with a simpler example would help? I think that idiomatic future use will end up looking similar to my examples. If that is too complex for most users then we have a problem. Concerning details of the specification: * Is it possible to have more than one Executor active at a time? Of course. The fact that as_completed() is a module-level function rather than an Executor method suggests that it is, but it would be good to have this spelled out one way or the other in the PEP. I'll add a note to the global functions that they can accept futures from different in the same call. Cheers, Brian -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 08:57:15 pm Greg Ewing wrote: * I'm bothered by the term future. To my mind, it's too long on cleverness and too short on explanativeness. Futures is a standard term in computer science which has been around for 33 years. http://en.wikipedia.org/wiki/Futures_and_promises I think that the standard library is no place for cuteness of naming. You mean like pickle, marshal, shelve, turtle, and even dict? * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. I'm sure that it can be done easily, although not quickly. For instance, we move threading into the concurrent namespace, and leave behind in its place a stub: from concurrent.threading import * Then for (say) 3.3 the stub could gain a PendingDeprecation warning, then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could be removed. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Hi On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! In thread.py the module automatically registers a handler with atexit. I don't think I'm alone in thinking libraries should not be doing this sort of thing unconditionally behind a user's back. I'm also not so sure how comfortable I am with the module-level globals. Would it not be possible to have an exit handler on each thread pool which the documentation reccomends you register with atexit if it suits your application? I think that would get rid of the global singletons and hidden atexit in a fairly elegant way. Lastly _base.py creates a LOGGER (connected to sys.stderr if I understand correctly) and only logs a critical message to it at the same time as a RuntimeError is raised. While I don't necessarily dislike that it uses a logger, I don't like that it's wired up to sys.stderr I rather think it's the application's duty to create a handler if it wants one. But given that it's only used at the same time as a RuntimeError it does seem redundant. Regards Floris PS: I've only looked at the threading part of the implementation. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010 11:56, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano st...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) I'm not sure how a Sumo approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a Sumo distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a sumo, just a couple of modules that nearly made it into the stdlib, at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them in the wild (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying support it yourself. I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a Sumo distribution style middle ground between stdlib and PyPI isn't really true... Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:19 AM, Paul Moore p.f.mo...@gmail.com wrote: On 26 May 2010 11:56, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano st...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) I'm not sure how a Sumo approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a Sumo distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a sumo, just a couple of modules that nearly made it into the stdlib, at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them in the wild (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying support it yourself. I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a Sumo distribution style middle ground between stdlib and PyPI isn't really true... Paul. The fat vs. thin stdlib was discussed on stdlib-sig some time ago (I am generally +1 to having a thin dist and a secondary fatter dist), however right now, it doesn't make sense packaging and dependency management is still a mess (but getting better), and there's a ton of other things to take into consideration, some of which has been iterated in this thread. That being said, we've now evolved into meta-meta-meta-discussion - if people seriously want to discuss the fat vs. thin subject, it should probably go to stdlib-sig. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:56, Antoine Pitrou wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Apranost...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) That's actually what happens with groups like Enthought and ActiveState - bundles with extra batteries. However, note that this isn't just a dysfunctional corporate culture issue, and I object to it being characterised as such (although dysfunctional cultures can certainly make it much, much worse). Vetting licenses for due diligence reasons, tracking releases of an external module, familiarising yourself with an additional API and code base, the risk of encountering bugs in that code base... these are all real costs that don't go away no matter how good the Python packaging ecosystem becomes. There is a trade off between do the simplest thing that could possibly work (but may cause you problems later) and spending the time to investigate third party solutions (with the risk that you end up rolling your own later anyway if you don't find anything suitable or, worse, find something that initially seems suitable but proves unworkable in practice). A module that makes it into the standard library, however, carries python-dev's stamp of approval. Except for some older legacy libraries, that means a module will have at least half decent documentation and an automated test suite that is regularly run on multiple platforms. Its design will also have run the gauntlet of python-dev approval. If we identify a good solution to a standard problem, and we have reason to believe that posting it in on PyPI as a separate module won't lead to a significant amount of additional real world testing, then it makes sense for it to go straight into the standard library. Such modules are going to be rare (since most non-trivial modules *will* benefit from some time on PyPI, and most trivial modules won't be added to the standard library at all), but they do exist (runpy, contextlib, collections, itertools and abc spring to mind). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/2010 13:19, Paul Moore wrote: On 26 May 2010 11:56, Antoine Pitrousolip...@pitrou.net wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Apranost...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) I'm not sure how a Sumo approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a Sumo distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a sumo, just a couple of modules that nearly made it into the stdlib, at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them in the wild (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying support it yourself. I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a Sumo distribution style middle ground between stdlib and PyPI isn't really true... Well... a middle ground certainly could exist; perhaps in the form of an Extended Standard Library (community distribution), with simple installation and management tools. It could be blessed by python-dev and maintain a high standard (only well established best-of-breed modules with a commitment of ongoing maintenance and more than one maintainer - something that the stdlib itself doesn't stick to). A common license could even be chosen, potentially allowing corporations to approve the extended package in a single pass. Lot of details to flesh out obviously - but it would be great to see something like this come into being. Obviously this would need to be a community initiative and would take some time to establish. A fat distribution like this, based on tools like pip and distribute would be great for both newbies and for experienced programmers in making it easier to find best solutions for standard problems. It could also act as an incubator for the standard library (perhaps with stable and experimental streams where stable has a more conservative update policy). All the best, Michael Foord Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:06, Floris Bruynooghe wrote: Hi On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! In thread.py the module automatically registers a handler with atexit. I don't think I'm alone in thinking libraries should not be doing this sort of thing unconditionally behind a user's back. I'm also not so sure how comfortable I am with the module-level globals. Would it not be possible to have an exit handler on each thread pool which the documentation reccomends you register with atexit if it suits your application? I think that would get rid of the global singletons and hidden atexit in a fairly elegant way. First let me explain why I install at atexit handler. Imagine that the you write a script like this: t = ThreadPoolExecutor(1) t.submit(lambda url: print(urllib.open(url).read()), 'http://www.apple.com/') You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I chose (2) but can be convinced otherwise. The obvious way to accomplish this is to make the worker thread non-daemon so the interpreter won't exit while it is running. But since the worker thread is part of a pool, it won't stop while it's executor is alive. So my approach was to make worker threads daemon and install an atexit handler that sets a global indicating that the interpreter is exiting so any workers should exit when when their work queues are empty. It then calls join on each worker thread so the interpreter will not exit until they are finished. I think that this approach is reasonable assuming that you want (2). I also don't have the aversion to globals that you do :-) Lastly _base.py creates a LOGGER (connected to sys.stderr if I understand correctly) and only logs a critical message to it at the same time as a RuntimeError is raised. While I don't necessarily dislike that it uses a logger, I don't like that it's wired up to sys.stderr I rather think it's the application's duty to create a handler if it wants one. But given that it's only used at the same time as a RuntimeError it does seem redundant. The LOGGER is only use for impossible exceptions (exceptions in the machinery of the module itself) that won't be propagated because they occur in worker threads. Cheers, Brian Regards Floris PS: I've only looked at the threading part of the implementation. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 17:38, Antoine Pitrou wrote: On Wed, 26 May 2010 09:54:13 +1000 Nick Coghlanncogh...@gmail.com wrote: What I would question here is what other things will be part of the concurrent package, and who will implement them. Are there plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like concurrent.pool to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Ha, I'm a bit surprised. Isn't it what futures already provides? (except that for some reason it insists on the SomeExecutor naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 22:32:33 +1000 Nick Coghlan ncogh...@gmail.com wrote: Ha, I'm a bit surprised. Isn't it what futures already provides? (except that for some reason it insists on the SomeExecutor naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:42 AM, Nick Coghlan ncogh...@gmail.com wrote: On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. I assume(d) it's sufficient to link to the mailing list threads where we hashed this out already ;) The namespace serves a few purposes: 1 We put futures where it makes sense - into a concurrent package. Futures are a concurrency construct; therefore it simply makes sense to put them within a sub package rather on the top level. 2 We carve out a box to move to, and add other concurrent things, such as generic pools, Actor implementations, etc. See java.util.concurrent. Things within multiprocessing that don't start with P and rhyme with rocess can go here too. Admittedly, it's mainly my own long-term vision to see python-core grow more concurrency tidbits - although I don't know too many people who would complain about it. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:42, Nick Coghlan wrote: On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. It doesn't at all. Are these plans formalized anywhere that I can link to? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 9:01 AM, Brian Quinlan br...@sweetapp.com wrote: On 26 May 2010, at 22:42, Nick Coghlan wrote: On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. It doesn't at all. Are these plans formalized anywhere that I can link to? Cheers, Brian Nope; and I don't think we need to worry about it right now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:50, Antoine Pitrou wrote: On Wed, 26 May 2010 22:32:33 +1000 Nick Coghlan ncogh...@gmail.com wrote: Ha, I'm a bit surprised. Isn't it what futures already provides? (except that for some reason it insists on the SomeExecutor naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) Interesting. Executor.submit() return a Future, which might not be useful in some ThreadPool fire-and-forget use cases but having them doesn't seem harmful. Java does take this approach and it gives you a lot more ways to customize the Executor thread pool i.e. the minimum number of threads running, the maximum number, the amount of time that a thread can be idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, priority). Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 23:01, Brian Quinlan wrote: _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. It doesn't at all. Are these plans formalized anywhere that I can link to? Just the previous lot of discussions. The main point that should be mentioned in the PEP is that futures on its own was ambiguous as to the applicable domain, but concurrent.futures was perfectly clear, without causing any readability problems the way a longer name could. Moving the general purpose pools out to their own module was just an example that occurred to us as something else that could go in that package rather than a concrete plan for implementation. Yes, we're setting ourselves up for inevitable questions as to why the existing modules are top level rather than part of this package, but the minimal pain response there would be to link to them from the package documentation with a note along the lines of for historical reasons, some modules you might reasonably expect to find in this package are instead provided as top level modules. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 23:29, Brian Quinlan wrote: On 26 May 2010, at 22:50, Antoine Pitrou wrote: I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) Interesting. Executor.submit() return a Future, which might not be useful in some ThreadPool fire-and-forget use cases but having them doesn't seem harmful. Java does take this approach and it gives you a lot more ways to customize the Executor thread pool i.e. the minimum number of threads running, the maximum number, the amount of time that a thread can be idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, priority). I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. It's something to investigate somewhere along the line, but, as Jesse pointed out, not something we need to worry about specifically for this PEP (except as an example of another module that may eventually end up in the concurrent package) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 6:56 AM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano st...@pearwood.info wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my Sumo proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) Not in a corporate environment, but I would gladly help with this. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:57, Greg Ewing wrote: (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, As a user, I agree with this. unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Is Steven D'Aprano's suggestion (in another post) possible? I'm sure that it can be done easily, although not quickly. For instance, we move threading into the concurrent namespace, and leave behind in its place a stub: from concurrent.threading import * Then for (say) 3.3 the stub could gain a PendingDeprecation warning, then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could be removed. On 5/26/2010 8:42 AM, Nick Coghlan wrote: _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. As a user, I disagree and think collecting together these and future modules (pun intended) would be beneficial. There are, from my viewpoint, too many top level modules already. [and in another thread] Yes, we're setting ourselves up for inevitable questions as to why the existing modules are top level rather than part of this package, Yes, forever until they are put in the package. but the minimal pain responsethere For the developers, for the short term would be to link to them from the package documentation with a note along the lines of for historical reasons, some modules you might reasonably expect to find in this package are instead provided as top level modules. You are, in my opinion, overly discounting the cognitive load and confusion on the part of new users. It would be much better to link *to* subpackage documentation *from* a top level entries (until deleted) and just say that the top level synonyms are present for the obvious historical reason that there once no package, just modules. I am suggesting that if we add a package, we do it right, from the beginning. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 3:57 AM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Having read through the PEP again, here are my thoughts. * I'm bothered by the term future. To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word future is not something that would spring readily to mind. Please go re-read the older threads on this. For example, http://mail.python.org/pipermail/python-dev/2010-March/098279.html. * It seems unnecessarily verbose to tack Executor onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. +0 * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called concurrent, I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. Again, please re-read the older thread (which you participated in). For example, http://mail.python.org/pipermail/python-dev/2010-March/098200.html. Jeffrey ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Brian Quinlan wrote: I think that Jesse was planning to add some functionality to this namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. I thought that the specification would be difficult to follow without examples to pave the way. Well, for me, what happened was that I saw the examples and thought WTF is going on here? Then I read the specification to figure out how the examples worked. It might be better to have a tutorial section preceeding the specification section, containing explanation interspersed with examples. I think that idiomatic future use will end up looking similar to my examples. Maybe, but code written for pedagogical purposes needs to meet a particularly high standard of clarity. Remember that the reader isn't yet familiar with the idioms, so idiomatic code isn't necessarily going to be easy for him to follow. * Is it possible to have more than one Executor active at a time? Of course. That's good, but I don't think that the of course is at all obvious, considering that things such as GUI event loops generally can't be mixed easily. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). A separate short PEP proposing a migration plan that could be accepted or rejected independently of PEP 3148 would likely be valuable. E.g. - no change in 2.x (obviously) - add concurrent.* alternate names in 3.x - rearrange documentation in 3.x, with pointers from old names to new names - put a PendingDeprecationWarning on the old names, but otherwise leave them alone indefinitely - add 2to3 fixers to translate from the old names to the new names in import statements Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 7:36 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Brian Quinlan wrote: I think that Jesse was planning to add some functionality to this namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. Not entirely; once concurrent.* comes into existence, I will seriously begin looking at what we can move out of multiprocessing, into concurrent.* alongside futures. You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:03 PM, Nick Coghlan ncogh...@gmail.com wrote: On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). A separate short PEP proposing a migration plan that could be accepted or rejected independently of PEP 3148 would likely be valuable. E.g. - no change in 2.x (obviously) - add concurrent.* alternate names in 3.x - rearrange documentation in 3.x, with pointers from old names to new names - put a PendingDeprecationWarning on the old names, but otherwise leave them alone indefinitely - add 2to3 fixers to translate from the old names to the new names in import statements Cheers, Nick. Agreed; and intended as a different PEP. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 09:36, Greg Ewing wrote: Brian Quinlan wrote: I think that Jesse was planning to add some functionality to this namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. futures.ThreadPoolExecutor would likely be refactored to inherit from the mooted pool.ThreadPool. I'd like to leave that option open, and having two classes with the same name from different modules in a single inheritance tree is one of the places where module namespacing still isn't quite as tidy as we might wish. I'd also consider a simple thread pool and an actual executor different things. I'm fine with the longer names, but if I was going to drop a word from the names, it would actually be Pool (i.e. ThreadExecutor, ProcessExecutor). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, 27 May 2010 10:19:50 +1000 Nick Coghlan ncogh...@gmail.com wrote: futures.ThreadPoolExecutor would likely be refactored to inherit from the mooted pool.ThreadPool. There still doesn't seem to be reason to have two different thread pool APIs, though. Shouldn't there be one obvious way to do it? I'd also consider a simple thread pool and an actual executor different things. I'm fine with the longer names, but if I was going to drop a word from the names, it would actually be Pool (i.e. ThreadExecutor, ProcessExecutor). To me, ThreadPool looks a lot more obvious than ThreadExecutor (obvious in that I can easily find it again, and I don't need to read some documentation to know what it is). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. Also, automatically doing (2) would seem to make it difficult for a program to bail out if something unexpected happens. It would have to explicitly shut down the thread pool instead of just letting an exception propagate. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Well... a middle ground certainly could exist; perhaps in the form of an Extended Standard Library (community distribution), with simple installation and management tools. It could be blessed by python-dev and maintain a high standard (only well established best-of-breed modules with a commitment of ongoing maintenance and more than one maintainer - something that the stdlib itself doesn't stick to). A common license could even be chosen, potentially allowing corporations to approve the extended package in a single pass. I read the 'sumo' thread before I read this (and replied in depth there), but I think Michael and I mean similar things. - Yaniv ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 7:15 PM, Yaniv Aknin ya...@aknin.name wrote: Well... a middle ground certainly could exist; perhaps in the form of an Extended Standard Library (community distribution), with simple installation and management tools. I'm not sure about the 'installation and management tools' part, but this is basically the idea I was trying to articulate: a middle ground between a 'fat' stdlib and a 'lean' one. It could be blessed by python-dev and maintain a high standard (only well established best-of-breed modules with a commitment of ongoing maintenance and more than one maintainer - something that the stdlib itself doesn't stick to). A common license could even be chosen, potentially allowing corporations to approve the extended package in a single pass. If we could do it that would be great, IMHO. I read the 'sumo' thread before I read this (and replied in depth there), but I think Michael and I mean similar things. - Yaniv I don't think I'm understanding you correctly in that thread then, ISTM that you're advocating better packaging systems as an alternative to this. Would you mind clarifying? Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 01:48, Nick Coghlan wrote: I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. Wouldn't this be better addressed by adding the relevant options to the futures pools, rather than adding another module that provides almost exactly the same thing with different options? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/26/2010 8:03 PM, Nick Coghlan wrote: On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a concurrent namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it futures, without putting it in a concurrent namespace. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 12:04, Jesse Noller wrote: Namespaces are only a honking great idea if you actually let them do the job they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. I was referring to the issue of ThreadPool vs. ThreadPoolExecutor etc. By your own argument above, concurrent.futures.ThreadPool is quite descriptive enough of what it provides. It's not a problem if some other module also provides something called a ThreadPool. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
I don't think I'm understanding you correctly in that thread then, ISTM that you're advocating better packaging systems as an alternative to this. Would you mind clarifying? Gladly. In my mind, 'better packaging' is not just about something that will let you do 'pypkg install foo' and 'pypkg remove foo' along with dependancies, it's also about meta-packages (packages that have nothing but dependencies) but also about separate repositories with different natures, some endorsed by python-dev, some not. It's not so much the packaging system that does the trick - it's the eco system around it. When you install something from the 'recommended' repository (should find a better name), you know its a 'blessed' package. You know it has unittests, documentation, active development, good license, a bug tracker and the general-soundedness which I hope users have come to expect of python-dev. It is not stdlib, it's not made by python-dev, it doesn't come with python.org's default build by default, but you have a lot of important assurances with ease. You know 'it could have been in stdlib, but happens not to be'. To make this really work, the future tool I called pypkg will come with all future Python versions (so people can rely on it), and will arrive configured by default to use only the 'recommended' repository. From a developer's standpoint, relying on package 'foo' of the 'recommended' repository is safe: even if their end users don't happen to have 'foo', the developer is certain that 'recommended' is configured on their user's Python and is a quick (automagical?) download away. - Yaniv (p.s.: again, think Ubuntu core vs universe vs multiverse vs. non-Ubuntu repositories vs. PPAs; forgive me for waving the Ubuntu flag, from the packaging systems /and accompanying eco-systems/ I know, they did it best) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 23/05/10 22:47, Antoine Pitrou wrote: On Sun, 23 May 2010 08:34:22 -0400 Jesse Nollerjnol...@gmail.com wrote: Brian has already agreed to name spacing it to concurrent.futures - this means it will be a small part to a much larger concurrent.* implementation ala Java. What I would question here is what other things will be part of the concurrent package, and who will implement them. Are there plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like concurrent.pool to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 23/05/10 21:56, Lennart Regebro wrote: On Sun, May 23, 2010 at 13:29, Brian Quinlanbr...@sweetapp.com wrote: Parts of it, yes. Just like I can replace most operations in os.path and urlparse with a few lines of code. Yeah, but parts of is not the question. I've read the PEP, and I do *not* know how to implement it. That means it's not a trivial module, so that argument doesn't hold up here, even if we accept it as valid (which I actually don't). I don't think any module in the stdlib is entirely trivial. Yes, even parsing an URL is non-trivial, as shown by the fact that the urlparse module apparently has a bug in it for urls like svn+ssh://foo.bar/frotz. ;-) In this case, the trivial refers to being able to write something that will get the job done for a specific task or application, but that isn't as solid from a reliability/maintainability/portability/scalability point of view. By providing solid infrastructure in the standard library, we can remove that choice between do it fast vs do it right, by providing ready-made robust infrastructure. Those that say just put it on PyPI may not recognise the additional overhead that can be involved in identifying, obtaining approval to use and ongoing management of additional dependencies in a corporate environment that is actually performing appropriate due diligence in regards to IP licensing. This overhead can be especially significant (and, depending on licence and contract details, even a dealbreaker) for companies with specific IP licensing provisions in their contracts with their clients. It doesn't matter *how* easy we make it to download PyPI packages, we can't do anything about such company IP management policies (except for making it easier for programmers to violate them thoughtlessly, of course). To use myself as an example, I have several utilities that I probably would have written differently if the futures module had been available in the standard library at the time I wrote them. As it is, they work well enough, but their usage of the threading module is fairly ad hoc (and migrating them to use multiprocessing would be a fairly complex task, and making that decision run-time selectable even more complicated). In the near-term, backports of future standard library modules are much easier to get through a corporate review process as the licensing is typically similar to the PSF license (or is even the PSF license itself) and the modules come with a clear roadmap for eliminating the dependency (i.e. once the baseline Python version employed by the company includes the module in the standard library, the external dependency is no longer needed). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Tue, May 25, 2010 at 7:54 PM, Nick Coghlan ncogh...@gmail.com wrote: On 23/05/10 22:47, Antoine Pitrou wrote: On Sun, 23 May 2010 08:34:22 -0400 Jesse Nollerjnol...@gmail.com wrote: Brian has already agreed to name spacing it to concurrent.futures - this means it will be a small part to a much larger concurrent.* implementation ala Java. What I would question here is what other things will be part of the concurrent package, and who will implement them. Are there plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like concurrent.pool to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Nick is correct - there's plenty of things in multiprocessing which belong in a more abstract package as they're useful for more things than just multiprocessing. I don't think they need to be called out as part of the PEP though. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 02:10, Nick Coghlan ncogh...@gmail.com wrote: Those that say just put it on PyPI may not recognise the additional ... Just a note, so we don't get sidelined by misunderstandings: I don't think anybody said that. ;-) There are two issues here, one generic and one specific: Generic: Modules should go on PyPI first, for a time, to stabilize (and so they can be used in earlier versions of Python) before they end up in stdlib. I suspect everyone actually agrees on that (but I could be wrong). Specific:Has futures been long enough on PyPI, and is it stable? I'm staying out of that discussion. :-) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 12:29, Lennart Regebro wrote: On Wed, May 26, 2010 at 02:10, Nick Coghlanncogh...@gmail.com wrote: Those that say just put it on PyPI may not recognise the additional ... Just a note, so we don't get sidelined by misunderstandings: I don't think anybody said that. ;-) Nah, that pseudo-quote wasn't from this discussion in particular. It's a reference to the ongoing tension between the batteries included advocates and the make the standard library as streamlined as possible crowd. Both sides have valid points, so the included battery vs optional download question needs to be decided on a case-by-case basis. There are two issues here, one generic and one specific: Generic: Modules should go on PyPI first, for a time, to stabilize (and so they can be used in earlier versions of Python) before they end up in stdlib. I suspect everyone actually agrees on that (but I could be wrong). That's the point I'm disagreeing with. For most modules it makes sense to do things that way, but for some low-level infrastructure elements, it is going to be less effective (because people will quickly throw together their own solutions instead of adding a new dependency for something simple). Other times we'll invent a new module because *we* need it for something (e.g. runpy). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 24/05/10 20:46, Stephen J. Turnbull wrote: Cameron Simpson writes: There's a lot to be said for a robust implementation of a well defined problem. Brian's module, had it been present and presuming it robust and debugged, would have been quite welcome. That, of course, is the consensus view, both in general and with respect to this particular module. The difference is over what constitutes sufficient evidence for your presumption of robust and debugged from the point of view of the users of the stdlib. At the very least, we'll be offering a promise to be more robust and more debugged than what you came up with in that coding marathon last night ;) Having a decent test suite that is regularly executed on multiple platforms (which will be the case for any accepted module by the time it is included in a Python release) also places anything we release a cut above a *lot* of in-house code. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Nick Coghlan writes: Those that say just put it on PyPI Nobody is saying that, AFAICS. Nobody is saying that *some* futures module shouldn't *eventually* go into the stdlib. The question is whether *this* futures module should go into the stdlib *now*. And it is clear that more time on PyPI would provide valuable information. This is a general principle that has served us well: put best current practice backed up by actual widespread usage in the stdlib, not theoretical wins based on the developer's experience. PyPI is a way to broaden usage to determine BCP, not an end in itself. People have been asking what's special about this module, to violate the BCP principle? There's nothing special about the fact that several people would use a robust and debugged futures module if it were in the stdlib. That's true of *every* module that is worth a PEP. But remember, in the case of ipaddr it was the people who wanted some such module badly who were also the most vocal opponents, because they could see offhand that it was going to serve their use cases badly. (It turned out that this was equally trivial to fix despite a week of hot debate, and everyone lived happily ever after. But that was smiling Luck, not inexorable Fate.) For this module, three people have said I 'would have' used it if it were available, but none of you has announced that you've started refactoring and the PEP 3148 API meets all expectations. I call that damning with faint praise. OTOH, Glyph has changed from why not more time on PyPI? to let's see if we can improve this a bit, then let's do it. He has published code (showing how to turn futures into Twisted Deferreds), and argues that based on download stats to date and the nature of the use cases it would take a lot of time on PyPI to demonstrate a BCP. Those are good arguments for an exception, IMHO. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 13:51, Stephen J. Turnbull wrote: People have been asking what's special about this module, to violate the BCP principle? There's nothing special about the fact that several people would use a robust and debugged futures module if it were in the stdlib. That's true of *every* module that is worth a PEP. I actually wrote a reply to that question earlier in the week, but failed at using gmail's web interface correctly and only sent it to Steve Holden. === The trick with futures and executor pools is that they're a *better* way of programming with threads in many cases. However, given the choices of: - hack together something simple with some worker Threads and a Queue (or two) - write my own futures and executor infrastructure - download a futures module from PyPI and live with the additional dependency I'll choose the first option every time, and my programs will be the worse for it. Put the capability to use futures and an executor into the stdlib, and it becomes something I can reach for without having to worry about additional dependencies beyond specifying a minimal Python version. It provides a higher level API that can be more readily switched between threading and multiprocessing back ends. It becomes something that can be taught as a standard Python technique for enabling concurrency in imperative code. This is something that is irrelevant to me as a module on PyPI, but has the potential to significantly affect my programming in the future as a standard library module. Even in the near term, backports of future standard library modules are often perceived differently when being discussed as potential additional dependencies for an application (i.e. I believe it would be worthwhile for a backport of the module to earlier Python versions than 3.2 to be made available on PyPI). === Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 24 May 2010 03:58, Cameron Simpson c...@zip.com.au wrote: I almost am Brian's hypothetical user. I've got a FuncMultiQueue that accepts callables in synchronous and asynchronous modes for future possibly-concurrent execution, just as the futures module does. I've spent a _lot_ of time debugging it. I pretty much am that user as well (whether or not I am hypothetical, I'll leave to others to determine...) I have a set of scripts that needed to do precisely the sort of thing that the futures module offers. I searched for a fair while for a suitable offering (this was before futures had been published) and found nothing suitable. So in the end I implemented my own - and I hit corner cases, and they needed a lot of work to fix. I now have a working solution, but it's too tangled in the application logic to be reusable :-( If futures had been in the stdlib, I'd have used it like a shot, and saved myself a lot of wasted time. There's a lot to be said for a robust implementation of a well defined problem. Brian's module, had it been present and presuming it robust and debugged, would have been quite welcome. Precisely my view. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Am 24.05.2010 01:51, schrieb Greg Ewing: Brian Quinlan wrote: The good news in this case is that the same API has been used successfully in Java and C++ for years so it is unlikely that any major changes will need to be made. That doesn't follow. An API that's appropriate for Java or C++ is not necessarily appropriate for Python. Slavishly copying an API from another language is often not the best approach when designing an API for a Python module. *cough* unittest *cough* Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 2010-05-23, Terry Reedy wrote: On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote: On Sat, May 22, 2010 at 4:12 PM, Brian Quinlanbr...@sweetapp.com wrote: Rename executor = executer -1 for consistency with Java. -10 for 'executer'. As far as I can find out, it is a misspelling of 'executor'. If the designers of some other language made a stupid mistake, let them correct it instead of us following them over a cliff. I'd suggested this because it seemed obvious to me, but clearly not. Compare: http://www.thefreedictionary.com/executor http://www.thefreedictionary.com/executer However, as I mentioned in the first place I didn't expect any change of this since Java uses the first spelling. [snip] -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy Advanced Qt Programming - ISBN 0321635906 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote: On May 23, 2010, at 2:37 AM, Brian Quinlan wrote: On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! As you wish! I retract my request ;-) May you get what you wish for, may you find what you are seeking :). The PEP should be consistent in its usage of terminology about callables. It alternately calls them callables, functions, and functions or methods. It would be nice to clean this up and be consistent about what can be called where. I personally like callables. Did you find the terminology confusing? If not then I propose not changing it. Yes, actually. Whenever I see references to the multiprocessing module, I picture a giant HERE BE (serialization) DRAGONS sign. When I saw that some things were documented as being functions, I thought that maybe there was intended to be a restriction like the these can only be top-level functions so they're easy for different executors to locate and serialize. I didn't realize that the intent was arbitrary callables until I carefully re-read the document and noticed that the terminology was inconsistent. ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed. But changing it in the user docs is probably a good idea. I like callables too. Great. Still, users will inevitably find the PEP and use it as documentation too. The execution context of callable code is not made clear. Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly. Any response to this bit? Did I miss something in the PEP? Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think. More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination. If I'm reading the sample implementation correctly, http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241 , it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread. The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine). The callbacks will always be called in a thread other than the main thread in the process that created the executor. Is that a strong enough contract? Sure. Really, almost any contract would work, it just needs to be spelled out. It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary. Your concerns is that the thread will be killed when the interpreter exits? It won't be. This is really minor, I know, but why does it say NOTE: This method can be used to create adapters from Futures to Twisted Deferreds? First of all, what's the deal with NOTE; it's the only NOTE in the whole PEP, and it doesn't seem to add anything. This sentence would read exactly the same if that word were deleted. Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted. But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination? If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write? I'm a bit surprised that this doesn't allow for better interoperability with Deferreds given this discussion: discussion snipped I did not communicate that well. As implemented, it's quite possible to implement a translation layer which turns a Future into a Deferred. What I meant by that comment was, the specification in the PEP was to loose to be sure that such a layer would work with arbitrary executors. For what it's worth, the Deferred translator would look like this, if you want to include it in the PEP (untested though, you may want to run it first): from twisted.internet.defer import Deferred from twisted.internet.reactor import callFromThread def future2deferred(future): d = Deferred() def invoke_deferred():
Re: [Python-Dev] PEP 3148 ready for pronouncement
Cameron Simpson writes: There's a lot to be said for a robust implementation of a well defined problem. Brian's module, had it been present and presuming it robust and debugged, would have been quite welcome. That, of course, is the consensus view, both in general and with respect to this particular module. The difference is over what constitutes sufficient evidence for your presumption of robust and debugged from the point of view of the users of the stdlib. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote: On May 22, 2010, at 8:47 PM, Brian Quinlan wrote: Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! As you wish! I retract my request ;-) The PEP should be consistent in its usage of terminology about callables. It alternately calls them callables, functions, and functions or methods. It would be nice to clean this up and be consistent about what can be called where. I personally like callables. Did you find the terminology confusing? If not then I propose not changing it. But changing it in the user docs is probably a good idea. I like callables too. The execution context of callable code is not made clear. Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly. More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination. If I'm reading the sample implementation correctly, http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241 , it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread. The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine). The callbacks will always be called in a thread other than the main thread in the process that created the executor. Is that a strong enough contract? This is really minor, I know, but why does it say NOTE: This method can be used to create adapters from Futures to Twisted Deferreds? First of all, what's the deal with NOTE; it's the only NOTE in the whole PEP, and it doesn't seem to add anything. This sentence would read exactly the same if that word were deleted. Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted. But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination? If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write? I'm a bit surprised that this doesn't allow for better interoperability with Deferreds given this discussion: At 02:36 PM 3/16/2010 -0700, Brian Quinlan wrote: From P.J Eby: On Mar 7, 2010, at 11:56 AM, P.J. Eby wrote: At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: Given a way to register on-done callbacks with the future, it would be straightforward to wait for a future without blocking, too. Yes, and with a few more additions besides that one, you might be on the way to an actual competitor for Deferreds. For example: retry support, chaining, logging, API for transparent result processing, coroutine support, co-ordination tools like locks, sempaphores and queues, etc. OK, but lets just think about making the APIs compatible e.g. you have some code that uses Futures and now you want to integrate it with some code that uses Deferreds. I think Jeff's suggestion of having a completion callback on Futures would make it possible to write a Future-to-Deferred adapter. Is that correct? As long as the callback signature included a way to pass in an error, then yes, that'd probably be sufficient. If add_done_callback doesn't help with twisted interoperability then I'd suggest removing it to allow for something that may be more useful to be added later. The fact that add_done_callback is implemented using a set is weird, since it means you can't add the same callback more than once. The set implementation also means that the callbacks get called in a semi-random order, potentially creating even _more_ hard-to-debug order of execution issues than you'd normally have with futures. And I think that this documentation will be unclear to a lot of novice developers: many people have trouble with the idea that a = Foo(); b = Foo(); a.bar_method != b.bar_method, but import foo_module; foo_module.bar_function == foo_module.bar_function. It's also weird that you can remove callbacks - what's the use case? Deferreds have no callback-removal mechanism and nobody has ever complained of the need for one, as far as I know. (But lots of people do add the same callback multiple times.) I suggest having have add_done_callback, implementing it with a list so that callbacks are always invoked in the
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan br...@sweetapp.com wrote: snip Finally, why isn't this just a module on PyPI? It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :). We've already had this discussion before. Could you explain why this module should *not* be in the stdlib e.g. does it have significantly less utility than other modules in stdlib? Is it significantly higher risk? etc? Inclusion in the stdlib is the exception, not the rule, and every exception should be issued for a good reason. I'd like to know what that reason is in this case, if only to get a clearer understanding of why the PEP was accepted. Issues like the ones I'm bringing up could be fixed pretty straightforwardly if it were just a matter of filing a bug on a small package, but fixing a stdlib module is a major undertaking. True but I don't think that is a convincing argument. A subset of the functionality provided by this module is already available in Java and C++ and (at least in Java) it is used extensively and without too much trouble. If there are implementation bugs then we can fix them just like we would with any other module. Guido made exactly the opposite argument during his keynote at PyCon. It seemed fairly reasonable at the time- why do you think it doesn't apply here? Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 23, 2010, at 7:15 PM, geremy condra wrote: On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan br...@sweetapp.com wrote: snip Finally, why isn't this just a module on PyPI? It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :). We've already had this discussion before. Could you explain why this module should *not* be in the stdlib e.g. does it have significantly less utility than other modules in stdlib? Is it significantly higher risk? etc? Inclusion in the stdlib is the exception, not the rule, and every exception should be issued for a good reason. I'd like to know what that reason is in this case, This package eliminates the need to construct the boilerplate present in many Python applications i.e. a thread or process pool, a work queue and result queue. It also makes it easy to take an existing Python application that executes (e.g. IO operations) in sequence and execute them in parallel. It package provides common idioms for two existing modules i.e. multiprocessing offers map functionality while threading doesn't. Those idioms are well understood and already present in Java and C++. if only to get a clearer understanding of why the PEP was accepted. It hasn't been accepted. Issues like the ones I'm bringing up could be fixed pretty straightforwardly if it were just a matter of filing a bug on a small package, but fixing a stdlib module is a major undertaking. True but I don't think that is a convincing argument. A subset of the functionality provided by this module is already available in Java and C++ and (at least in Java) it is used extensively and without too much trouble. If there are implementation bugs then we can fix them just like we would with any other module. Guido made exactly the opposite argument during his keynote at PyCon. It seemed fairly reasonable at the time- why do you think it doesn't apply here? Could you be a little more specific about Guido's argument at PyCon? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Sun, May 23, 2010 at 11:39, Brian Quinlan br...@sweetapp.com wrote: This package eliminates the need to construct the boilerplate present in many Python applications i.e. a thread or process pool, a work queue and result queue. It also makes it easy to take an existing Python application that executes (e.g. IO operations) in sequence and execute them in parallel. It package provides common idioms for two existing modules i.e. multiprocessing offers map functionality while threading doesn't. Those idioms are well understood and already present in Java and C++. It can do that as a separate package as well. And not only that, it could then be available on PyPI for earlier versions of Python as well, making it much more likely to gain widespread acceptance. Could you be a little more specific about Guido's argument at PyCon? A module in stdlib has to be dead. After it's included in the stdlib it can not go through any major changes since that would mean loss of backwards incompatibility. Also, you can't fix bugs except by releasing new versions of Python. Therefore the API must be completely stable, and the product virtually bugfree before it should be in stdlib. The best way of ensuring that is to release it as a separate module on PyPI, and let it stabilize for a couple of years. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 23, 2010, at 7:54 PM, Lennart Regebro wrote: On Sun, May 23, 2010 at 11:39, Brian Quinlan br...@sweetapp.com wrote: This package eliminates the need to construct the boilerplate present in many Python applications i.e. a thread or process pool, a work queue and result queue. It also makes it easy to take an existing Python application that executes (e.g. IO operations) in sequence and execute them in parallel. It package provides common idioms for two existing modules i.e. multiprocessing offers map functionality while threading doesn't. Those idioms are well understood and already present in Java and C++. It can do that as a separate package as well. You could make the same argument about any module in the stdlib. And not only that, it could then be available on PyPI for earlier versions of Python as well, making it much more likely to gain widespread acceptance. I doubt it. Simple modules are unlikely to develop a following because it is too easy to partially replicate their functionality. urlparse and os.path are very useful modules but I doubt that they would have been successful on PyPI. Could you be a little more specific about Guido's argument at PyCon? A module in stdlib has to be dead. After it's included in the stdlib it can not go through any major changes since that would mean loss of backwards incompatibility. The good news in this case is that the same API has been used successfully in Java and C++ for years so it is unlikely that any major changes will need to be made. Also, you can't fix bugs except by releasing new versions of Python. Therefore the API must be completely stable, and the product virtually bugfree before it should be in stdlib. The best way of ensuring that is to release it as a separate module on PyPI, and let it stabilize for a couple of years. Yeah but that model isn't likely to work with this package. Cheers, Brian___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
