Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
I've updated the PEP to include: - completion callbacks (for interoperability with Twisted Deferreds) - a pointer to the discussion on stdlig-sig See: http://svn.python.org/view/peps/trunk/pep-3148.txt?r1=78618r2=80679 Rejected ideas: - Having a registration system for executors Not yet addressed: - where the package should live (someone in a concurrent package seems fine) - having global executors with unbounded worker counts as a convenience [1] [1] There are a few issues with global executors that need to be thought thought through i.e. when should workers be created and when should they be terminated. I'd be happy to defer this idea unless someone is passionate about it (in which case it would be great if they'd step in with concrete ideas). Cheers, Brian___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Nick Coghlan wrote: You may want to consider providing global thread and process executors in the futures module itself. Code which just wants to say do this in the background without having to manage the lifecycle of its own executor instance is then free to do so. I've had a lot of experience with a framework that provides this and it is *very* convenient (it's also a good way to avoid deadlocks due to synchronous notification APIs). This seems like a reasonable idea to me. I take it that the thread/process pool should be unlimited in size. Should every thread/process exit when it finishes its job or should there be a smarter collection strategy? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: I take it that the thread/process pool should be unlimited in size. Should every thread/process exit when it finishes its job or should there be a smarter collection strategy? I'd be inclined to do something slightly smarter, similar to what we do with memory overallocation for mutable containers. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 10 Mar 2010, at 23:32, Nick Coghlan wrote: Brian Quinlan wrote: Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. I'm not sure what you mean, could you clarify? Assuming your question refers to the second sentence, Jean-Paul is referring to a trick of the CPython interpreter when it terminates. To maximise the chances of objects being deleted properly rather than just dumped from memory when the process exits, module dictionaries are filled with None values before the interpreter shuts down. This can cause weirdness (usually intermittent name errors during shutdown) when __del__ methods directly or indirectly reference module globals. Ah. I'm familiar with this problem. My approach was to install an exit handler that ensures that all pending futures are complete and all threads and processes exit before allowing the interpreter to exit. Cheers, Brian One of the easiest ways to avoid that is to put the state on a singleton object, then give the affected classes a reference to that object. Cheers, Nick. P.S. This problem is actually the reason we don't have a context manager for temporary directories yet. Something that should have been simple became a twisty journey down the rabbit hole: http://bugs.python.org/issue5178 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 10 Mar 2010, at 08:32, Dj Gilcrease wrote: On Mon, Mar 8, 2010 at 2:11 PM, exar...@twistedmatrix.com wrote: Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. Ok the new patch is submitted @ http://code.google.com/p/pythonfutures/issues/detail?id=1 Cool, thanks. *note there are 2 tests that fail and 1 test that dead locks on windows even without this patch, the deadlock test I am skipping in the patch and the two that fail do so for a reason that does not make sense to me. I'll investigate but I don't have convenient access to a windows machine. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 08:39, Greg Ewing wrote: Terry Reedy wrote: Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Seems to me that if you care about the order of the results, you should be able to just wait for each result separately in the order you want them. Something like task1 = start_task(proc1) task2 = start_task(proc2) task3 = start_task(proc3) result1 = task1.wait_for_result() result2 = task2.wait_for_result() result3 = task3.wait_for_result() You can write this as: executor = ... future1 = executor.submit(proc1) future2 = executor.submit(proc2) future3 = executor.submit(proc3) result1 = task1.result() result2 = task2.result() result3 = task3.result() This would also be a natural way to write things even if you don't care about the order, but you need all the results before proceeding. You're going to be held up until the longest-running task completes anyway, so it doesn't matter if some of them finish earlier and have to sit around waiting for you to collect the result. Often you don't want to continue if there is a failure. In the example that you gave, if proc3 raises an exception immediately, you still wait for proc1 and proc2 to complete even though you will end up discarding their results. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 08:11, exar...@twistedmatrix.com wrote: On 08:56 pm, digitalx...@gmail.com wrote: On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease digitalx...@gmail.com wrote: A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. So I decided to play with this design a little and since I made it a singleton I decided to place all the thread/process tracking and exit handle code in it instead of having the odd semi-global scoped _shutdown, _thread_references, _remove_dead_thread_references and _python_exit objects floating around in each executor file, seems to work well. The API would be from concurrent.futures import executors executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default To create your own executor you create your executor class and add the following at the end Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. I'm not sure what you mean, could you clarify? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 03:21, Terry Reedy wrote: On 3/6/2010 4:20 AM, Brian Quinlan wrote: On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com mailto:br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Also, the PEP should probably link to the discussions on stdlib-sig? I thoug ht about that but this discussion is spread over many threads and many months. This is pretty typical. I would say just that, and link to the first. This PEP was discussed over many months in many threads in the stdlib-sig list. The first was . Python-dev discussion occured in this thread. I'll add that. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. I'm not sure what you mean, could you clarify? Assuming your question refers to the second sentence, Jean-Paul is referring to a trick of the CPython interpreter when it terminates. To maximise the chances of objects being deleted properly rather than just dumped from memory when the process exits, module dictionaries are filled with None values before the interpreter shuts down. This can cause weirdness (usually intermittent name errors during shutdown) when __del__ methods directly or indirectly reference module globals. One of the easiest ways to avoid that is to put the state on a singleton object, then give the affected classes a reference to that object. Cheers, Nick. P.S. This problem is actually the reason we don't have a context manager for temporary directories yet. Something that should have been simple became a twisty journey down the rabbit hole: http://bugs.python.org/issue5178 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
s...@pobox.com wrote: I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) Nick As Guido said, let's call the nice task queuing system futures Nick and point people wanting a full-power asynchronous process model Nick to Twisted Can this module at least be pushed down into a package? I think concurrent or concurrency were both suggested at one point. Yep, I believe concurrent.futures was picked as the name elsewhere in the thread. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Le Mon, 08 Mar 2010 21:11:45 -, exar...@twistedmatrix.com a écrit : Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. +1. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 3/8/2010 4:39 PM, Greg Ewing wrote: Terry Reedy wrote: Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Seems to me that if you care about the order of the results, you should be able to just wait for each result separately in the order you want them. Something like task1 = start_task(proc1) task2 = start_task(proc2) task3 = start_task(proc3) result1 = task1.wait_for_result() result2 = task2.wait_for_result() result3 = task3.wait_for_result() *If* I understand the first example correctly, this is effectively what the first example does with two loops. But I was hoping for clarification and amplification in the PEP. This would also be a natural way to write things even if you don't care about the order, but you need all the results before proceeding. You're going to be held up until the longest-running task completes anyway, so it doesn't matter if some of them finish earlier and have to sit around waiting for you to collect the result. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Mon, Mar 8, 2010 at 2:11 PM, exar...@twistedmatrix.com wrote: Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. Ok the new patch is submitted @ http://code.google.com/p/pythonfutures/issues/detail?id=1 *note there are 2 tests that fail and 1 test that dead locks on windows even without this patch, the deadlock test I am skipping in the patch and the two that fail do so for a reason that does not make sense to me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
P.J. Eby wrote: I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) As Guido said, let's call the nice task queuing system futures and point people wanting a full-power asynchronous process model to Twisted - while the Deferred API may technically be independent of the rest of the framework, you need at least some of the other tools for asynchronous operations to make it really shine. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Dj Gilcrease wrote: On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller jnol...@gmail.com wrote: Making the tests and examples happy on windows is fine; but some explanation is needed for the API changes. My primary motivation behind the API change is so there is just a single public Executor class that you tell what system to use instead of two separate classes. The use case I was thinking about is when a user is is unsure which system (threads or processes) they want to use so they just build the system with the defaults (which is threads) then it is a little easier to switch it to processes in the future instead of having to change imports and all instances of the class you just change the use keyword to switch between systems Wouldn't a factory function serve that purpose just as well? Or even just from concurrent.futures import ProcessPoolExecutor as TaskExecutor. That last form has the virtue that you can retrieve your executor from anywhere rather than being limited to the two provided by the concurrent.futures model. I think the string based approach actually unduly constrains the API despite superficially appearing to make it more flexible. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 3/8/2010 6:14 AM, Nick Coghlan wrote: P.J. Eby wrote: I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) As Guido said, let's call the nice task queuing system futures and I was confused by 'futures' also until Philip explained it as task-queue or task-pool, and hence also do not like it. Since the examples in the PEP do *NOT* give example output, it was not clear to me whether execution or the termination thereof is ordered (queue) or not (pool). Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Adding 'will print ...' and 'might print ...' outputs would help. point people wanting a full-power asynchronous process model to Twisted That could be done in the PEP to clarify its scope. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 3/6/2010 4:20 AM, Brian Quinlan wrote: On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com mailto:br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Also, the PEP should probably link to the discussions on stdlib-sig? I thoug ht about that but this discussion is spread over many threads and many months. This is pretty typical. I would say just that, and link to the first. This PEP was discussed over many months in many threads in the stdlib-sig list. The first was . Python-dev discussion occured in this thread. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Mon, Mar 8, 2010 at 4:25 AM, Nick Coghlan ncogh...@gmail.com wrote: Wouldn't a factory function serve that purpose just as well? Or even just from concurrent.futures import ProcessPoolExecutor as TaskExecutor. That last form has the virtue that you can retrieve your executor from anywhere rather than being limited to the two provided by the concurrent.futures model. I think the string based approach actually unduly constrains the API despite superficially appearing to make it more flexible. mm you are correct, I went with the string approach because I was experimenting with 3 additional executor types and wanted to be able to switch between or intermix them them without having to change imports and didnt feel like writing a register class with a factory method. A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. Would this style be better or would it be better to just leave it with the two executor classes? I tend to dislike multiple classes for a what is essentially a Strategy of a concept and factories are something I tend to forget about until well after my initial idea has formed into a proof of concept. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) Nick As Guido said, let's call the nice task queuing system futures Nick and point people wanting a full-power asynchronous process model Nick to Twisted Can this module at least be pushed down into a package? I think concurrent or concurrency were both suggested at one point. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease digitalx...@gmail.com wrote: A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. So I decided to play with this design a little and since I made it a singleton I decided to place all the thread/process tracking and exit handle code in it instead of having the odd semi-global scoped _shutdown, _thread_references, _remove_dead_thread_references and _python_exit objects floating around in each executor file, seems to work well. The API would be from concurrent.futures import executors executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default To create your own executor you create your executor class and add the following at the end from concurrent.futures import executors, ExecutorBase class MyExecutor(ExecutorBase): ... executors.register(NAME, MyExecutor) It checks to see if your executor is a subclass of ExecutorBase, but only does a UserWarning if it is not since you should know what methods are required to be an executor like object, so if you are not subclassing ExecutorBase you should suppress the UserWarning before you register you class and un-suppress it after Some Helper Methods/Properties on the executors Singleton add_joinable_ref - This replaces the _thread_references.add system of tracking threads, and it allows for adding processes as well hence the joinable_ref name. It does check to make sure the ref being passes has a join method then creates a weakref to it and adds it to a set. For every thread or process your executor creates you should call this with so it will be tracked properly cleanup_joinable_refs - This replaces the _remove_dead_thread_references that had to be written for each executor individually. This should be called periodically, currently it is only called when you create a new executor since it is a blocking method (it uses a thread lock to make sure the set of references does not change while it is discarding old ones) shutdown - is a readonly property and replaces the _shutdown global var that had to be created for each executor individually, it is set in the executors destructor __del__ - replaces the _python_exit method that had to be written for each executor individually If this API change isnt accepted its no big deal, since I am only changing it due to personal pet peeves and the only real issue I had with the package was scoping which has already been addressed by adding it to a concurrent package ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 08:56 pm, digitalx...@gmail.com wrote: On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease digitalx...@gmail.com wrote: A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. So I decided to play with this design a little and since I made it a singleton I decided to place all the thread/process tracking and exit handle code in it instead of having the odd semi-global scoped _shutdown, _thread_references, _remove_dead_thread_references and _python_exit objects floating around in each executor file, seems to work well. The API would be from concurrent.futures import executors executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default To create your own executor you create your executor class and add the following at the end Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Terry Reedy wrote: Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Seems to me that if you care about the order of the results, you should be able to just wait for each result separately in the order you want them. Something like task1 = start_task(proc1) task2 = start_task(proc2) task3 = start_task(proc3) result1 = task1.wait_for_result() result2 = task2.wait_for_result() result3 = task3.wait_for_result() This would also be a natural way to write things even if you don't care about the order, but you need all the results before proceeding. You're going to be held up until the longest-running task completes anyway, so it doesn't matter if some of them finish earlier and have to sit around waiting for you to collect the result. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Dj Gilcrease wrote: executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default from concurrent.futures import executors, ExecutorBase class MyExecutor(ExecutorBase): ... executors.register(NAME, MyExecutor) I don't understand the reason for using a registration system rather than just importing names from a module. You mentioned wanting to globally change the executor class being used by a program without having to make changes throughout. Registering a different class under the same name would be one way to do that, but you could achieve the same thing just by assigning to a name in a module. In other words, instead of inventing your own mechanism for managing a namespace, just use a module as your namespace. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sat, Mar 6, 2010 at 10:09 PM, Dj Gilcrease digitalx...@gmail.com wrote: After playing with the API for a while running into many issues with the examples tests crashing windows I decided to modify the API a little and fix up the examples so they dont crash windows based computers. http://code.google.com/p/pythonfutures/issues/detail?id=1 API Change that changes the current Executor to ExecutorBase and adds a new Executor class that is used like futures.Executor() # creates an executor that uses threading and a max_workers = to the number of cpus futures.Executor(use='process') # Creates an executor that uses multiprocessing and a max_workers = to the number of cpus futures.Executor(max_workers=5) # threading again, just specifying the number of workers futures.Executor(use='process', max_workers=5) # back to multiprocessing, but with the max_workers specified Making the tests and examples happy on windows is fine; but some explanation is needed for the API changes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: P.J. Eby wrote: (Personally, I think it would be better to just drop the ambitious title and scope, and go for the nice task queue scope. I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) This may be a terminology thing - to me futures *are* just a nice way to handle farming tasks out to worker threads or processes. You seem to see them as something more comprehensive than that. Actual futures are, yes. Specifically, futures are a mechanism for asynchronous computation, whereas the PEP seems to be all about synchronously managing parallel tasks. That's a huge difference. Technically, the things in the PEP (and by extension, Java's futures) match the letter of the definition of a future, but not (IMO) the spirit. There's no clean way to compose them, and at base they're more about parallelism than asynchrony. I agree the PEP should just target what the current implementation provides and put whatever scope limitations are needed in the preamble text to make that clear. Yep. I'm just saying parallel task queueing is a much better description of what the implementation is/does, and would suggest renaming Future - Task and Executor - WorkerPool or some such. These names would be *much* clearer to people who've never heard of futures, as well as more appropriate to the actual scope of what this does. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby p...@telecommunity.com wrote: At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: P.J. Eby wrote: (Personally, I think it would be better to just drop the ambitious title and scope, and go for the nice task queue scope. I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) This may be a terminology thing - to me futures *are* just a nice way to handle farming tasks out to worker threads or processes. You seem to see them as something more comprehensive than that. Actual futures are, yes. Specifically, futures are a mechanism for asynchronous computation, whereas the PEP seems to be all about synchronously managing parallel tasks. That's a huge difference. Technically, the things in the PEP (and by extension, Java's futures) match the letter of the definition of a future, but not (IMO) the spirit. There's no clean way to compose them, and at base they're more about parallelism than asynchrony. Do you have an example of a language or library that uses the term future to refer to what you're talking about? I'm curious to see what it looks like. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby p...@telecommunity.com wrote: At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: P.J. Eby wrote: (Personally, I think it would be better to just drop the ambitious title and scope, and go for the nice task queue scope. Â I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) This may be a terminology thing - to me futures *are* just a nice way to handle farming tasks out to worker threads or processes. You seem to see them as something more comprehensive than that. Actual futures are, yes. Â Specifically, futures are a mechanism for asynchronous computation, whereas the PEP seems to be all about synchronously managing parallel tasks. Â That's a huge difference. Technically, the things in the PEP (and by extension, Java's futures) match the letter of the definition of a future, but not (IMO) the spirit. Â There's no clean way to compose them, and at base they're more about parallelism than asynchrony. Do you have an example of a language or library that uses the term future to refer to what you're talking about? I'm curious to see what it looks like. The wikipedia page menetioned earlier in the discussion has several examples. Twisted's Deferreds are probably the best example of such a system in Python, though, as far as I know. The features proposed in the PEP are basically akin to Twisted's deferToThread(), i.e. return a future for running this code in some other thread. However, you could easily make your own thread or process pool for executing tasks, using Deferreds as your future objects; deferToThread() is not particularly special, it's just one possible executor, to use the PEP's terminology. Likewise, deferreds are not limited to representing parallel task execution - they can also represent asynchronous events such as incoming I/O or a waited-for mouse click. Deferreds allow composing calculation by stacking callbacks (in continuation-passing style), but they're most Pythonic (IMO) when used with a coroutine wrapping system like Eventlet or my earlier peak.events framework, so that you can use them in yield expressions. That way you can write synchronous-looking code that nonetheless executes asynchronously. Twisted uses Deferreds extensively, both for I/O and IPC, and of course for database access. However, the 'defer' module itself is conceptually distinct from both the I/O system, event loop, and threads: it's a largely-generic framework for managing asynchronous computation. Parallel task queueing is simply *one* possible application of Deferreds; their actual purpose is to provide a common API for working with values that aren't yet known. That is to say, futures. In contrast, the PEP manages only a very limited sort of future - the completion of a single task, which must be synchronously waited for. If one future in the PEP needs to wait for another, you tie up that thread or process in a busy loop... which rather limits how much parallelism you can have. Deferreds also allow you to write code that doesn't *care* whether the value has been determined or not... you can simply stack on what code you want to have run when the value is available, and it runs immediately if the value is there. Likewise, a deferred is not tied to the execution of a single piece of code - it's more of a rendezvous or synchronization point, that can be given a value to fire with. (In contrast, the PEP's futures are tightly coupled to a task execution model: they have a notion of running and being cancelled, for example.) For these reasons, I'd say that Deferreds are much more deserving of the title futures, especially since there is no need to call a simple task queue a futures system. (Unless you're Java, I suppose, in which case you might want the name for marketing reasons. ;-) But in Python I assume we'd like to call a spade a spade, and a task queue a task queue.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sun, 07 Mar 2010 10:48:09 -0500, P.J. Eby p...@telecommunity.com wrote: At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: I agree the PEP should just target what the current implementation provides and put whatever scope limitations are needed in the preamble text to make that clear. Yep. I'm just saying parallel task queueing is a much better description of what the implementation is/does, and would suggest renaming Future - Task and Executor - WorkerPool or some such. These names would be *much* clearer to people who've never heard of futures, as well as more appropriate to the actual scope of what this does. For what it's worth: I don't have any particular knowledge in this area. I did loosely follow the stdlib-sig discussion. I wasn't really sure exactly what the module was about or what a 'future' was, or why I would want to use one. I did get that it was about parallel execution of tasks, but it seemed like there had to be more to it than that. Hearing it called a 'worker pool' makes a lightbulb go off and I can now understand why this would be a useful facility to have in the standard library. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sun, Mar 7, 2010 at 9:57 AM, P.J. Eby p...@telecommunity.com wrote: At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: Do you have an example of a language or library that uses the term future to refer to what you're talking about? I'm curious to see what it looks like. The wikipedia page menetioned earlier in the discussion has several examples. Twisted's Deferreds are probably the best example of such a system in Python, though, as far as I know. The features proposed in the PEP are basically akin to Twisted's deferToThread(), i.e. return a future for running this code in some other thread. Nearly all of the wikipedia page's examples refer to exactly this kind of system, but with syntax to support it instead of putting it in a library. The distinction between futures and deferreds is that futures are designed to block, while deferreds are designed to continue through callbacks. We could say that futures are to deferreds as threading.Lock is to mutex.mutex. That way you can write synchronous-looking code that nonetheless executes asynchronously. Asynchronous in the PEP refers to how the task behind the future runs, not how users wait for the future. Is there a better way to say that? Twisted uses Deferreds extensively, both for I/O and IPC, and of course for database access. However, the 'defer' module itself is conceptually distinct from both the I/O system, event loop, and threads: it's a largely-generic framework for managing asynchronous computation. Parallel task queueing is simply *one* possible application of Deferreds; their actual purpose is to provide a common API for working with values that aren't yet known. That is to say, futures. In contrast, the PEP manages only a very limited sort of future - the completion of a single task, which must be synchronously waited for. If one future in the PEP needs to wait for another, you tie up that thread or process in a busy loop... which rather limits how much parallelism you can have. So is it that you just don't like the idea of blocking, and want to stop anything that relies on it from getting into the standard library? Given the set_result and set_exception methods, it's pretty straightforward to fill in the value of a future from something that isn't purely computational. Given a way to register on-done callbacks with the future, it would be straightforward to wait for a future without blocking, too. Jeffrey ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: So is it that you just don't like the idea of blocking, and want to stop anything that relies on it from getting into the standard library? Um, no. As I said before, call it a parallel task queue or parallel task manager or something to that general effect and I'm on board. It may not be in the Zen of Python, but ISTM that names should generally follow use cases. It is something of a corollary to one obvious way to do it, in that if you see something whose name matches what you want to do, then it should be obvious that that's the way in question. ;-) The use cases for parallel task queues, however, are a subset of those for futures in the general case. Since the proposed module addresses most of the former but very little of the latter, calling it futures is inappropriate. Specifically, it's: 1. Confusing to people who don't know what futures are (see e.g R.D. Murray's post), and 2. Underpowered for people who expect/want a more fully-featured futures system along the lines of E or Deferreds. It seems that the only people for whom it's an intuitively correct description are people who've only had experience with more limited futures models (like Java's). However, these people should not have a problem understanding the notion of parallel task queueing or task management, so changing the name isn't really a loss for them, and it's a gain for everybody else. Given the set_result and set_exception methods, it's pretty straightforward to fill in the value of a future from something that isn't purely computational. Those are described as internal methods in the PEP; by contrast, the Deferred equivalents are part of the public API. Given a way to register on-done callbacks with the future, it would be straightforward to wait for a future without blocking, too. Yes, and with a few more additions besides that one, you might be on the way to an actual competitor for Deferreds. For example: retry support, chaining, logging, API for transparent result processing, coroutine support, co-ordination tools like locks, sempaphores and queues, etc. These are all things you would very likely want or need if you actually wanted to write a program using futures as *your main computational model*, vs. just needing to toss out some parallel tasks in a primarily synchronous program. Of course, Deferreds are indeed overkill if all you're ever going to want is a few parallel tasks, unless you're already skilled in using Twisted or some wrapper for it. So, I totally support having a simple task queue in the stdlib, as there are definitely times I would've used such a thing for a quick script, if it were available. However, I've *also* had use cases for using futures as a computational model, and so that's what I originally thought this PEP was about. After the use cases were clarified, though, it seems to me that *calling* it futures is a bad idea, because it's really just a nice task queuing system. I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller jnol...@gmail.com wrote: Making the tests and examples happy on windows is fine; but some explanation is needed for the API changes. My primary motivation behind the API change is so there is just a single public Executor class that you tell what system to use instead of two separate classes. The use case I was thinking about is when a user is is unsure which system (threads or processes) they want to use so they just build the system with the defaults (which is threads) then it is a little easier to switch it to processes in the future instead of having to change imports and all instances of the class you just change the use keyword to switch between systems ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sun, Mar 7, 2010 at 11:56 AM, P.J. Eby p...@telecommunity.com wrote: At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote: So is it that you just don't like the idea of blocking, and want to stop anything that relies on it from getting into the standard library? Um, no. As I said before, call it a parallel task queue or parallel task manager or something to that general effect and I'm on board. It may not be in the Zen of Python, but ISTM that names should generally follow use cases. It is something of a corollary to one obvious way to do it, in that if you see something whose name matches what you want to do, then it should be obvious that that's the way in question. ;-) The use cases for parallel task queues, however, are a subset of those for futures in the general case. Since the proposed module addresses most of the former but very little of the latter, calling it futures is inappropriate. Specifically, it's: 1. Confusing to people who don't know what futures are (see e.g R.D. Murray's post), and This is not a problem. We will document what we consider a future. 2. Underpowered for people who expect/want a more fully-featured futures system along the lines of E or Deferreds. This sounds like an underhanded slur towards the PEP. It seems that the only people for whom it's an intuitively correct description are people who've only had experience with more limited futures models (like Java's). However, these people should not have a problem understanding the notion of parallel task queueing or task management, so changing the name isn't really a loss for them, and it's a gain for everybody else. I expect that the majority of Python users fall either in camp #1 (never heard of futures, will be happy to learn about what Python calls futures) or camp #3 (have used Java futures). The users of E can be counted on a few hands. Deferreds are used heavily in some Python circles but most Python users (myself included) have at most a very vague idea of them. Also, as you clarify below, Deferreds are so much more powerful that they can't possibly be mistaken for futures (as defined by this PEP). Plus they already have a name. Given the set_result and set_exception methods, it's pretty straightforward to fill in the value of a future from something that isn't purely computational. Those are described as internal methods in the PEP; by contrast, the Deferred equivalents are part of the public API. Given a way to register on-done callbacks with the future, it would be straightforward to wait for a future without blocking, too. Yes, and with a few more additions besides that one, you might be on the way to an actual competitor for Deferreds. For example: retry support, chaining, logging, API for transparent result processing, coroutine support, co-ordination tools like locks, sempaphores and queues, etc. These are all things you would very likely want or need if you actually wanted to write a program using futures as *your main computational model*, vs. just needing to toss out some parallel tasks in a primarily synchronous program. Of course, Deferreds are indeed overkill if all you're ever going to want is a few parallel tasks, unless you're already skilled in using Twisted or some wrapper for it. So, I totally support having a simple task queue in the stdlib, as there are definitely times I would've used such a thing for a quick script, if it were available. However, I've *also* had use cases for using futures as a computational model, and so that's what I originally thought this PEP was about. After the use cases were clarified, though, it seems to me that *calling* it futures is a bad idea, because it's really just a nice task queuing system. I'm +1 on adding a nice task queuing system, -1 on calling it by any other name. ;-) So let's focus on the functionality of the task queuing system, and stick to roughly the functionality proposed in the PEP. The name is a non-issue and further discussion ought to be sent to null-...@python.org. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 6 Mar 2010, at 07:38, Brett Cannon wrote:
The PEP says that futures.wait() should only use keyword arguments
past its first positional argument, but the PEP has the function
signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.
Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?
Hi Brett,
That recommendation was designed to make it easy to change the API
without breaking code.
I'd don't think that recommendation makes sense anymore any I'll
update the PEP.
Cheers,
Brian
On Thu, Mar 4, 2010 at 22:03, Brian Quinlan br...@sweetapp.com
wrote:
Hi all,
I recently submitted a daft PEP for a package designed to make it
easier to execute Python functions asynchronously using threads and
processes. It lets the user focus on their computational problem
without having to build explicit thread/process pools and work queues.
The package has been discussed on stdlib-sig but now I'd like this
group's feedback.
The PEP lives here:
http://python.org/dev/peps/pep-3148/
Here are two examples to whet your appetites:
Determine if several numbers are prime.
import futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
for number, is_prime in zip(PRIMES, executor.map(is_prime,
PRIMES)):
print('%d is prime: %s' % (number, is_prime))
Print out the size of the home pages of various new sites (and
Fox News).
import futures
import urllib.request
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
'http://www.bbc.co.uk/',
'http://some-made-up-domain.com/']
def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()
with futures.ThreadPoolExecutor(max_workers=5) as executor:
# Create a future for each URL load.
future_to_url = dict((executor.submit(load_url, url, 60), url)
for url in URLS)
# Iterate over the futures in the order that they complete.
for future in futures.as_completed(future_to_url):
url = future_to_url[future]
if future.exception() is not None:
print('%r generated an exception: %s' % (url,
future.exception()))
else:
print('%r page is %d bytes' % (url, len(future.result(
Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/brett%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 6 Mar 2010, at 08:42, Jesse Noller wrote: If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. It doesn't really matter to me. We can either update this PEP to propose the concurrent.futures name or you can draft a more complete PEP that describes what other functionality should live in the concurrent package. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 6 Mar 2010, at 09:54, Antoine Pitrou wrote: Le Fri, 5 Mar 2010 17:03:02 +1100, Brian Quinlan br...@sweetapp.com a écrit : The PEP lives here: http://python.org/dev/peps/pep-3148/ Ok, here is my take on it: cancel() Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True. I think it shouldn't return anything, and raise an exception if cancelling failed. It is really an error condition, and ignoring the result doesn't seem right. In my experience with futures, canceling them is a best-effort optimization that people use when another future fails. For example: futures = [executor.submit(CopyDirectory, src, dest) for dest in ...] finished, unfinished = wait(futures, return_when=FIRST_EXCEPTION) # If there are unfinished futures then there must have been a failure for f in unfinished: # No reason to waste bandwidth copying files if the operation has already failed. f.cancel() for f in finished(): if f.exception(): raise f.exception() Future.running() Return True if the call is currently being executed and cannot be cancelled. Future.done() Return True if the call was successfully cancelled or finished running. These don't really make sense since the future is executing concurrently. By the time the result is returned, it can already be wrong. I advocate removing those two methods. There methods are useful for logging - by displaying the count of pending, running and completed futures you can estimate the progress of the system. The following Future methods are meant for use in unit tests and Executor implementations. Their names should then be preceded by an underscore '_'. We don't want people to think they are public APIs and start relying on them. Actually, as discussed on the stdlib-sig, these methods are designed to make it possible for users to implement their own Executors so we'll have keep the interface stable. wait(fs, timeout=None, return_when=ALL_COMPLETED) [...] This method should always be called using keyword arguments I don't think this is right. Keyword arguments are nice, but mandating them too often is IMO a nuisance (after all, it makes things longer to type and requires you to remember the exact parameter names). Especially when the method only takes at most 3 arguments. IMO, keyword-only arguments are mostly useful when there are a lot of positional arguments before, and you want to help the user use the right calling signature. I agree, I'll change this. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: On 6 Mar 2010, at 08:42, Jesse Noller wrote: If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. It doesn't really matter to me. We can either update this PEP to propose the concurrent.futures name or you can draft a more complete PEP that describes what other functionality should live in the concurrent package. I think a concurrent.futures name works - it gives the scoping desired by the folks with an finance background and gives us a bucket for future thread/process agnostic concurrency tools (such as a pools and message queues). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Jeffrey Yasskin writes: It seems like a good idea to follow the choice other languages have used for the name (if they tend to agree) For the record, I've conceded that point. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: IOW, as far as I can tell from the PEP, it doesn't look like you can compose futures without *global* knowledge of the application... and in and of itself, this seems to negate the PEP's own motivation to prevent duplication of parallel execution handling! That is, if I use code from module A and module B that both want to invoke tasks asynchronously, and I want to invoke A and B asynchronously, what happens? Based on the design of the API, it appears there is nothing you can do except refactor A and B to take an executor in a parameter, instead of creating their own. A and B could both use their own executor instances. You would need to refactor A and B if you wanted to manage thread and process counts globally. You may want to consider providing global thread and process executors in the futures module itself. Code which just wants to say do this in the background without having to manage the lifecycle of its own executor instance is then free to do so. I've had a lot of experience with a framework that provides this and it is *very* convenient (it's also a good way to avoid deadlocks due to synchronous notification APIs). On PJE's broader point, async event loops with non-blocking I/O and messages passed back to the event loop to indicate completion of operations and relying on threads and processes to farm out tasks (which you will later block on in order to retrieve the results) are completely different programming models. This PEP doesn't change that - it just makes certain aspects of the latter approach easier to handle. Trying to design an API that can cope with either model strikes me as a fool's errand. They differ at such a fundamental level that I don't see how a hybrid API could be particularly optimal for either approach. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sat, Mar 6, 2010 at 6:43 AM, Nick Coghlan ncogh...@gmail.com wrote: You may want to consider providing global thread and process executors in the futures module itself. Code which just wants to say do this in the background without having to manage the lifecycle of its own executor instance is then free to do so. +1 -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
I have been playing with the feedback branch of this package for py3 and there seems to be a rather serious bug in the Process version. Using the code @ http://dpaste.com/hold/168795/ When I was running in debug mode I found that as soon as p = multiprocessing.Process( target=_process_worker, args=(self._call_queue, self._result_queue, self._shutdown_process_event)) was called (yes even before p.start() was called) the processes just started launching all by themselves. I am also wondering why you are launching the process directly instead of using a Pool since you are limiting the number of processes always wouldnt it be better to launch the worker processes up front then just add worker items to the queue? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
I am also getting an odd error on odd error on exit Error in atexit._run_exitfuncs: TypeError: print_exception(): Exception expected for value, str found ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Mar 6, 2010, at 4:20 PM, Dj Gilcrease digitalx...@gmail.com wrote: I have been playing with the feedback branch of this package for py3 and there seems to be a rather serious bug in the Process version. Using the code @ http://dpaste.com/hold/168795/ When I was running in debug mode I found that as soon as p = multiprocessing.Process( target=_process_worker, args=(self._call_queue, self._result_queue, self._shutdown_process_event)) was called (yes even before p.start() was called) the processes just started launching all by themselves. Did you run the provided example code on windows by chance? If so, look at the multiprocessing docs, there are restrictions on windows (see the __main__ note) - not following the guidelines can result in lots of processes spawning. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Stephen J. Turnbull step...@xemacs.org writes: I have to admit Jean-Paul's explanation a pretty convincing reason for adopting future rather than promise. But I'm with Skip, I would prefer that the module be named future rather than futures. Has anyone in this very long thread raised the issue that Python *already* uses this term for the name of a module with a totally unrelated purpose; the ‘__future__’ pseudo-module? That alone seems a pretty strong reason to avoid the word “future” (singular or plural) for some other module name. -- \ “Creativity can be a social contribution, but only in so far as | `\society is free to use the results.” —Richard Stallman | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Mar 6, 2010, at 5:47 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Stephen J. Turnbull step...@xemacs.org writes: I have to admit Jean-Paul's explanation a pretty convincing reason for adopting future rather than promise. But I'm with Skip, I would prefer that the module be named future rather than futures. Has anyone in this very long thread raised the issue that Python *already* uses this term for the name of a module with a totally unrelated purpose; the ‘__future__’ pseudo-module? That alone seems a pretty strong reason to avoid the word “future” (singular or plural) for some other module name. Yes, they have, and putting it in a sub namespace has also come up. In the thread. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Jeffrey Yasskin wrote: The caller can't avoid the error here by querying the future, because of the problem you point out below, so I'm inclined to think that the future was already started should be a return value rather than an exception (although that may be my C++ background showing through). I think it's your C++ background showing. In Python philosophy, there's no particlular connection between whether something can be tested for and whether it should raise an exception. The thing to consider, I think, is whether it makes sense in a large proportion of use cases to ignore the fact that the cancel failed and carry on regardless. If not, then raising an exception makes it much harder to accidentally ignore the situation. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Phillip J. Eby wrote: while at the same time creating yet another alternative (and mutually incompatible) event loop system in the stdlib, beyond the ones that are already in asyncore, tkinter, and the various SocketServer subclasses. Aaargh... that's the *last* thing we need! I've been thinking for a while that it would be a big help if there were one, standardised module in the stdlib for handling async events, and all the other gui toolkits etc. were made to use it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 06/03/2010 23:37, Greg Ewing wrote: Phillip J. Eby wrote: while at the same time creating yet another alternative (and mutually incompatible) event loop system in the stdlib, beyond the ones that are already in asyncore, tkinter, and the various SocketServer subclasses. Aaargh... that's the *last* thing we need! I've been thinking for a while that it would be a big help if there were one, standardised module in the stdlib for handling async events, and all the other gui toolkits etc. were made to use it. Wouldn't it have to be the Tcl event loop then? Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: That recommendation was designed to make it easy to change the API without breaking code. I'd don't think that recommendation makes sense anymore any I'll update the PEP. I don't think there's anything wrong with stating that the order of the arguments is not a guaranteed part of the API. There isn't necessarily any need to enforce that, though. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sat, Mar 6, 2010 at 5:38 PM, Michael Foord fuzzy...@voidspace.org.ukwrote: On 06/03/2010 23:37, Greg Ewing wrote: I've been thinking for a while that it would be a big help if there were one, standardised module in the stdlib for handling async events, and all the other gui toolkits etc. were made to use it. Wouldn't it have to be the Tcl event loop then? I image he means a standardized Abstract Base Class, which each GUI toolkit would subclass. That's more or less how Twisted's reactors work. That way non-GUI async code can just use the ABC and not worry about what event loop is running underneath (be it TCL, GTK, or just poll()). -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Michael Foord wrote: Wouldn't it have to be the Tcl event loop then? No, tcl/tk would have to be threatened with the comfy chair until it allowed itself to be spliced into the official event loop somehow. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Ben Has anyone in this very long thread raised the issue that Python Ben *already* uses this term for the name of a module with a totally Ben unrelated purpose; the â__future__â pseudo-module? Yes, it's already come up. While not quite the same it does remind me of the __builtin__/__builtins__ confusion. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sat, Mar 6, 2010 at 2:58 PM, Jesse Noller jnol...@gmail.com wrote: Did you run the provided example code on windows by chance? If so, look at the multiprocessing docs, there are restrictions on windows (see the __main__ note) - not following the guidelines can result in lots of processes spawning. Yes, on win7. I might recommend then that the examples in the PEP be restructured to work correctly on windows by including the if __name__ == '__main__': ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: At 05:32 AM 3/6/2010, Brian Quinlan wrote: Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Twisted's Deferred API has nothing to do with I/O. I see, you just mean the API and not the underlying model. We discussed the Deferred API on the stdlib-sig and I don't think that anyone expressed a preference for it over the one described in the PEP. Do you have any concrete criticism? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 02:10 am, br...@sweetapp.com wrote: On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: At 05:32 AM 3/6/2010, Brian Quinlan wrote: Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Twisted's Deferred API has nothing to do with I/O. I see, you just mean the API and not the underlying model. We discussed the Deferred API on the stdlib-sig and I don't think that anyone expressed a preference for it over the one described in the PEP. Do you have any concrete criticism? From reading some of the stdlib-sig archives, it sounds like there is general agreement that Deferreds and Futures can be used to complement each other, and that getting code that is primarily Deferred-based to integrate with Future-based code or vice versa should eventually be possible. Do I have the right sense of people's feelings? And relatedly, once Futures are accepted and implemented, are people going to use them as an argument to exclude Deferreds from the stdlib (or be swayed by other people making such arguments)? Hopefully not, given what I read on stdlib-sig, but it doesn't hurt to check... Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Sat, Mar 6, 2010 at 9:34 PM, exar...@twistedmatrix.com wrote: On 02:10 am, br...@sweetapp.com wrote: On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: At 05:32 AM 3/6/2010, Brian Quinlan wrote: Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Twisted's Deferred API has nothing to do with I/O. I see, you just mean the API and not the underlying model. We discussed the Deferred API on the stdlib-sig and I don't think that anyone expressed a preference for it over the one described in the PEP. Do you have any concrete criticism? From reading some of the stdlib-sig archives, it sounds like there is general agreement that Deferreds and Futures can be used to complement each other, and that getting code that is primarily Deferred-based to integrate with Future-based code or vice versa should eventually be possible. Do I have the right sense of people's feelings? And relatedly, once Futures are accepted and implemented, are people going to use them as an argument to exclude Deferreds from the stdlib (or be swayed by other people making such arguments)? Hopefully not, given what I read on stdlib-sig, but it doesn't hurt to check... Jean-Paul Generally speaking; I don't see futures as an exclusion to Deferreds, or other asynchronous doodads. I just see it as a useful construct on top of threads and processes primarily. So in my mind, no. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
After playing with the API for a while running into many issues with the examples tests crashing windows I decided to modify the API a little and fix up the examples so they dont crash windows based computers. http://code.google.com/p/pythonfutures/issues/detail?id=1 API Change that changes the current Executor to ExecutorBase and adds a new Executor class that is used like futures.Executor() # creates an executor that uses threading and a max_workers = to the number of cpus futures.Executor(use='process') # Creates an executor that uses multiprocessing and a max_workers = to the number of cpus futures.Executor(max_workers=5) # threading again, just specifying the number of workers futures.Executor(use='process', max_workers=5) # back to multiprocessing, but with the max_workers specified ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 01:10 PM 3/7/2010 +1100, Brian Quinlan wrote: On 7 Mar 2010, at 03:04, Phillip J. Eby wrote: At 05:32 AM 3/6/2010, Brian Quinlan wrote: Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Twisted's Deferred API has nothing to do with I/O. I see, you just mean the API and not the underlying model. We discussed the Deferred API on the stdlib-sig and I don't think that anyone expressed a preference for it over the one described in the PEP. Do you have any concrete criticism? Of the PEP, yes, absolutely, and I've already stated much of it. My quibbles are with the PEP *itself*, not so much the API or implementation. I think that said API and implementation is fine, but FAR too narrowly scoped to claim to be futures or execute computations asynchronously, as the PEP calls it. It's really just a nice task queuing system. Now, if the PEP were *scoped* as such, i.e., hey, let's just have a nice multithread/multiprocess task queuing implementation in the stdlib, I would be +1. It's a handy utility to have. But I think that the scope given by the PEP appears overly ambitious compared to what is actually being delivered; this seems less of a futures API and more like a couple of utility functions for waiting on threads and processes. To rise to the level of an API, it seems to me that it would need to address interop with coroutines and async frameworks, where the idea of futures seems much more relevant than simple synchronous-but-parallel scripts. (It should also have better tools for working with futures asynchronously, because, hey, it says right there in the title, execute computations asynchronously.) Anyway, I'd like to see the answers to (at *least*) the following issues fleshed out in the PEP, if you want it to really be a futures API, vs. nice task queue in the stdlib: * Compare/contrast alternatives now available * Address the issue of competing event loops and sharing/passing executors among code * Either offer a way for executed code to re-enter its own executor (e.g. via an optional parameter), or explain why this was considered and rejected * Address interoperability with coroutines and async frameworks, or clearly explain why such is out of scope (Personally, I think it would be better to just drop the ambitious title and scope, and go for the nice task queue scope. I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Greg Ewing wrote: The thing to consider, I think, is whether it makes sense in a large proportion of use cases to ignore the fact that the cancel failed and carry on regardless. If not, then raising an exception makes it much harder to accidentally ignore the situation. Cancelling operations is generally a best effort activity - having to wrap it an exception handler would be annoying. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
P.J. Eby wrote: (Personally, I think it would be better to just drop the ambitious title and scope, and go for the nice task queue scope. I imagine, too, that in that case Jean-Paul wouldn't need to worry about it being raised as a future objection to Deferreds or some such getting into the stdlib.) This may be a terminology thing - to me futures *are* just a nice way to handle farming tasks out to worker threads or processes. You seem to see them as something more comprehensive than that. I agree the PEP should just target what the current implementation provides and put whatever scope limitations are needed in the preamble text to make that clear. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
A young library solving an old problem in a way that conflicts with
many of the other implementations available for years and with zero
apparent users in the wild is not an appropriate candidate for a PEP.
On Fri, Mar 5, 2010 at 1:03 AM, Brian Quinlan br...@sweetapp.com wrote:
Hi all,
I recently submitted a daft PEP for a package designed to make it easier to
execute Python functions asynchronously using threads and processes. It lets
the user focus on their computational problem without having to build
explicit thread/process pools and work queues.
The package has been discussed on stdlib-sig but now I'd like this group's
feedback.
The PEP lives here:
http://python.org/dev/peps/pep-3148/
Here are two examples to whet your appetites:
Determine if several numbers are prime.
import futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, is_prime))
Print out the size of the home pages of various new sites (and Fox
News).
import futures
import urllib.request
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
'http://www.bbc.co.uk/',
'http://some-made-up-domain.com/']
def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()
with futures.ThreadPoolExecutor(max_workers=5) as executor:
# Create a future for each URL load.
future_to_url = dict((executor.submit(load_url, url, 60), url)
for url in URLS)
# Iterate over the futures in the order that they complete.
for future in futures.as_completed(future_to_url):
url = future_to_url[future]
if future.exception() is not None:
print('%r generated an exception: %s' % (url,
future.exception()))
else:
print('%r page is %d bytes' % (url, len(future.result(
Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman ironfro...@gmail.com wrote: A young library solving an old problem in a way that conflicts with many of the other implementations available for years and with zero apparent users in the wild is not an appropriate candidate for a PEP. Baloney. A young library providing some syntactic sugar which uses primitives in the standard library to implement a common pattern is fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig list prior to bringing it here. By the same argument, we should shunt all of the recent unittest changes and improvements into space, since golly, other people did it, why should we. This is something relatively simple, which I would gladly add in an instant to the multiprocessing package - but Brian's one-upped me in that regard and is providing something which works with both threads and processes handily. Take a look at multiprocessing.Pool for example - all that is some sugar on top of the primitives, but it's good sugar, and is used by a fair number of people. Let me also state - my vision of where futures would live would be in a concurrent package - for example: from concurrent import futures The reason *why* is that I would like to also move the abstractions I have in multiprocessing *out* of that module, make them work with both threads and processes (if it makes sense) and reduce the multiprocessing module to the base primitive Process object. A concurrent package which implements common patterns built on top of the primitives we support is an objectively Good Thing. For example, how many of us have sat down and implemented a thread pool on top of threading, I would hazard to say that most of us who use threading have done this, and probably more than once. It stands to reason that this is a common enough pattern to include in the standard library. In any case; consider me a strong +1 to adding it. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
I revert my objections. I still would like to see this in use in the wild and I might even use it thusly, myself. On Fri, Mar 5, 2010 at 9:50 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman ironfro...@gmail.com wrote: A young library solving an old problem in a way that conflicts with many of the other implementations available for years and with zero apparent users in the wild is not an appropriate candidate for a PEP. Baloney. A young library providing some syntactic sugar which uses primitives in the standard library to implement a common pattern is fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig list prior to bringing it here. By the same argument, we should shunt all of the recent unittest changes and improvements into space, since golly, other people did it, why should we. This is something relatively simple, which I would gladly add in an instant to the multiprocessing package - but Brian's one-upped me in that regard and is providing something which works with both threads and processes handily. Take a look at multiprocessing.Pool for example - all that is some sugar on top of the primitives, but it's good sugar, and is used by a fair number of people. Let me also state - my vision of where futures would live would be in a concurrent package - for example: from concurrent import futures The reason *why* is that I would like to also move the abstractions I have in multiprocessing *out* of that module, make them work with both threads and processes (if it makes sense) and reduce the multiprocessing module to the base primitive Process object. A concurrent package which implements common patterns built on top of the primitives we support is an objectively Good Thing. For example, how many of us have sat down and implemented a thread pool on top of threading, I would hazard to say that most of us who use threading have done this, and probably more than once. It stands to reason that this is a common enough pattern to include in the standard library. In any case; consider me a strong +1 to adding it. jesse -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Jesse Noller wrote: The reason *why* is that I would like to also move the abstractions I have in multiprocessing *out* of that module, make them work with both threads and processes (if it makes sense) and reduce the multiprocessing module to the base primitive Process object. A concurrent package which implements common patterns built on top of the primitives we support is an objectively Good Thing. Yes, I've often thought we should have a pool model that covers threads as well as processes. The reason I think futures works as a PEP and potential standard library addition is that it is small enough to be readily reviewed and maintained, and serves as a useful building block for more complex usage. For a developer to get anything similar from a third party library is almost certainly going to require buying into a much heavier framework. A simple futures module provides a way to farm out worker tasks in a standard fashion without having to build as much of your own infrastructure every time. I've read the various PEP checkins as they went by on the checkins list - it gets a +0 from me (the only reason it isn't a +1 is because I personally tend to write with a Thread+Queue style. However, I could easily become a futures convert if they were readily available in the standard library) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Also, the PEP should probably link to the discussions on stdlib-sig? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Jesse Futures is a common term for this, and implemented named this in Jesse other languages. I don't think we should be adopting things that Jesse are common, and found elsewhere and then renaming them. Perhaps, but is it a common term for Python programmers (or the target population for Python)? I've never heard of it. futures to me are futures contracts in a trading environment (that's the industry I work in). No matter how well known the term is in the environment where it's used today you have to be sensitive to other meanings of the term. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 11:56 AM, s...@pobox.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Jesse Futures is a common term for this, and implemented named this in Jesse other languages. I don't think we should be adopting things that Jesse are common, and found elsewhere and then renaming them. Perhaps, but is it a common term for Python programmers (or the target population for Python)? I've never heard of it. futures to me are futures contracts in a trading environment (that's the industry I work in). No matter how well known the term is in the environment where it's used today you have to be sensitive to other meanings of the term. It's a common programming term. I don't think we should make a new name - I mean, how many different names for green threads / coroutines or flavors of those concepts are out there because someone painted the bike shed a different color? I mean, there's prior art here: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. Another common term for this is a promise. -- Curt Hagenlocher c...@hagenlocher.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller jnol...@gmail.com wrote: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html According to that link, Java has a module named Concurrent with an interface named Future. You're proposing a module named Futures with a class named Future. Why not name your module concurrent? That would eliminate the confusion with from __future__. I don't see a problem with keeping the class name. Plus, a concurrent module might be useful for things other than Futures, in the future. ;-) Just my 0.02 cents, -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Jesse http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html Without reading that I can assure you that not everybody has drunk the Java Kool-Aid. Just because Sun thought it was a fine term doesn't mean everyone else will. I've been a professional programmer for about 30 years, have never touched Java, and the only times I've seen its effect in Python I have been unimpressed (logging and unittest). Google for future. The Java doc is aways down on the front page. Now Google for futures. It's nowhere to be found. As far as I can tell, all but one hit on the first page are for my definition of futures. At the bottom of the page the searches related to _futures_ are all related to trading futures or other derivatives except one. That appears to be related to the LPGA futures tour. Java and its definition of the term is nowhere to be found. I suppose you can argue that it should be called future. I still contend that the name is *not* intuitive, no matter how well known it happens to be within the Java world. Don't name it futures though. That has nothing to do with common definitions of the term. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller jnol...@gmail.com wrote: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html According to that link, Java has a module named Concurrent with an interface named Future. You're proposing a module named Futures with a class named Future. Why not name your module concurrent? That would eliminate the confusion with from __future__. I don't see a problem with keeping the class name. Plus, a concurrent module might be useful for things other than Futures, in the future. ;-) Brian's module is named futures; I am +1'ing his proposal, and also suggesting we put it under concurrent/ package name. This means you would do the following: from concurrent import futures and in the future: from concurrent import pool And so on. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 05:06 pm, c...@hagenlocher.org wrote: On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. Another common term for this is a promise. Promises aren't exactly the same. This would be a particularly bad name to apply here. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 9:55 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller jnol...@gmail.com wrote: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html According to that link, Java has a module named Concurrent with an interface named Future. You're proposing a module named Futures with a class named Future. Why not name your module concurrent? That would eliminate the confusion with from __future__. I don't see a problem with keeping the class name. Plus, a concurrent module might be useful for things other than Futures, in the future. ;-) Brian's module is named futures; I am +1'ing his proposal, and also suggesting we put it under concurrent/ package name. This means you would do the following: from concurrent import futures and in the future: from concurrent import pool And so on. Future is a pretty standard CS term for this concept (as noted promise is another), and it wasn't invented by Java. I am not worried at all about confusion with __future__ (which is after all surrounded by dunders and only appears in one specific context). FWIW, there are two different ways of designing the API for such a concept. In one, the future or promise object masquerades as an instance of the eventual result, like a proxy, and accessing any attribute would block until the result is available. I don't like this design much; the masquerading is never perfect, and attribute accesses become risks of exceptions related to the promised work. In the other design, which this PEP proposes, there is an explicit call to get the value, which blocks as needed. I found a wikipedia article (http://en.wikipedia.org/wiki/Futures_and_promises#Implicit_vs_explicit) that calls the former implicit and the latter explicit. According to this article, the term promise was coined slightly older (by a year or so). I have no strong preference for either term; I suspect that if this were the only objection to the PEP Brian would be happy to rename futures to promises. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
s...@pobox.com wrote: Jesse http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html Without reading that I can assure you that not everybody has drunk the Java Kool-Aid. Just because Sun thought it was a fine term doesn't mean everyone else will. I've been a professional programmer for about 30 years, have never touched Java, and the only times I've seen its effect in Python I have been unimpressed (logging and unittest). Google for future. The Java doc is aways down on the front page. Now Google for futures. It's nowhere to be found. As far as I can tell, all but one hit on the first page are for my definition of futures. At the bottom of the page the searches related to _futures_ are all related to trading futures or other derivatives except one. That appears to be related to the LPGA futures tour. Java and its definition of the term is nowhere to be found. This term, the future, has been in use at least since 1985, even if the Web hasn't, and even if Google can't seem to separate this use from drivel about stock trading. See Halstead's 1985 TOPLAS paper on MultiLisp. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.97.1841 Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
http://home.pipeline.com/~hbaker1/Futures.html (1977) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 10:30 AM, exar...@twistedmatrix.com wrote: On 05:06 pm, c...@hagenlocher.org wrote: On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. Another common term for this is a promise. Promises aren't exactly the same. This would be a particularly bad name to apply here. Please explain. Even the Wikipedia article (http://en.wikipedia.org/wiki/Futures_and_promises), despite promising to explain the difference, didn't explain it. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 07:10 pm, gu...@python.org wrote: On Fri, Mar 5, 2010 at 10:30 AM, exar...@twistedmatrix.com wrote: On 05:06 pm, c...@hagenlocher.org wrote: On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. Another common term for this is a promise. Promises aren't exactly the same. This would be a particularly bad name to apply here. Please explain. Even the Wikipedia article (http://en.wikipedia.org/wiki/Futures_and_promises), despite promising to explain the difference, didn't explain it. The explicit futures on the wikipedia page seems to cover what is commonly referred to as a future. For example, Java's futures look like this. The implicit futures are what is generally called a promise. For example, E's promises look like this. Though the difference is mainly one of API, it turns out to make a significant difference in what you can accomplish. Promises are much more amenable to the pipelining optimization, for example. They're also much harder to implement in Python without core language changes. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 09:55, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller jnol...@gmail.com wrote: http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html According to that link, Java has a module named Concurrent with an interface named Future. You're proposing a module named Futures with a class named Future. Why not name your module concurrent? That would eliminate the confusion with from __future__. I don't see a problem with keeping the class name. Plus, a concurrent module might be useful for things other than Futures, in the future. ;-) Brian's module is named futures; I am +1'ing his proposal, and also suggesting we put it under concurrent/ package name. This means you would do the following: from concurrent import futures and in the future: from concurrent import pool And so on. So I don't quite get what you are after here. Are you wanting to eventually have a generic pool class that you can simply import and use that is always set to the best option for the platform? And as for moving stuff from multiprocessing into the concurrent namespace, are you thinking like concurrent.multiprocessing? I guess I am just trying to figure out what the abstraction is you are after in the package namespace. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. - -1 to the name from me as well: it isn't scoped properly to make it clear what the module is about. If they were inside a pacakge named 'concurrency' or some such (as hinted by Jesse Noller, I think), the clash would go away. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuRapAACgkQ+gerLs4ltQ7dBwCfRmMuq6X9VE8usYgSScXEA1D0 1PsAoI8MR5hjPjq9C7MFPTZhcO/T+NM4 =7wpK -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
The PEP says that futures.wait() should only use keyword arguments past its
first positional argument, but the PEP has the function signature as
``wait(fs, timeout=None, return_when=ALL_COMPLETED)``. Should it be
``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?
On Thu, Mar 4, 2010 at 22:03, Brian Quinlan br...@sweetapp.com wrote:
Hi all,
I recently submitted a daft PEP for a package designed to make it easier to
execute Python functions asynchronously using threads and processes. It lets
the user focus on their computational problem without having to build
explicit thread/process pools and work queues.
The package has been discussed on stdlib-sig but now I'd like this group's
feedback.
The PEP lives here:
http://python.org/dev/peps/pep-3148/
Here are two examples to whet your appetites:
Determine if several numbers are prime.
import futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, is_prime))
Print out the size of the home pages of various new sites (and Fox
News).
import futures
import urllib.request
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
'http://www.bbc.co.uk/',
'http://some-made-up-domain.com/']
def load_url(url, timeout):
return urllib.request.urlopen(url, timeout=timeout).read()
with futures.ThreadPoolExecutor(max_workers=5) as executor:
# Create a future for each URL load.
future_to_url = dict((executor.submit(load_url, url, 60), url)
for url in URLS)
# Iterate over the futures in the order that they complete.
for future in futures.as_completed(future_to_url):
url = future_to_url[future]
if future.exception() is not None:
print('%r generated an exception: %s' % (url,
future.exception()))
else:
print('%r page is %d bytes' % (url, len(future.result(
Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/brett%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 3:31 PM, Brett Cannon br...@python.org wrote: So I don't quite get what you are after here. Are you wanting to eventually have a generic pool class that you can simply import and use that is always set to the best option for the platform? And as for moving stuff from multiprocessing into the concurrent namespace, are you thinking like concurrent.multiprocessing? I guess I am just trying to figure out what the abstraction is you are after in the package namespace. -Brett My goal would be to put futures into a concurrent package - as it is an abstraction that allows for threads, and processes to be used. By default; I don't think we'd make a guess based on the platform, but rather pick a sane default (such as threads). After that; I would begin to remove chunks of multiprocessing (such as pool) and adapt it to the same type of pick threads or processes that futures uses. For example: from concurrent import Pool x = Pool(4, worker_primitive=Thread()) And so on. The end-goal would be to make concurrent.* a package containing common abstractions/patterns which can use threads or processes interchangeably. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 12:18 PM, exar...@twistedmatrix.com wrote: The explicit futures on the wikipedia page seems to cover what is commonly referred to as a future. For example, Java's futures look like this. The implicit futures are what is generally called a promise. For example, E's promises look like this. Fair enough, though the article confuses the matter by using the words more or less interchangeably. Though the difference is mainly one of API, it turns out to make a significant difference in what you can accomplish. Promises are much more amenable to the pipelining optimization, for example. They're also much harder to implement in Python without core language changes. That's why implicit futures (by any name) aren't on the table. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. - -1 to the name from me as well: it isn't scoped properly to make it clear what the module is about. If they were inside a pacakge named 'concurrency' or some such (as hinted by Jesse Noller, I think), the clash would go away. If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 1:42 PM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. - -1 to the name from me as well: it isn't scoped properly to make it clear what the module is about. If they were inside a pacakge named 'concurrency' or some such (as hinted by Jesse Noller, I think), the clash would go away. If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. A simple renaming of futures to concurrency.futures seems easy enough to swallow. (Though I haven't kept track of what other modules the PEP proposes.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Le Fri, 5 Mar 2010 17:03:02 +1100, Brian Quinlan br...@sweetapp.com a écrit : The PEP lives here: http://python.org/dev/peps/pep-3148/ Ok, here is my take on it: cancel() Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True. I think it shouldn't return anything, and raise an exception if cancelling failed. It is really an error condition, and ignoring the result doesn't seem right. Future.running() Return True if the call is currently being executed and cannot be cancelled. Future.done() Return True if the call was successfully cancelled or finished running. These don't really make sense since the future is executing concurrently. By the time the result is returned, it can already be wrong. I advocate removing those two methods. The following Future methods are meant for use in unit tests and Executor implementations. Their names should then be preceded by an underscore '_'. We don't want people to think they are public APIs and start relying on them. wait(fs, timeout=None, return_when=ALL_COMPLETED) [...] This method should always be called using keyword arguments I don't think this is right. Keyword arguments are nice, but mandating them too often is IMO a nuisance (after all, it makes things longer to type and requires you to remember the exact parameter names). Especially when the method only takes at most 3 arguments. IMO, keyword-only arguments are mostly useful when there are a lot of positional arguments before, and you want to help the user use the right calling signature. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jesse Noller wrote: On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach dan...@stutzbachenterprises.com wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan br...@sweetapp.com wrote: import futures +1 on the idea, -1 on the name. It's too similar to from __future__ import Futures is a common term for this, and implemented named this in other languages. I don't think we should be adopting things that are common, and found elsewhere and then renaming them. - -1 to the name from me as well: it isn't scoped properly to make it clear what the module is about. If they were inside a pacakge named 'concurrency' or some such (as hinted by Jesse Noller, I think), the clash would go away. If people agree with this; do you feel the proposal of said namespace should be a separate PEP, or piggy back on this? I don't want to piggy back on Brian's hard work. I'm just expressiong a preference for scoping the name, and don't want to preempt the process. If your proposed work on factoring common stuff out of multiprocessing would sit in the same conceptual space, then sharing the package name seems like a good plan to me. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuRj74ACgkQ+gerLs4ltQ5RNACeKYku88A9PBuQR46QTl7GrEwo mPEAoLdYyi+TLGYFw4SRAIM8zBsNvwxr =iPkb -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou solip...@pitrou.net wrote: Le Fri, 5 Mar 2010 17:03:02 +1100, Brian Quinlan br...@sweetapp.com a écrit : The PEP lives here: http://python.org/dev/peps/pep-3148/ Ok, here is my take on it: cancel() Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True. I think it shouldn't return anything, and raise an exception if cancelling failed. It is really an error condition, and ignoring the result doesn't seem right. The caller can't avoid the error here by querying the future, because of the problem you point out below, so I'm inclined to think that the future was already started should be a return value rather than an exception (although that may be my C++ background showing through). Would calling the method try_cancel() work better? Future.running() Return True if the call is currently being executed and cannot be cancelled. Future.done() Return True if the call was successfully cancelled or finished running. These don't really make sense since the future is executing concurrently. By the time the result is returned, it can already be wrong. I advocate removing those two methods. done() can only be wrong in one direction though. If it returns True, it'll never again return False, which can be useful (although perhaps only for polling, which might be an argument for removing it anyway). running() becomes True and then False again, so I agree with your objection. A started() function would only go from False to True once. Maybe that's a better function? Jeffrey ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Guido van Rossum writes: Future is a pretty standard CS term for this concept (as noted promise is another), I like the term promise better. Future is very generic (not now, but later), whereas a promise is something I don't get from you now, but you will give me later. The wikipedia article is not very helpful on the implicit vs. explicit distinction. As far as I can tell from it, that distinction isn't really attached to future vs promise. The only distinction the article described was in the context of the Alice language, where a future = promise (read-only) plus resolver (mutator). IMO that's not a compelling reason for adopting future in Python. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
exar...@twistedmatrix.com writes: The explicit futures on the wikipedia page seems to cover what is commonly referred to as a future. For example, Java's futures look like this. The implicit futures are what is generally called a promise. For example, E's promises look like this. *sigh* All I can say is it's a damned shame that there are no native speakers of English working in computer science.wink I have to admit Jean-Paul's explanation a pretty convincing reason for adopting future rather than promise. But I'm with Skip, I would prefer that the module be named future rather than futures. (Especially when wearing my Professional Economist sweatshirt. :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 01:03 AM 3/5/2010, Brian Quinlan wrote: Hi all, I recently submitted a daft PEP for a package designed to make it easier to execute Python functions asynchronously using threads and processes. It lets the user focus on their computational problem without having to build explicit thread/process pools and work queues. The package has been discussed on stdlib-sig but now I'd like this group's feedback. My immediate reaction is that this would be a lot more useful if it built on an API for coroutine yielding/interaction, similar to what's in say, Eventlet. That would seem to make it easier to write synchronous-looking code that operates on futures, and allow futures to be composed more cleanly. ISTM that if futures were standardized before a coroutine API, it would lead to effective orphaning ala what happened with asyncore, especially since the new library is, well, new. I'm somewhat concerned that, as described, the proposed API adds little over what's relatively easy to do with a mature coroutine framework like Eventlet, while at the same time creating yet another alternative (and mutually incompatible) event loop system in the stdlib, beyond the ones that are already in asyncore, tkinter, and the various SocketServer subclasses. As far as naming goes, Twisted uses the term Deferred for this concept (and also has a very mature API for handling them). And speaking of Twisted, it seems to me that the PEP would be much improved in general by learning from some of the lessons of other systems. I don't think that Java's example is really the best one to follow in this instance, compared to the many existing async frameworks that have Python-specific experience and APIs to learn from. Or, to put it another way, something that worries me about this PEP is that nearly all of its Python-related citations are for *discussions* of futures, with the only previous Python implementation cited being a crude sketch of a cookbook recipe. The PEP also doesn't address questions of interoperability with existing solutions, compare features with them, or even so much as say, There are other production implementations of this concept in Python, but we are going to pretend they don't exist. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby p...@telecommunity.com wrote: I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and mutually incompatible) event loop system in the stdlib ... Futures are a blocking construct; they don't involve an event loop. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote: On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby p...@telecommunity.com wrote: I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and mutually incompatible) event loop system in the stdlib ... Futures are a blocking construct; they don't involve an event loop. And where they block is in a loop, waiting for events (completed promises) coming back from other threads or processes. The Motivation section of the PEP also stresses avoiding reinvention of such loops, and points to the complication of using more than one at a time as a justification for the mechanism. It seems relevant to at least address why wrapping multiprocessing and multithreading is appropriate, but *not* dealing with any other form of sync/async boundary, *or* composition of futures. On which subject, I might add, the PEP is silent on whether executors are reentrant to the called code. That is, can I call a piece of code that uses futures, using the futures API? How will the called code know what executor to use? Must I pass it one explicitly? Will that work across threads and processes, without explicit support from the API? IOW, as far as I can tell from the PEP, it doesn't look like you can compose futures without *global* knowledge of the application... and in and of itself, this seems to negate the PEP's own motivation to prevent duplication of parallel execution handling! That is, if I use code from module A and module B that both want to invoke tasks asynchronously, and I want to invoke A and B asynchronously, what happens? Based on the design of the API, it appears there is nothing you can do except refactor A and B to take an executor in a parameter, instead of creating their own. It seems therefore to me that either the proposal does not define its scope/motivation very well, or it is not well-equipped to address the problem it's setting out to solve. If it's meant to be something less ambitious -- more like a recipe or example -- it should properly motivate that scope. If it's intended to be a robust tool for composing different pieces of code, OTOH, it should absolutely address the issue of writing composable code... since, that seems to be what it says the purpose of the API is. (I.e., composing code to use a common waiting loop.) And, existing Python async APIs (such as Twisted's Deferreds) actually *address* this issue of composition; the PEP does not. Hence my comments about not looking at existing implementations for API and implementation guidance. (With respect to what the API needs, and how it needs to do it, not necessarily directly copying actual APIs or implementations. Certainly some of the Deferred API naming has a rather, um, twisted vocabulary.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
