Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 10 Mar 2010, at 08:32, Dj Gilcrease wrote: On Mon, Mar 8, 2010 at 2:11 PM, wrote: Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. Ok the new patch is submitted @ http://code.google.com/p/pythonfutures/issues/detail?id=1 Cool, thanks. *note there are 2 tests that fail and 1 test that dead locks on windows even without this patch, the deadlock test I am skipping in the patch and the two that fail do so for a reason that does not make sense to me. I'll investigate but I don't have convenient access to a windows machine. Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 08:39, Greg Ewing wrote: Terry Reedy wrote: Looking more close, I gather that the prime results will be printed 'in order' (waiting on each even if others are done) while the url results will be printed 'as available'. Seems to me that if you care about the order of the results, you should be able to just wait for each result separately in the order you want them. Something like task1 = start_task(proc1) task2 = start_task(proc2) task3 = start_task(proc3) result1 = task1.wait_for_result() result2 = task2.wait_for_result() result3 = task3.wait_for_result() You can write this as: executor = ... future1 = executor.submit(proc1) future2 = executor.submit(proc2) future3 = executor.submit(proc3) result1 = task1.result() result2 = task2.result() result3 = task3.result() This would also be a natural way to write things even if you don't care about the order, but you need all the results before proceeding. You're going to be held up until the longest-running task completes anyway, so it doesn't matter if some of them finish earlier and have to sit around waiting for you to collect the result. Often you don't want to continue if there is a failure. In the example that you gave, if "proc3" raises an exception immediately, you still wait for "proc1" and "proc2" to complete even though you will end up discarding their results. Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 08:11, [email protected] wrote: On 08:56 pm, [email protected] wrote: On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease wrote: A style I have used in my own code in the past is a Singleton class with register and create methods, where the register takes a name(string) and the class and the create method takes the name and *args, **kwargs and acts as a factory. So I decided to play with this design a little and since I made it a singleton I decided to place all the thread/process tracking and exit handle code in it instead of having the odd semi-global scoped _shutdown, _thread_references, _remove_dead_thread_references and _python_exit objects floating around in each executor file, seems to work well. The API would be from concurrent.futures import executors executor = executors.create(NAME, *args, **kwargs) # NAME is 'process' or 'thread' by default To create your own executor you create your executor class and add the following at the end Getting rid of the process-global state like this simplifies testing (both testing of the executors themselves and of application code which uses them). It also eliminates the unpleasant interpreter shutdown/module globals interactions that have plagued a number of stdlib systems that keep global state. I'm not sure what you mean, could you clarify? Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
On 9 Mar 2010, at 03:21, Terry Reedy wrote: On 3/6/2010 4:20 AM, Brian Quinlan wrote: On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote: On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan mailto:[email protected]>> wrote: import futures +1 on the idea, -1 on the name. It's too similar to "from __future__ import ...". Also, the PEP should probably link to the discussions on stdlib-sig? I thoug ht about that but this discussion is spread over many threads and many months. This is pretty typical. I would say just that, and link to the first. "This PEP was discussed over many months in many threads in the stdlib-sig list. The first was . Python-dev discussion occured in . I'll add that. Cheers, Brian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
Brian Quinlan wrote: >> Getting rid of the process-global state like this simplifies testing >> (both testing of the executors themselves and of application code >> which uses them). It also eliminates the unpleasant interpreter >> shutdown/module globals interactions that have plagued a number of >> stdlib systems that keep global state. > > I'm not sure what you mean, could you clarify? Assuming your question refers to the second sentence, Jean-Paul is referring to a trick of the CPython interpreter when it terminates. To maximise the chances of objects being deleted properly rather than just dumped from memory when the process exits, module dictionaries are filled with None values before the interpreter shuts down. This can cause weirdness (usually intermittent name errors during shutdown) when __del__ methods directly or indirectly reference module globals. One of the easiest ways to avoid that is to put the state on a singleton object, then give the affected classes a reference to that object. Cheers, Nick. P.S. This problem is actually the reason we don't have a context manager for temporary directories yet. Something that should have been simple became a twisty journey down the rabbit hole: http://bugs.python.org/issue5178 -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
[email protected] wrote: > >> I'm +1 on adding a nice task queuing system, -1 on calling it by any > >> other name. ;-) > > Nick> As Guido said, let's call the nice task queuing system "futures" > Nick> and point people wanting a full-power asynchronous process model > Nick> to Twisted > > Can this module at least be pushed down into a package? I think > "concurrent" or "concurrency" were both suggested at one point. Yep, I believe "concurrent.futures" was picked as the name elsewhere in the thread. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] argparse ugliness
Greg Ewing wrote: > Steven Bethard wrote: > >> Because the names are so long and you'd have to import them, I've left >> them as private attributes of the module, but if there's really >> demand, we could rename them to argparse.StoreTrueAction, etc. > > What's wrong with just StoreTrue? All of this discussion about the class names is ignoring the main benefit of using the string names: - with Python variables, you just get a generic name error at the reference site (which may or may not be useful, depending on the program structure) - with a string, the parser *tells* you that the problem is with the requested action for a particular argument The second, explicit error is going to be more informative in most cases. Using strings also reduces the verbosity of the code, avoiding either an "argparse." buried in the middle of the function call, or else a "from argparse import ". Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] argparse ugliness
Greg Ewing wrote:
> Xavier Morel wrote:
>> So you'd have to write add_argument('--plot',
>> action=actions.store_true) which is straight from the department of
>> redundant redundancies.
>
> This could easily be fixed with
>
> from argparse.actions import store_true
Converting argparse from a module to a package (which would be necessary
to make the above work) is above the bar I set for "easy".
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
---
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] argparse ugliness
Nick Coghlan wrote: Greg Ewing wrote: Steven Bethard wrote: Because the names are so long and you'd have to import them, I've left them as private attributes of the module, but if there's really demand, we could rename them to argparse.StoreTrueAction, etc. What's wrong with just StoreTrue? All of this discussion about the class names is ignoring the main benefit of using the string names: - with Python variables, you just get a generic name error at the reference site (which may or may not be useful, depending on the program structure) - with a string, the parser *tells* you that the problem is with the requested action for a particular argument The second, explicit error is going to be more informative in most cases. Using strings also reduces the verbosity of the code, avoiding either an "argparse." buried in the middle of the function call, or else a "from argparse import ". I agree that it should be left as a string. About the only reason I would suggest changing it from a string to an object is that it makes it obvious that argparse does accept an object that conforms to the Action API. But this is already well documented, so I don't see the need. Just leave it as a string. -- Eric. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] argparse ugliness
On Wed, Mar 10, 2010 at 7:46 AM, Nick Coghlan wrote: > All of this discussion about the class names is ignoring the main > benefit of using the string names: Another benefit of strings is that data-driven argparse configuration will usually be slightly simpler. Some of us find things like that useful. :-) -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.6.5 release candidate 2 now available
Hi everyone, The source tarballs and Windows installer for Python 2.6.5 release candidate 2 are now available: http://www.python.org/download/releases/2.6.5/ As usual, please download, install, and test them with your favorite projects and environments. A number of regressions and build problems on OS X have been fixed since rc1, and I'm really hoping we will not have to do an rc3. I'm currently planning on releasing 2.6.5 final on March 19, 2010. Enjoy, -Barry P.S. The Mac installer will hopefully be available soon. signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FWD: http://www.python.org/dev/patches/
On 09/03/2010 16:08, Aahz wrote: - Forwarded message from Bob Vadnais - Date: Tue, 09 Mar 2010 00:37:33 -0500 From: Bob Vadnais To: [email protected] Subject: http://www.python.org/dev/patches/ Submit documentation patches the same way. When adding the patch, be> sure to set the "Category" field to "Documentation". There doesn't appear to be a field named "Category". Perhaps there used to be? The correct field name now appears to be "Components". Yep, 'Components' is correct. I'm amending that page appropriately. Thanks Michael Foord - End forwarded message - -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Catch SIGINT at Python startup
Hi, Some news about my patch. Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a écrit : > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum wrote: > > Actually it sounds like there's some overly general except clause > > somewhere that should be adjusted to catch just "Exception" instead of > > "*". > > There is at least one that prints "import 'site' failed" and continues > to run your program. I commited to patch and broke all buildbots \o/ There were errors in test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems with sys.executable and modified sys.argv[0]). I consider that it's a good point to detect the errors earlier, but it breaks code based on the errors :-) I will no backport the fix to 2.6 (or at least, not the fix about importing the site module) Should I backport the fix to 3.1? (The backport to py3k is not done yet) -- Victor Stinner http://www.haypocalc.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
