Actually I am swayed by this argument. If the waiter is cancelled they aren't going to continue iterating so the remaining tasks are orphaned. In the rare case where a waiter doesn't want this to happen they can use shield().
On Tue, Mar 3, 2015 at 4:45 PM, Gustavo Carneiro <[email protected]> wrote: > On 3 March 2015 at 19:34, Guido van Rossum <[email protected]> wrote: > >> I looked a bit more. The only thing that ever waits is the caller in the >> for-loop (for f in as_completed(...): x = yield from f; ...). It will >> always wait for _wait_for_one(), which in turn will always wait for >> done.get(). So if the above yield-from is cancelled, _wait_for_one() is >> cancelled, which in turn cancels the Queue.get() call. If this is >> cancelled, the corresponding put_nowait() call may well cancelled (I >> stopped following the code carefully) but this is called from the >> completion callback for a completed future, so that future isn't >> cancellable, and there's nothing that cancels the futures still in the todo >> queue. >> >> I'm skeptical that it's desirable to cancel the original futures though. >> Have you thought through the use case? (Admittedly I haven't.) >> > > I never used as_completed either, but I found an article with an > example[1]: > > @asyncio.coroutinedef process_as_results_come_in(): > coroutines = [get_url(url) for url in ['URL1', 'URL2', 'URL3']] > for coroutine in asyncio.as_completed(coroutines): > url, wait_time = yield from coroutine > print('Coroutine for {} is done'.format(url)) > > > So now I get it, it is used to be able to process results of multiple > tasks running in parallel as soon as each task gets done. The problem I > see here is that, typically, in the for loop like the one above, if you get > a TimeoutError you do not know easily which of the tasks completed and > which are still running. You could blindly ask to cancel all the input > tasks. A more elegant way would be if as_completed itself cancelled > uncompleted tasks, by default. > > I feel it makes code simpler. Cancelling tasks for me is like freeing > memory: it is much better to have a Garbage Collector to free memory, and > is nicer also to have tasks get cancelled automatically when waiting for > them times out. > > Not sure I am able to convince you. I am only half convinced myself... :-) > > > [1] http://sahandsaba.com/understanding-asyncio-node-js-python-3-4.html > > > >> On Tue, Mar 3, 2015 at 11:01 AM, Gustavo Carneiro <[email protected]> >> wrote: >> >>> >>> >>> On 2 March 2015 at 10:12, Victor Stinner <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> To answer Luciano's question on as_completed(), I read the source code >>>> of the function. It's not clear to me what happens on timeout. >>>> >>>> >>>> Should we abort something on timeout? The documentation doesn't >>>> explain if tasks are cancelled or if an exception is set. In the code, >>>> it's clear that the task is not touched. It's probably just a >>>> documentation issue. >>>> >>> >>> I think it should cancel the wrapped task. I think this is what the >>> programmer will want most of the time, and in case he doesn't want it he >>> can use shield() to prevent it. >>> >>> Also it makes it consistent with wait_for: >>> >>> https://code.google.com/p/tulip/issues/detail?id=107 >>> http://bugs.python.org/issue23219 >>> >>> >>>> >>>> >>>> Coroutine objects are wrapped into tasks: on timeout, these temporary >>>> tasks may be destroyed before their completion. A warning is emited in >>>> this case. The warning is useless, since the caller doesn't have >>>> access to all these temporary tasks. As gather(), we should probably >>>> set the _log_destroy_pending attribute of these tasks to False. What >>>> do you think? >>> >>> >>>> Victor >>>> >>> >>> >>> >>> -- >>> Gustavo J. A. M. Carneiro >>> Gambit Research >>> "The universe is always one step beyond logic." -- Frank Herbert >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > -- > Gustavo J. A. M. Carneiro > Gambit Research > "The universe is always one step beyond logic." -- Frank Herbert > -- --Guido van Rossum (python.org/~guido)
