[issue33533] Provide an async-generator version of as_completed

[Justin Arthur]

Justin Arthur  added the comment:

Thanks, Hrvoje. I've updated the patch to match this bug's suggested format and 
have updated the documentation and What's New.

The one quirk that comes with the benefit of getting the same futures back is 
that we still allow both coroutines and futures to be passed in. Coroutines 
will not be yielded back in their original form, but instead a new task.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

Hi, thanks for providing a PR. One thing I noticed is that the implementation 
in the PR yields results of the futures from the generator. This issue proposes 
a generator that instead yields the futures passed to as_completed. This is 
needed not just for consistency with concurrent.futures.as_completed, but also 
to allow associating results with the requests that produced them, which is an 
important use case for as_completed.

An example of how this is useful is the snippet at 
https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example
 which you cannot implement with an as_completed that doesn't yield the 
original futures. The StackOverflow question at [1] inquires about that issue, 
and would be easily resolved by yielding the futures as 
concurrent.futures.as_completed does.

As far as I can tell, the needed change is simple: just yield `f` instead of 
`f.result()` from `_wait_for_one` when invoked from __anext__. (It must still 
return f.result() when called from __next__ for backward compatibility.)

[1] 
https://stackoverflow.com/questions/50028465/python-get-reference-to-original-task-after-ordering-tasks-by-completion

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Justin Arthur]

Justin Arthur  added the comment:

I've added a new PR, PR 22491. This one has as_completed returning an iterator 
and includes tests for both the old and new style.

I see a trivial amount of latency added in extra call stack over Guido's 
original implementation. Should we decide to deprecate the 
plain-iterator-of-awaitables form, we can possibly regain that latency in 
refactoring this as an asynchronous generator.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Justin Arthur]

Change by Justin Arthur :


--
nosy: +JustinTArthur
nosy_count: 5.0 -> 6.0
pull_requests: +21508
pull_request: https://github.com/python/cpython/pull/22491

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Roman Evstifeev]

Change by Roman Evstifeev :


--
nosy: +Roman.Evstifeev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Michael DePalatis]

Change by Michael DePalatis :


--
keywords: +patch
pull_requests: +9565
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

I didn't start working on the PR, so please go ahead if you're interested.

One small suggestion: If you're implementing this, please note that the 
proof-of-concept implementation shown in the description is not very efficient 
because each call to `wait` has to iterate over all the futures (which can be 
potentially large in number) to set up and tear down the done callbacks on each 
one. A more efficient implementation would set up the callbacks only once - see 
https://stackoverflow.com/a/51403277/1600898 for an example.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Michael DePalatis]

Michael DePalatis  added the comment:

Is there any progress on this? I was thinking the exact same thing regarding 
the backwards-compatible approach and would like to work on it if no one else 
is.

--
nosy: +mivade

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

If there is interest in this, I'd like to attempt a PR for a sync/async variant 
of as_completed.

Note that the new docs are *much* clearer, so the first (documentation) problem 
from the description is now fixed. Although the documentation is still brief, 
it now contains the key pieces of information: 1) that the futures are actually 
run in parallel, and 2) that each yielded future produces the next result that 
becomes available. Neither was actually stated in the old docs (!), so this is 
a marked improvement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

Another option occurred to me: as_completed could return an object that 
implements both synchronous and asynchronous iteration protocol:


class as_completed:
def __init__(fs, *, loop=None, timeout=None):
self.__fs = fs
self.__loop = loop
self.__timeout = timeout

def __iter__(self):
# current implementation here
...

async def __aiter__(self):
# new async implementation here
...

def __next__(self):
# defined for backward compatibility with code that expects
# as_completed() to return an iterator rather than an iterable
if self._iter is None:
self._iter = iter(self)
return next(self._iter)

With that design there wouldn't need to be a new function under a different 
name; instead, as_completed could just be documented as an asynchronous 
iterable, with the old synchronous iteration supported for backward 
compatibility.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Yury Selivanov]

Yury Selivanov  added the comment:

I like the idea. Let's revisit it after Python 3.7 is released.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Change by Hrvoje Nikšić :


--
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

Of course, `yield from done` would actually have to be `for future in done: 
yield future`, since async generators don't support yield from.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33533] Provide an async-generator version of as_completed

[Hrvoje Nikšić]

New submission from Hrvoje Nikšić :

Judging by questions on the StackOverflow python-asyncio tag[1][2], it seems 
that users find it hard to understand how to use as_completed correctly. I have 
identified three issues:

* It's somewhat sparingly documented.

A StackOverflow user ([2]) didn't find it obvious that it runs the futures in 
parallel. Unless one is already aware of the meaning, the term "as completed" 
could suggest that they are executed and completed sequentially.

* Unlike its concurrent.futures counter-part, it's non-blocking.

This sounds like a good idea because it's usable from synchronous code, but it 
means that the futures it yields aren't completed, you have to await them 
first. This is confusing for a function with "completed" in the name, and is 
not the case with concurrent.futures.as_completed, nor with other waiting 
functions in asyncio (gather, wait, wait_for).

* It yields futures other than those that were passed in.

This prevents some usual patterns from working, e.g. associating the results 
with context data, such as Python docs itself uses for 
concurrent.futures.as_completed in 
https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor-example
 .  See SO question [1] for a similar request in asyncio.


Here is my proposal to address the issues.

I believe the usage problems stem from as_completed predating the concept of 
async iterators. If we had async iterators back then, as_completed would have 
been an obvious candidate to be one. In that case it could be both "blocking" 
(but not for the event loop) and return the original futures. For example:

async def as_completed2(fs):
pending = set(map(asyncio.ensure_future(fs)))
while pending:
done, pending = await asyncio.wait(pending, 
return_when=asyncio.FIRST_COMPLETED)
yield from done

(It is straightforward to add support for a timeout argument.)

I propose to deprecate asyncio.as_completed and advertise the async-iterator 
version like the one presented here - under a nicer name, such as as_done(), or 
as_completed_async().



[1] 
https://stackoverflow.com/questions/50028465/python-get-reference-to-original-task-after-ordering-tasks-by-completion
[2] 
https://stackoverflow.com/questions/50355944/yield-from-async-generator-in-python-asyncio

--
components: asyncio
messages: 316773
nosy: asvetlov, hniksic, yselivanov
priority: normal
severity: normal
status: open
title: Provide an async-generator version of as_completed
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com