Hi Christian,
You're right -- writing a for-loop that produces values as they become
available is a little awkward. The only example I can think of in asyncio
itself is as_completed():
https://docs.python.org/3/library/asyncio-task.html?#asyncio.as_completed
The recommended usage pattern is:
for f in as_completed(fs):
result = yield from f # The 'yield from' may raise
# Use result
Here the pattern is somewhat simpler: there's no yield in the for-loop
header; also, the implementation doesn't need to hold back a value to raise
StopIteration because the number of results is known ahead of time (it's
len(fs) :-).
Looking at your examples, I'm wondering several things:
- Why do you still need a "yield from" in the for-loop header if you have
one in the body already?
- The notation listall().lines feels a little odd -- perhaps you can have a
separate method for this, e.g. listall_async()?
- Having to hold back one value just so you can raise StopIteration makes
me sad.
I'm not entirely sure what to propose instead. One idea is something I did
in Google App Engine NDB -- for every API there is a synchronous method and
a corresponding async method; the names are always foo() and foo_async(),
where foo() returns a synchronous result while foo_async() returns a
Future. The foo() method is always implemented by calling foo_async() and
waiting for the future -- this would have to use
EventLoop.run_until_complete() in asyncio. Except when there are multiple
results, then foo_async() returns a list of futures and foo() returns a
list of results -- this can be done in asyncio by combining
run_until_complete() with gather().
But again this doesn't solve the problem when you don't know how many
results there will be ahead of time. For that, I wonder if you might be
better off just using some kind of special looping object and a while-loop
so that the caller can write:
it = listall()
while True:
value = yield from it.next()
if not value:
break
# Use value
A slightly more concise variant (and more correct if valud values can be
falsey):
it = listall()
while (yield from it.more()):
value = it.value
# Use value
The idea here is that "it" is a specially constructed object whose more()
method returns a fresh Future, where the Future becomes complete when
either the next value is available or you know that there is no next value.
This arranges for the Future to return True if there's a value or False if
it's the end of the loop (convenient to break out of the while-loop) and as
a side effect sets it.value to the value if there is one (and maybe to None
at the end).
Let me try to write this up as an example using an asyncio StreamReader and
its readline() method (just to have a specific source of data that doesn't
know in advance how many items it will produce):
class Its:
def __init__(self, stream):
self._queue = asyncio.Queue()
asyncio.Task(self._feeder(stream))
@asyncio.coroutine
def _feeder(self, stream):
while True:
line = yield from stream.readline()
if not line:
self._queue.put_nowait((False, None))
break
else:
self._queue.put_nowait((True, line))
@asyncio.coroutine
def more(self): # Or maybe next()?
flag, value = yield from self._queue.get()
self.value = value
return flag
Well, I'm not sure how much I like this implementation; using a Queue feels
a little heavy-handed. And the whole protocol with a .more() method and a
.value attribute feels a bit un-Pythonic. But I can't think of a variant
whose usage pattern is more concise, without being more error-prone, that
doesn't require all valid values to be truthy, and doesn't require holding
a value back for the for-loop.
Of course, it would be nice if we had a variant of the for-loop notation
instead, so that we could just write this:
for value yield in listall():
# Use value
Note the currently unsyntactic placement of yield, which would signal that
this is no ordinary for-loop but one that must yield each result. However,
this would requires a syntactic change, which would require waiting until
Python 3.5 at the earliest.
Just some thoughts,
--Guido
On Sun, Apr 20, 2014 at 9:45 AM, chrysn <[email protected]> wrote:
> hello tulip list,
>
> when porting the python-mpd2 library[1], i tried to port functions that
> originally worked like this:
>
> >>> for song in client.listall():
> ... print(song)
> first song
> second song
>
> the trivial port, especially with small responses or protocols where
> you can retrieve the items only when the complete response is there,
> would be
>
> >>> for song in (yield from client.listall()):
> ... print(song)
>
> . most applications would want to use this kind of interface, as it is
> straightforward -- but it delays the execution of the loop until all the
> results are there. my current approach is implementing what i'm calling
> MultilineFuture (for historical reasons, if it were to be used more
> widely i'd probably call it ListlikeFuture) that can be used as above,
> but also supports the alternative usage
>
> >>> for cursor in (yield from client.listall().lines):
> ... song = yield from cursor
> ... print(song)
>
> that has the same effect but starts printing earlier.
>
> the fulfilling side of the future does not do `future.set_result(["first
> song", ...])`, but does
>
> >>> multiline = MultilineFuture()
> >>> multiline.send_line("first song")
> >>> multiline.send_line("second song")
> >>> multiline.set_completed()
>
> . i think the design works well for these applications; the only
> shortcoming i see is that one response always needs to be held back, as
> the StopIteration for the `for` loop needs to be raised in
> `.lines.__next__` and not in `yield from`.
>
> my questions now are:
>
> * is there an established pattern already in place that could do this /
> did i reinvent the wheel?
>
> * is there a better way than keeping back one item to make the loop stop
> at the right time? (short of coming up with non-python3.4 syntax like
> `for song yield from ...`)
>
> * would such a pattern be a useful addition to the common mechanisms
> used in asyncio?
>
> * do you see the need for more features in such a mechanism (eg a
> `for i, cursor in (yield from client.listall().lines_enumerated)` for
> progress indicators)?
>
> i would appreciate your feedback
> chrysn
>
> [1] https://github.com/Mic92/python-mpd2/issues/30
>
> --
> To use raw power is to make yourself infinitely vulnerable to greater
> powers.
> -- Bene Gesserit axiom
>
--
--Guido van Rossum (python.org/~guido)
