On Sun, Feb 9, 2014 at 3:58 PM, Victor Stinner <[email protected]>wrote:
> I wrote asyncio.wait(*readlines) in my test program and Python became
> slow and ate more and more memory. readlines is declared as
> [proc.stdout.readline(), proc.stderr.readline()].
>
I guess that was Trollius? Tulip's wait() has a lone * after the first
argument so it would have raised an immediate error. (Please always state
whether you observed something in Trollius or Tulip. They may be similar,
but details like this are often different. As you wrote earlier, Python 3
really is better. :-)
> The problem is that wait() expects a list of futures :
> asyncio.wait(readlines). Maybe asyncio.wait(futures) should raise an
> error if futures is not an iterable.
>
Heh, but it would have been a Future instance which *is* iterable (at least
in Tulip, where it is iterable so it works with 'yield from').
> asyncio.wait(future) calls set(future) which can enter an unlimited
> loop (it's the case for StreamReader.readline coroutine at least...)
> :-(
>
Wow, that feels weird. I guess this is also only Trollius? In Tulip
iterating over a Future gives you an exception at the second __next__()
call.
> It's not easy to remember when a function expects a list of futures
> (futures) or futures (*futures).
>
> - list: asyncio.as_completed(futures), wait(futures)
> - multiple: asyncio.gather(*futures)
>
> Why is gather() different??
>
wait() and as_completed() are low-level functions that were defined in PEP
3148 (concurrent.futures) as taking sets of Futures. And they return
Futures.
But gather() is a higher-level convenience function that returns results
(after blocking). The intended use is
res1, res2, res3 = yield from gather(<call1>, <call2>, <call3>)
which is supposed to be very similar to
res1 = yield from <call1>
res2 = yield from <call2>
res3 = yield from <call3>
with the twist that all three <call>s are run concurrently. (Each <call> is
an expression that produces a Future or a coroutine, e.g. your
proc.stdin.readline().)
Because of this intended use, asking the caller to provide an extra set of
parentheses/brackets is considered an inconvenient.
And what you wanted would have been
line_from_stdout, line_from_stderr = asyncio.gather(proc.stdout.readline(),
proc.stderr.readline())
Similar trap: subprocess.Popen(args) expects a list whereas
> EventLoop.subprocess_exec(*args) expects multiple paramters.
> EventLoop.subprocess_exec() should maybe also raise an error if the
> first argument is a iterable (but not bytes or str).
>
For those it would be a good idea to add strict type checking. (But note
that there are no Futures involved.)
For wait() and as_completed(), I suppose you could add an explicit
assert not isinstance(fs, futures.Future) and not iscoroutine(fs)
at the top (or raise an explicit TypeError).
--
--Guido van Rossum (python.org/~guido)
