Hi Brett,
Good to see you here. You'll have a blast. :-)
Tulip is just different in this regard; it depends more on the event
loop. I recommend that you just learn to do things the Tulip way. For
example, Tulip guarantees that callbacks are only run directly by the
event loop, never in the context of the code that triggers the event.
There are no nested event loops.
Some suggestions:
- Tulip is somewhat optimized for server-like applications, where
there are usually many tasks handling different connections. (In
contrast, NDB is focusing on the client side of things, where handling
multiple incoming connections is up to the higher-level App Engine
framework.)
- If you have a big CPU-heavy job, it's best to run it in a separate
thread using run_in_executor() so it doesn't get in the way of I/O
handling. Otherwise you'd have to put frequent yields in your
processing, or break it up in tiny bits, which can get tedious.
- You can spell the sleep using just plain "yield" (no expression, no
"from"). This is undocumented because I want people to feel bad when
they're explicitly yielding to the scheduler at all. :-)
- The usual way to spell running several coroutines concurrently is to
use gather() -- your example would become something like
quotient, remainder = yield from asyncio.gather(bc_child(divide),
bc_child(mod))
(If you have more processing to do in the current task, turn it into
another coroutine and wait for that too.)
- If you really want the effect you are describing, you can do it by
writing a non-generator function that does the processing and then
returns a Future. Such a function should not be decorated with
@coroutine, but you can wait on it using yield-from just as if it was
one. Because it is not a generator the function will run as soon as
you call it. (In fact, it could even return a coroutine.)
- You could also probably write your own version of gather() that
requires that its arguments are coroutines (i.e. generators) and calls
next() on each of them before passing them to asyncio.gather(). (Or it
could do this only for those arguments that are generators.) You'd
have to crib some code from Task._step() for how to handle the various
things next() can return or raise. You can use asyncio.iscoroutine()
to check for a coroutine object (don't just use inspect.isgenerator()
-- read the source code for the reason).
Good luck!
On Wed, Dec 25, 2013 at 10:58 PM, Brett Slatkin <[email protected]> wrote:
> Hey all!
>
> This is my first post to this group. I'm excited about Tulip in general, and
> especially how you can write async code with coroutines. I've used the NDB
> library for App Engine a lot. I'm trying to fit my understanding on to
> Tulip.
>
>
> One behavior I noticed was if you use asyncio.async(), you also need to
> yield from asyncio.sleep(0) to get the "fast dispatch" behavior I'm used to
> from NDB. What I want is to start running a sub-task's eventloop until its
> first yield, and *then* return the future for that task to the caller.
>
> This behavior has been great for async RPC systems where getting the RPC
> packed up and ready for the wire is not I/O bound. Instead the RPC client
> just grabs a mutex and puts the packed-up data on a queue for another
> thread.
>
> To reproduce the behavior I want to see, I wrote a silly "greatest common
> denominator" program using Tulip. Here's a snippet from the program I've
> attached that demonstrates fast dispatch:
>
> @asyncio.coroutine
> def int_divide(numerator, denominator):
> print('int_divide stared: %r %r' % (numerator, denominator))
> divide = '%d/%d' % (numerator, denominator)
> quotient_future = asyncio.async(bc_child(divide))
> mod = '%d%%%d' % (numerator, denominator)
> mod_future = asyncio.async(bc_child(mod))
> # bc_child calls above won't run until we yield to the event loop
> yield from asyncio.sleep(0)
> print('Waiting on int_divide futures')
> quotient = yield from quotient_future
> remainder = yield from mod_future
> return int(quotient), int(remainder)
>
>
> With the yield from asyncio.sleep(0) line present you see this output:
>
> gcd started: 1238482 410
> int_divide stared: 1238482 410
> bc_child child stared: '1238482/410'
> run_child stared: 'bc'
> bc_child child stared: '1238482%410'
> run_child stared: 'bc'
> Waiting on int_divide futures << important line
> 1238482 / 410 = 3020 remainder 282
>
>
> With the yield from asyncio.sleep(0) line commented out, you see this
> output:
>
> gcd started: 1238482 410
> int_divide stared: 1238482 410
> Waiting on int_divide futures << important line
> bc_child child stared: '1238482/410'
> run_child stared: 'bc'
> bc_child child stared: '1238482%410'
> run_child stared: 'bc'
> 1238482 / 410 = 3020 remainder 282
>
>
> Note how the location of "Waiting on int_divide futures" moves to the top
> when the sleep is commented out. That's the slow dispatch case. It means
> none of the sub-task event loops start until I wait on at least one of their
> futures.
>
> Imagine I want to fire off two async RPCs and then do a bunch of computation
> and then wait for both RPCs to finish. Without the sleep() call my
> computation would delay the RPCs from ever starting. With the sleep call
> these fast dispatch RPCs will work.
>
> What's the best practice here? Am I doing it wrong? Doing explicit sleep()
> calls seems really gross. My ideal would be a decorator to cause this fast
> dispatch behavior, but then again I noticed that the @task decorator is now
> gone.
>
> Anyways, thanks in advance for the help!
>
> -Brett
>
--
--Guido van Rossum (python.org/~guido)
