On 06/02/14 13:22, Joshua Harlow wrote:
Has there been any investigation into heat.
Heat has already used parts of the coroutine approach (for better or
worse).
An example:
https://github.com/openstack/heat/blob/master/heat/engine/scheduler.py#L230
"""
Decorator for a task that needs to drive a subtask.
This is essentially a replacement for the Python 3-only "yield from"
keyword (PEP 380), using the "yield" keyword that is supported in
Python 2. For example::
"""....
I heard my name :)
For those not familiar, here is what we're doing in Heat: we use both
eventlet and co-routines. We use eventlet pretty conventionally - it
allows us to run multiple user requests on the same engine concurrently,
so that when one is blocked on I/O, the others run. We also use
co-routines to organise our workflow within a single user request, so
that we don't sit in a tight loop polling for resources to finish being
created, but instead move on with any other non-dependent resources
before coming back to poll again.
There are actually some advantages to keeping these mechanisms separate
- it's pretty clear which operations are atomic (within the context of a
single request), and this is orthogonal to when you do I/O. I'm not sure
if Futures would have the same property?
The docstring quoted above is from the taskwrapper decorator, which
effectively makes any 'yield's in the wrapped co-routine act like 'yield
from' in Python 3. I always describe 'yield from' as like a function
call but for coroutines... however, anecdotally I can confirm that
people still find this very confusing. Maybe in 5-10 years when there
are a lot of asyncio programs floating around this will make more sense.
[BTW am I right in thinking that in trollius you _have_ to drive your
co-routines from a top-level Task? That's not the case in Heat (although
we do anyway), or IIUC in asyncio - I was expecting e.g. the exception
handling for Return to happen in the coroutine decorator rather than in
Task. Just curious.]
Porting the existing co-routines part of Heat over to trollius ought to
be fairly easy - I discovered Tulip just as I was finishing the
implementation and it looked like what we have is pretty similar to a
subset of it. I'm not sure this would be especially useful though.
Long-term, I'm more interested in Josh's taskflow stuff allowing us to
distribute the workflow across multiple engines (though we have bigger
architectural fish to fry before we can do that).
Substantially all of our blocking I/O consists of:
1) Database calls; and
2) Calls to the OpenStack Python client libraries
So the first step in implementing trollius in Heat would probably be to
make trollius versions of the client libraries. I'm not sure that's the
Right Thing for clients in general though, which suggests to me that
Heat might not be the best place to start this migration; you may not be
able to get there from here.
The main reason asyncio exists (as opposed to, say, twisted) is that
it's supposed to be the standard place to get an event loop from, so
that everybody ends up using the same event loop. Right now, though,
there are still lots of competing event loops (including eventlet) and
it's difficult to see how introducing another one helps in the short
term. That makes Josh's idea of porting eventlet to sit on top of
asyncio sound less terrible than it did at first sight ;) even though it
doesn't address any of the fundamental issues with eventlet.
cheers,
Zane.
I bet trollius would somewhat easily replace a big piece of that code.
-Josh
-----Original Message-----
From: victor stinner <victor.stin...@enovance.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Date: Thursday, February 6, 2014 at 1:55 AM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] Asynchrounous programming: replace eventlet
with asyncio
Sean Dague wrote:
First, very cool!
Thanks.
This is very promising work. It might be really interesting to figure
out if there was a smaller project inside of OpenStack that could be
test ported over to this (even as a stackforge project), and something
we could run in the gate.
Oslo Messaging is a small project, but it's more a library. For a full
daemon, my colleague Mehdi Abaakouk has a proof-on-concept for Ceilometer
replacing eventlet with asyncio. Mehdi told me that he doesn't like to
debug eventlet race conditions :-)
Our experience is the OpenStack CI system catches bugs in libraries and
underlying components that no one else catches, and definitely getting
something running workloads hard on this might be helpful in maturing
Trollius. Basically coevolve it with a piece of OpenStack to know that
it can actually work on OpenStack and be a viable path forward.
Replacing eventlet with asyncio is a huge change. I don't want to force
users to use it right now, nor to do the change in one huge commit. The
change will be done step by step, and when possible, optional. For
example, in Olso Messaging, you can choose the executor: eventlet or
blocking (and I want to add asyncio).
Victor
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
