Doug Hellmann wrote:
On Nov 24, 2014, at 12:57 PM, Mike Bayer<[email protected]> wrote:
On Nov 24, 2014, at 12:40 PM, Doug Hellmann<[email protected]> wrote:
This is a good point. I’m not sure we can say “we’ll only use explicit/implicit
async in certain cases" because most of our apps actually mix the cases. We
have WSGI apps that send RPC messages and we have other apps that receive RPC
messages and operate on the database. Can we mix explicit and implicit operating
models, or are we going to have to pick one way? If we have to pick one, the
implicit model we’re currently using seems more compatible with all of the various
libraries and services we depend on, but maybe I’m wrong?
IMHO, in the ideal case, a single method shouldn’t be mixing calls to a set of
database objects as well as calls to RPC APIs at the same time, there should be
some kind of method boundary to cross. There’s a lot of ways to achieve that.
The database calls are inside the method invoked through RPC. System 1 sends an RPC
message (call or cast) to system 2 which receives that message and then does
something with the database. Frequently “system 1” is an API layer service (mixing
WSGI and RPC) and "system 2” is something like the conductor (mixing RPC and DB
access).
What is really needed is some way that code can switch between explicit yields
and implicit IO on a per-function basis. Like a decorator for one or the
other.
The approach that Twisted takes of just using thread pools for those IO-bound
elements that aren’t compatible with explicit yields is one way to do this.
This might be the best way to go, if there are in fact issues with mixing in
implicit async systems like eventlet. I can imagine, vaguely, that the
eventlet approach of monkey patching might get in the way of things in this
more complicated setup.
Part of what makes this confusing for me is that there’s a lack of clarity over
what benefits we’re trying to get from the async work. If the idea is, the GIL
is evil so we need to ban the use of all threads, and therefore must use defer
for all IO, then that includes database IO which means we theoretically benefit
from eventlet monkeypatching - in the absence of truly async DBAPIs, this is
the only way to have deferrable database IO.
If the idea instead is, the code we write that deals with messaging would be
easier to produce, organize, and understand given an asyncio style approach,
but otherwise we aren’t terribly concerned what highly sequential code like
database code has to do, then a thread pool may be fine.
A lot of the motivation behind the explicit async changes started as a way to
drop our dependency on eventlet because we saw it as blocking our move to
Python 3. It is also true that a lot of people don’t like that eventlet
monkeypatches system libraries, frequently inconsistently or incorrectly.
Apparently the state of python 3 support for eventlet is a little better than
it was when we started talking about this a few years ago, but the
monkeypatching is somewhat broken. lifeless suggested trying to fix the
monkeypatching, which makes sense. At the summit I think we agreed to continue
down the path of supporting both approaches. The issues you’ve raised with
using ORMs (or indeed, any IO-based libraries that don’t support explicit
async) make me think we should reconsider that discussion with the additional
information that didn’t come up in the summit conversation.
I think victor is proposing fixes here recently,
https://lists.secondlife.com/pipermail/eventletdev/2014-November/001195.html
So that seems to be ongoing to fix up that support (the eventlet
community is smaller and takes more time to accept pull requests and
such, from what I've seen, but this is just how it works).
Doug
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
