On 7/4/14, 4:45 AM, Julien Danjou wrote: > On Thu, Jul 03 2014, Mark McLoughlin wrote: > >> We're attempting to take baby-steps towards moving completely from >> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the >> first victim. > Thumbs up for the plan, that sounds like a good approach from what I > got. I just think there's a lot of things that are going to be > synchronous anyway because not everything provide a asynchronous > alternative (i.e. SQLAlchemy or requests don't yet AFAIK). It doesn't > worry me much as there nothing we can do on our side, except encourage > people to stop writing synchronous API¹. > > And big +1 for using Ceilometer as a test bed. :) Allowing SQLAlchemy to be fully compatible with an explicit async programming approach, which note is distinctly different from allowing SQLAlchemy to run efficiently within an application that uses explicit async, has been studied and as of yet, it does not seem possible without ruining the performance of the library (including Core-only), invalidating the ORM entirely, and of course doing a rewrite of almost the whole thing (see http://stackoverflow.com/questions/16491564/how-to-make-sqlalchemy-in-tornado-to-be-async/16503103#16503103, http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html#gevent-and-pep-3156).
But before you even look at database abstraction layers, you need a database driver. What's the explicitly async-compatible driver for MySQL? Googling around I found https://github.com/eliast/async-MySQL-python, but not much else. Note that for explicit async, a driver that allows monkeypatching is no longer enough. You need an API like Psycopg2s asynchronous support: http://initd.org/psycopg/docs/advanced.html#async-support. Note that psycopg2's API is entirely an extension to the Python DBAPI: http://legacy.python.org/dev/peps/pep-0249/. So an all explicit async approach necessitates throwing out this out as well; as an alternative, here is twisted's adbapi extension to pep-249's API: https://twistedmatrix.com/documents/current/core/howto/rdbms.html. I'm not sure if Twisted provides an explicit async API for MySQL. If you are writing an application that runs in an explicit, or even an implicitly async system, and your database driver isn't compatible with that, your application will perform terribly - because you've given up regular old threads, and your app now serializes most of what it does through a single, blocking pipe. That is the current status of all Openstack apps that rely heavily on MySQLdb and Eventlet at the same time. Explicitly asyncing it will help in that we won't get hard-to-predict context switches that deadlock against the DB driver (also solvable just by using an appropriate patchable driver), but it won't help performance until that is solved. Nick's post points the way towards a way that everyone can have what they want - which is that once we get a MySQL database adapter that is implicit-async-patch-capable, the explicit async parts of openstack call into database routines that are using implicit async via a gevent-like approach. That way SQLAlchemy's source code does not have to multiply it's function call count by an order of magnitude, or be rewritten, and the ORM-like features that folks like to complain about as they continue to use them like crazy (e.g. lazy loading) can remain intact. If we are in fact considering going down this latest rabbit hole which claims that program code cannot possibly be efficient or trusted unless all blocking operations are entirely written literally by humans, yielding all the way down to the system calls, I would ask that we make a concerted effort to face just exactly what that means we'd be giving up. Because the cost of how much has to be thrown away may be considerably higher than people might realize. For those parts of an app that make sense using explicit async, we should be doing so. However, we should ensure that those code sections more appropriate as implicit async remain first class citizens as well. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev