Re: [openstack-dev] [all] Replace eventlet with asyncio

[Dolph Mathews]

On Wed, Feb 25, 2015 at 4:12 AM, Victor Stinner  wrote:

> Hi,
>
> > I also just put up another proposal to consider:
> > https://review.openstack.org/#/c/156711/
> > """Sew over eventlet + patching with threads"""
>
> My asyncio spec is unclear about WSGI, I just wrote
>
> "The spec doesn't change OpenStack components running WSGI servers
> like nova-api. The specific problem of using asyncio with WSGI will
> need a separated spec."
>
> Joshua's threads spec proposes:
>
> "I would prefer to let applications such as apache or others handle
> the request as they see fit and just make sure that our applications
> provide wsgi entrypoints that are stateless and can be horizontally
> scaled as needed (aka remove all eventlet and thread ... semantics
> and usage from these entrypoints entirely)."
>

+1!


>
> Keystone wants to do the same:
> https://review.openstack.org/#/c/157495/
> "Deprecate Eventlet Deployment in favor of wsgi containers
>
> This deprecates Eventlet support in documentation and on invocation
> of keystone-all."
>

We don't just *want* to do the same: we actually made eventlet a second
class citizen long ago (that's going to be a tough first step for many
projects that are tightly coupled with eventlet). We have little reason to
continue supporting eventlet, so we're finally proposing it for official
deprecation.


> I agree: we don't need concurrency in the code handling a single HTTP
> request: use blocking functions calls. You should rely on highly efficient
> HTTP servers like Apache, nginx, werkzeug, etc. There is a lot of choice,
> just pick your favorite server ;-) Each HTTP request is handled in a
> thread. You can use N processes and each process running M threads. It's a
> common architecture design which is efficient.
>
> For database accesses, just use regular blocking calls (no need to modify
> SQLAchemy). According to Mike Bayer's benchmark (*), it's even the fastest
> method if your code is database intensive. You may share a pool of database
> connections between the threads, but a connection should only be used by a
> single thread.
>
> (*)
> http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
>
> I don't think that we need a spec if everybody already agree on the design
> :-)
>
> Victor
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Replace eventlet with asyncio

[Clint Byrum]

Excerpts from Victor Stinner's message of 2015-02-25 02:12:05 -0800:
> Hi,
> 
> > I also just put up another proposal to consider:
> > https://review.openstack.org/#/c/156711/
> > """Sew over eventlet + patching with threads"""
> 
> My asyncio spec is unclear about WSGI, I just wrote
> 
> "The spec doesn't change OpenStack components running WSGI servers
> like nova-api. The specific problem of using asyncio with WSGI will
> need a separated spec."
> 
> Joshua's threads spec proposes:
> 
> "I would prefer to let applications such as apache or others handle
> the request as they see fit and just make sure that our applications
> provide wsgi entrypoints that are stateless and can be horizontally
> scaled as needed (aka remove all eventlet and thread ... semantics
> and usage from these entrypoints entirely)."
> 
> Keystone wants to do the same:
> https://review.openstack.org/#/c/157495/
> "Deprecate Eventlet Deployment in favor of wsgi containers
> 
> This deprecates Eventlet support in documentation and on invocation
> of keystone-all."
> 
> I agree: we don't need concurrency in the code handling a single HTTP 
> request: use blocking functions calls. You should rely on highly efficient 
> HTTP servers like Apache, nginx, werkzeug, etc. There is a lot of choice, 
> just pick your favorite server ;-) Each HTTP request is handled in a thread. 
> You can use N processes and each process running M threads. It's a common 
> architecture design which is efficient.
> 
> For database accesses, just use regular blocking calls (no need to modify 
> SQLAchemy). According to Mike Bayer's benchmark (*), it's even the fastest 
> method if your code is database intensive. You may share a pool of database 
> connections between the threads, but a connection should only be used by a 
> single thread.
> 
> (*) http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
> 
> I don't think that we need a spec if everybody already agree on the design :-)
> 

+1

This leaves a few pieces of python which don't operate via HTTP
requests. There are likely more, but these come to mind:

* Nova conductor
* Nova scheduler/Gantt
* Nova compute
* Neutron agents
* Heat engine

I don't have a good answer for them, but my gut says none of these
gets as crazy with concurrency as the API services which have to talk
to all the clients with their terrible TCP stacks, and awful network
connectivity. The list above is always just talking on local buses, and
thus can likely just stay on eventlet, or use a multiprocessing model to
take advantage of local CPUs too. I know for Heat's engine, we saw quite
an improvement in performance of Heat just by running multiple engines.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Replace eventlet with asyncio

[Victor Stinner]

Hi,

> I also just put up another proposal to consider:
> https://review.openstack.org/#/c/156711/
> """Sew over eventlet + patching with threads"""

My asyncio spec is unclear about WSGI, I just wrote

"The spec doesn't change OpenStack components running WSGI servers
like nova-api. The specific problem of using asyncio with WSGI will
need a separated spec."

Joshua's threads spec proposes:

"I would prefer to let applications such as apache or others handle
the request as they see fit and just make sure that our applications
provide wsgi entrypoints that are stateless and can be horizontally
scaled as needed (aka remove all eventlet and thread ... semantics
and usage from these entrypoints entirely)."

Keystone wants to do the same:
https://review.openstack.org/#/c/157495/
"Deprecate Eventlet Deployment in favor of wsgi containers

This deprecates Eventlet support in documentation and on invocation
of keystone-all."

I agree: we don't need concurrency in the code handling a single HTTP request: 
use blocking functions calls. You should rely on highly efficient HTTP servers 
like Apache, nginx, werkzeug, etc. There is a lot of choice, just pick your 
favorite server ;-) Each HTTP request is handled in a thread. You can use N 
processes and each process running M threads. It's a common architecture design 
which is efficient.

For database accesses, just use regular blocking calls (no need to modify 
SQLAchemy). According to Mike Bayer's benchmark (*), it's even the fastest 
method if your code is database intensive. You may share a pool of database 
connections between the threads, but a connection should only be used by a 
single thread.

(*) http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/

I don't think that we need a spec if everybody already agree on the design :-)

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Replace eventlet with asyncio

[Joshua Harlow]

I also just put up another proposal to consider:

https://review.openstack.org/#/c/156711/

"""Sew over eventlet + patching with threads"""

It goes along the thread usage case, and seems useful to consider/think 
about (and if it's thrown away, that's ok IMHO, after all thats the 
point of these discussions) when making this kind of analysis and decision.


-Josh

Mike Bayer wrote:

I’ve spent most of the past week deeply reviewing the asyncio system,
including that I’ve constructed a comprehensive test suite designed to
discover exactly what kinds of latencies and/or throughput advantages or
disadvantages we may see each from: threaded database code, gevent-based
code using Psycopg2’s asynchronous API, and asyncio using aiopg. I’ve
written a long blog post describing a bit of background about non-blocking
IO and its use in Python, and listed out detailed and specific reasons why I
don’t think asyncio is an appropriate fit for those parts of Openstack that
are associated with relational databases. We in fact don’t get much benefit
from eventlet either in this regard, and with the current situation of
non-eventlet compatible DBAPIs, our continued use of eventlet for
database-oriented code is hurting Openstack deeply.

My recommendations are that whether or not we use eventlet or asyncio in
order to receive HTTP connections, the parts of our application that focus
on querying and updating databases should at least be behind a thread pool.
I’ve also responded to the notions that asyncio-style programming will lead
to fewer bugs and faster production of code, and in that area I think there
are also some misconceptions regarding code that’s designed to deal with
relational databases.

The blog post is at
http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and
you’ll find links to the test suite, which is fully runnable, within that
post.

Victor Stinner  wrote:


Hi,

I wrote a second version of my cross-project specification "Replace eventlet with 
asyncio". It's now open for review:

https://review.openstack.org/#/c/153298/

I copied it below if you prefer to read it and/or comment it by email. Sorry, 
I'm not sure that the spec will be correctly formatted in this email. Use the 
URL if it's not case.

Victor

..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.

http://creativecommons.org/licenses/by/3.0/legalcode

=
Replace eventlet with asyncio
=

This specification proposes to replace eventlet, implicit async programming,
with asyncio, explicit async programming. It should fix eventlet issues,
prepare OpenStack for the future (asyncio is now part of the Python language)
and may improve overall OpenStack performances. It also makes usage of native
threads simpler and more natural.

Even if the title contains "asyncio", the spec proposes to use trollius. The
name asyncio is used in the spec because it is more well known than trollius,
and because trollius is almost the same thing than asyncio.

The spec doesn't change OpenStack components running WSGI servers like
nova-api.  Compatibility issue between WSGI and asyncio should be solved first.

The spec is focused on Oslo Messaging and Ceilometer projects. More OpenStack
components may be modified later if the Ceilometer port to asyncio is
successful. Ceilometer will be used to find and solve technical issues with
asyncio, so the same solutions can be used on other OpenStack components.

Blueprint: 
https://blueprints.launchpad.net/oslo.messaging/+spec/greenio-executor

Note: Since Trollius will be used, this spec is unrelated to Python 3. See the
`OpenStack Python 3 wiki page`_ to
get the status of the port.


Problem description
===

OpenStack components are designed to "scale". There are differenet options
to support a lot of concurrent requests: implicit asynchronous programming,
explicit programming, threads, processes, and combination of these options.

In the past, the Nova project used Tornado, then Twisted and it is now using
eventlet which also became the defacto standard in OpenStack. The rationale to
switch from Twisted to eventlet in Nova can be found in the old `eventlet vs
Twisted
`_
article.

Eventlet issues
---

This section only gives some examples of eventlet issues. There are more
eventlet issues, but tricky issues are not widely discussed and so not well
known. Most interesting issues are issues caused by the design of eventlet,
especially the monkey-patching of the Python standard library.

Eventlet itself is not really evil. Most issues come from the monkey-patching.
The problem is that eventlet is almost always used with monkey-patching in
OpenStack.

The implementation of the monkey-patching is fragile. It's easy to forget to
patch a function or have issues when the standard libr

Re: [openstack-dev] [all] Replace eventlet with asyncio

[Mike Bayer]

I’ve spent most of the past week deeply reviewing the asyncio system,
including that I’ve constructed a comprehensive test suite designed to
discover exactly what kinds of latencies and/or throughput advantages or
disadvantages we may see each from: threaded database code, gevent-based
code using Psycopg2’s asynchronous API, and asyncio using aiopg. I’ve
written a long blog post describing a bit of background about non-blocking
IO and its use in Python, and listed out detailed and specific reasons why I
don’t think asyncio is an appropriate fit for those parts of Openstack that
are associated with relational databases. We in fact don’t get much benefit
from eventlet either in this regard, and with the current situation of
non-eventlet compatible DBAPIs, our continued use of eventlet for
database-oriented code is hurting Openstack deeply. 

My recommendations are that whether or not we use eventlet or asyncio in
order to receive HTTP connections, the parts of our application that focus
on querying and updating databases should at least be behind a thread pool.
I’ve also responded to the notions that asyncio-style programming will lead
to fewer bugs and faster production of code, and in that area I think there
are also some misconceptions regarding code that’s designed to deal with
relational databases.

The blog post is at
http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ and
you’ll find links to the test suite, which is fully runnable, within that
post.

Victor Stinner  wrote:

> Hi,
> 
> I wrote a second version of my cross-project specification "Replace eventlet 
> with asyncio". It's now open for review:
> 
> https://review.openstack.org/#/c/153298/
> 
> I copied it below if you prefer to read it and/or comment it by email. Sorry, 
> I'm not sure that the spec will be correctly formatted in this email. Use the 
> URL if it's not case.
> 
> Victor
> 
> ..
> This work is licensed under a Creative Commons Attribution 3.0 Unported
> License.
> 
> http://creativecommons.org/licenses/by/3.0/legalcode
> 
> =
> Replace eventlet with asyncio
> =
> 
> This specification proposes to replace eventlet, implicit async programming,
> with asyncio, explicit async programming. It should fix eventlet issues,
> prepare OpenStack for the future (asyncio is now part of the Python language)
> and may improve overall OpenStack performances. It also makes usage of native
> threads simpler and more natural.
> 
> Even if the title contains "asyncio", the spec proposes to use trollius. The
> name asyncio is used in the spec because it is more well known than trollius,
> and because trollius is almost the same thing than asyncio.
> 
> The spec doesn't change OpenStack components running WSGI servers like
> nova-api.  Compatibility issue between WSGI and asyncio should be solved 
> first.
> 
> The spec is focused on Oslo Messaging and Ceilometer projects. More OpenStack
> components may be modified later if the Ceilometer port to asyncio is
> successful. Ceilometer will be used to find and solve technical issues with
> asyncio, so the same solutions can be used on other OpenStack components.
> 
> Blueprint: 
> https://blueprints.launchpad.net/oslo.messaging/+spec/greenio-executor
> 
> Note: Since Trollius will be used, this spec is unrelated to Python 3. See the
> `OpenStack Python 3 wiki page `_ to
> get the status of the port.
> 
> 
> Problem description
> ===
> 
> OpenStack components are designed to "scale". There are differenet options
> to support a lot of concurrent requests: implicit asynchronous programming,
> explicit programming, threads, processes, and combination of these options.
> 
> In the past, the Nova project used Tornado, then Twisted and it is now using
> eventlet which also became the defacto standard in OpenStack. The rationale to
> switch from Twisted to eventlet in Nova can be found in the old `eventlet vs
> Twisted
> `_
> article.
> 
> Eventlet issues
> ---
> 
> This section only gives some examples of eventlet issues. There are more
> eventlet issues, but tricky issues are not widely discussed and so not well
> known. Most interesting issues are issues caused by the design of eventlet,
> especially the monkey-patching of the Python standard library.
> 
> Eventlet itself is not really evil. Most issues come from the monkey-patching.
> The problem is that eventlet is almost always used with monkey-patching in
> OpenStack.
> 
> The implementation of the monkey-patching is fragile. It's easy to forget to
> patch a function or have issues when the standard library is modified. The
> eventlet port to Python 3 showed how the patcher highly depends on the 
> standard
> library. A recent eventlet change (v0.16) "turns off __builtin__ monkey
> patching by default" to fix a tricky race condi

[openstack-dev] [all] Replace eventlet with asyncio

[Victor Stinner]

Hi,

I wrote a second version of my cross-project specification "Replace eventlet 
with asyncio". It's now open for review:

https://review.openstack.org/#/c/153298/

I copied it below if you prefer to read it and/or comment it by email. Sorry, 
I'm not sure that the spec will be correctly formatted in this email. Use the 
URL if it's not case.

Victor

..
 This work is licensed under a Creative Commons Attribution 3.0 Unported
 License.

 http://creativecommons.org/licenses/by/3.0/legalcode

=
Replace eventlet with asyncio
=

This specification proposes to replace eventlet, implicit async programming,
with asyncio, explicit async programming. It should fix eventlet issues,
prepare OpenStack for the future (asyncio is now part of the Python language)
and may improve overall OpenStack performances. It also makes usage of native
threads simpler and more natural.

Even if the title contains "asyncio", the spec proposes to use trollius. The
name asyncio is used in the spec because it is more well known than trollius,
and because trollius is almost the same thing than asyncio.

The spec doesn't change OpenStack components running WSGI servers like
nova-api.  Compatibility issue between WSGI and asyncio should be solved first.

The spec is focused on Oslo Messaging and Ceilometer projects. More OpenStack
components may be modified later if the Ceilometer port to asyncio is
successful. Ceilometer will be used to find and solve technical issues with
asyncio, so the same solutions can be used on other OpenStack components.

Blueprint: 
https://blueprints.launchpad.net/oslo.messaging/+spec/greenio-executor

Note: Since Trollius will be used, this spec is unrelated to Python 3. See the
`OpenStack Python 3 wiki page `_ to
get the status of the port.


Problem description
===

OpenStack components are designed to "scale". There are differenet options
to support a lot of concurrent requests: implicit asynchronous programming,
explicit programming, threads, processes, and combination of these options.

In the past, the Nova project used Tornado, then Twisted and it is now using
eventlet which also became the defacto standard in OpenStack. The rationale to
switch from Twisted to eventlet in Nova can be found in the old `eventlet vs
Twisted
`_
article.

Eventlet issues
---

This section only gives some examples of eventlet issues. There are more
eventlet issues, but tricky issues are not widely discussed and so not well
known. Most interesting issues are issues caused by the design of eventlet,
especially the monkey-patching of the Python standard library.

Eventlet itself is not really evil. Most issues come from the monkey-patching.
The problem is that eventlet is almost always used with monkey-patching in
OpenStack.

The implementation of the monkey-patching is fragile. It's easy to forget to
patch a function or have issues when the standard library is modified. The
eventlet port to Python 3 showed how the patcher highly depends on the standard
library. A recent eventlet change (v0.16) "turns off __builtin__ monkey
patching by default" to fix a tricky race condition: see `eventlet recursion
error after RPC timeout
`_ and `Second
simultaneous read on fileno can be raised on a closed socket #94
`_ issues. Modules implemented
in C cannot be fully monkey-patched. Recent example: the `Fix
threading.Condition with monkey-patching on Python 3.3 and newer #187
`_ change forces to use the
Python implementation of ``threading.RLock``, because the C implementation
doesn't use the monkey-patched ``threading.get_ident()`` function to get the
thread identifier, but directly a C function.

Depending on the import order, modules may or may not be monkey-patched. It's a
common trap with eventlet. Monkey-patching makes writing unit tests harder.

Some libraries must be modified to support eventlet monkey-patching. Because
they have to use original modules, not patched modules, for example. Some
patched functions behave differently, which causes issues
in applications using them. Example of an OpenStack issue report to the qpid
mailing list, `QPID and eventlet.monkey_patch()
`_:
"The lock-up occurs because select() returns that the pipe is ready to be read
from before anything has been written to the pipe".

Since eventlet uses threads, "green" threads, concurrent code must be carefully
written to avoid race condition. The section `Explicit async versus implicit
async programming`_ below explains this problem.

See also drawbacks in the `Eventlet`_ section.


Explicit async versus implicit asy