[issue22239] asyncio: nested event loop

[Kumar Aditya]

Change by Kumar Aditya :


--
nosy: +kumaraditya303

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Rob Moore]

Change by Rob Moore :


--
nosy: +rob.moore

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Douglas Raillard]

Douglas Raillard  added the comment:

Drive by comment: I landed on this thread for the exact same reason:
> This situation is very frequent when e.g. a library is designed to be 
> async-first, and also provides a blocking API which just wraps the async code 
> by running it until complete.

The library in question is "devlib", which abstracts over SSH/adb/local shell. 
We cannot make a "full" switch to async as it would be a big breaking change. 
To workaround that, I came up with a decorator that wraps a corountine, and 
"replaces" it such that:

@asyncf
async def f(...):
...

# Blocking call under its "normal" name, for backward compat
f()

# Used in an async environment
await f.asyn()

This allows converting bit by bit the whole library, with full backward 
compatibility for both users and internal calls.

On top of that, that library is heavily used in jupyter notebooks, so all in 
all, nest-asyncio is impossible to avoid.

--
nosy: +douglas-raillard-arm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Yury Selivanov]

Yury Selivanov  added the comment:

> nest-asyncio library (https://github.com/erdewit/nest_asyncio)

Seeing that the community actively wants to have support for nested loops I'm 
slowly changing my opinion on this.

Guido, maybe we should allow nested asyncio loops disabled by default?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[David Brochart]

David Brochart  added the comment:

Regarding the initial message in this issue, and enabling recursive event 
loops, this has proved to be very useful when an event loop is already running 
and some non-async code needs to run async code. This situation is very 
frequent when e.g. a library is designed to be async-first, and also provides a 
blocking API which just wraps the async code by running it until complete.
The nest-asyncio library (https://github.com/erdewit/nest_asyncio) allows that 
by patching asyncio's event loop, but obviously this doesn't work with other 
event loops such as uvloop. I was wondering if things had changed since the 
original post of this issue, and if such a feature had any chance to make it 
into the standard library.

--
nosy: +davidbrochart

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

I tested "cancellation", shutting down the DB connection mid query.  Because 
the greenlet is only in the middle and not at the endpoints, it propagates the 
exception and there does not seem to be anything different except for the 
greenlet sequence in the middle, which is also clear:

https://gist.github.com/zzzeek/9e0d78eff14b3bbd5cf12fed8b02bce6

the first comment on the gist has the stack trace produced.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

as far as cancellation, I gather you're referring to what in gevent / greenlet 
is the GreenletExit exception.  Sure, that thing is a PITA.   Hence we're all 
working to provide asyncio frontends and networking backends so that the 
effects of cancellation I (handwavy handwavy) believe would work smoothly as 
long as the middle part is done right.   cancellation is likely a more 
prominent issue with HTTP requests and responses because users are hitting 
their browser stop buttons all the time.  With databases this typically is 
within the realm of network partitioning or service restarts, or if the driver 
is screwing up in some way which with the monkeypatching thing is more likely, 
but "cancellation" from a database perspective is not the constant event that I 
think it would be in an HTTP perspective.


> I think I either disagree or am missing something :-). Certainly for both 
> edgedb and urllib3, when they're running in sync mode, they end up using 
> synchronous network APIs at the "bottom", and it works fine.

OK it took me a minute to understand what you're saying, which is, if we are 
doing the coroutine.send() thing you illustrated below, we're not in an event 
loop anyway so we can just call blocking code.   OK I did not understand that.  
I haven't looked at the coroutine internals through all of this (which is part 
of my original assertion that I should not have been the person proposing this 
whole greenlet thing anyway :) ).

Why did urllib3 write unasync?  https://pypi.org/project/unasync/strictly 
so they can have a python 2 codebase and that's it?   

SQLAlchemy goes python 3 only in version 2.0.  I did bench the coro example 
against a non-coro example and it's 3x slower likely due to the StopIteration 
but as mentioned earlier if this is only once per front-to-back then it would 
not amount to anything in context.   Still, the risk factor of a rewrite like 
that, where risk encompasses just all the dumb mistakes and bugs that would be 
introduced by rewriting everything, does not seem worth it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[pmp-p]

Change by pmp-p :


--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Nathaniel Smith]

Nathaniel Smith  added the comment:

> 90% of everything people are doing here are in the context of HTTP services.  
>  The problem of, "something.a now creates state that other tasks might see" 
> is not a real "problem" that is solved by using IO-only explicit context 
> switching.  This is because in a CRUD-style application, "something" is not 
> going to be a process-local yet thread-global object that had to be created 
> specifically for the application (there's things like the database connection 
> pool and some registries that the ORM uses, but those problems are taken care 
> of and aren't specific to one particular application).

Yeah, in classic HTTP CRUD services the concurrency is just a bunch of 
stateless handlers running simultaneously. This is about the simplest possible 
kind of concurrency. There are times when async is useful here, but to me the 
main motivation for async is for building applications with more complex 
concurrency, that currently just don't get written because of the lack of good 
frameworks. So that 90% number might be accurate for right now, but I'm not 
sure it's accurate for the future.

> In the realm of Python HTTP/CRUD applications, async is actually very popular 
> however it is in the form of gevent and sometimes eventlet monkeypatching, 
> often because people are using async web servers like gunicorn.

A critical difference between gevent-style async and newer frameworks like 
asyncio and trio is that the newer frameworks put cancellation support much 
more in the foreground. To me cancellation is the strongest argument for 
'await' syntax, so I'm not sure experience with gevent is representative.

I am a bit struck that you haven't mentioned cancellation handling at all in 
your replies. I can't emphasize enough how much cancellation requires care and 
attention throughout the whole ecosystem.

> w.r.t the issue of writing everything as async and then using the coroutine 
> primitives to convert to "sync" as means of maintaining both facades, I don't 
> think that covers the fact that most DBAPI drivers are sync only

I think I either disagree or am missing something :-). Certainly for both 
edgedb and urllib3, when they're running in sync mode, they end up using 
synchronous network APIs at the "bottom", and it works fine.

The greenlet approach does let you skip adding async/await annotations to your 
code, so it saves some work that way. IME this isn't particularly difficult 
(you probably don't need to change any logic at all, just add some extra 
annotations), and to me the benefits outweigh that, but I can see how you might 
prefer greenlets either temporarily as a transition hack or even in the long 
term.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

slight correction: it is of course possible to use gevent with a database 
driver without monkeypatching, as I wrote my own gevent benchmarks using 
psycogreen.  I think what I'm getting at is that it's a good thing if async 
DBAPIs could target asyncio explicitly rather than having to write different 
gevent/eventlet specific things, and that tools like SQLAlchemy can allow for 
greenlet style coding against those DBAPIs without one having to install/run 
the whole gevent event loop.   Basically I like the greenlet style of coding 
but I would be excited to skip the gevent part, never do any monkeypatching 
again, and also have other parts of the app doing asyncio work with other kinds 
of services. this is about interoperability.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

> Oh, I thought the primary problem for SQLAlchemy supporting async is that the 
> ORM needs to do IO from inside __getattr__ methods. So I assumed that the 
> reason you were so excited about greenlets was that it would let you use 
> await_() from inside those __getattr__ calls, which would involve exposing 
> your use of greenlets as part of your public API.


The primary problem is people want to execute() a SQL statement using await, 
and then they want to use a non-blocking database driver (basically asyncpg, 
I'm not sure there are any others, maybe there's one for MySQL also) on the 
back.Tools like aiopg have provided partial SQLAlchemy-like front-ends to 
accomplish this but they can't do ORM support, not because the ORM has lazy 
loading, but just to do explicit operations like query.all() or session.flush() 
that can sometimes require a lot of front-to-back database operations to 
complete which would be very involved to rewrite all that code using 
async/await.

Then there's the secondary problem of ORMs doing lazy loading, which is what 
you refer towards as "IO inside __getattr__ methods".   SQLAlchemy is not 
actually as dependent on lazy loading as other ORMs as we support a wide range 
of ways to "eagerly" load data up front.  With the SQLAlchemy 2.0-style ORM API 
that has a clear spot for "await" to occur, they can call "await 
session.execute(select(SomeObject))" and get a whole traversible graph of 
things loaded up front.We even have a loader called "raiseload" that is 
specifically anti-lazy loading, it's a loader that raises an error if you try 
to access something that wasn't explicitly loaded already.  So for a lot of 
cases we are already there.

But then, towards your example of "something.b = x", or more commonly in ORMS a 
get operation like "something.b" emitting SQL, the extension I'm building will 
very likely include some kind of feature that they can do this with an explicit 
call.  At the moment with the preliminary code that's in there, this might look 
like:

   await greenlet_spawn(getattr, something, "b")

not very pretty at the moment but that general idea.   

But the thing is, greenlet_spawn() can naturally apply to anything.  So it 
remains to be seen both how I would want to present this notion, as well as if 
people are going to be interested in it or not, but as a totally extra thing 
beyond the "await session.execute()" API that is the main thing, someone could 
do something like this:

   await greenlet_spawn(my_business_orm_method)

and then in "my_business_orm_method()", all the blocking style ORM things that 
async advocates warn against could be happening in there. I'm certainly not 
going to tell people they have to be doing that, but I dont think I should 
discourage it either, because if the above business method is written 
"reasonably" (see next paragraph), there really is no problem introduced by 
implicit IO.

By "written reasonably" I'm referring to the fact that in this whole situation, 
90% of everything people are doing here are in the context of HTTP services.   
The problem of, "something.a now creates state that other tasks might see" is 
not a real "problem" that is solved by using IO-only explicit context 
switching.  This is because in a CRUD-style application, "something" is not 
going to be a process-local yet thread-global object that had to be created 
specifically for the application (there's things like the database connection 
pool and some registries that the ORM uses, but those problems are taken care 
of and aren't specific to one particular application). There is certainly 
going to be global mutable state with the CRUD/HTTP application which is the 
database itself.  Event based programming doesn't save you from concurrency 
issues here because any number of processes maybe accessing the database at the 
same time.  There are well-established concurrency patterns one uses wit
 h relational databases, which include first and foremost transaction 
isolation, but also things like compare-and-swap, "select for update", ensuring 
MVCC is turned on (SQL Server), table locks, etc.  These techniques are 
independent of the concurrency pattern used within the application, and they 
are arguably better suited to blocking-style code in any case because on the 
database side we must emit our commands within a transaction serially in any 
case.   The major convenient point of "async" that we can fire off a bunch of 
web service requests in parallel does not apply to the CRUD-style business 
methods within our web service request because we can only do things in our 
ACID transaction one at a time.

The problem of "something.a" emitting IO needs to be made sane against other 
processes also viewing or altering "something.a", assuming "something" is a 
database-bound object like a row in a table, using traditional database 
concurrency constructs such as choosing an appropriate isolation 

[issue22239] asyncio: nested event loop

[Nathaniel Smith]

Nathaniel Smith  added the comment:

> Whether or not one buys that, the point of my approach is that SQLAlchemy 
> itself *will* publish async methods.  End user code *will not* ever context 
> switch to another task without them explicitly calling using an await.

Oh, I thought the primary problem for SQLAlchemy supporting async is that the 
ORM needs to do IO from inside __getattr__ methods. So I assumed that the 
reason you were so excited about greenlets was that it would let you use 
await_() from inside those __getattr__ calls, which would involve exposing your 
use of greenlets as part of your public API.

If you're just talking about using greenlets internally and then writing both 
sync and async shims to be your public API, then obviously that reduces the 
risks. Maybe greenlets will cause you problems, maybe not, but either way you 
know what you're getting into and the decision only affects you :-). But, if 
that's all you're using them for, then I'm not sure that they have a 
significant advantage over the edgedb-style synchronous wrapper or the 
unasync-style automatically generated sync code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Yury Selivanov]

Yury Selivanov  added the comment:

> The community is hurting *A LOT* right now because asyncio is intentionally 
> non-compatible with the traditional blocking approach that is not only still 
> prevalent it's one that a lot of us think is *easier* to work with.

Mike, I'm super happy with having you here and I encourage you to propose 
feature requests etc. That said, please don't use arguments like this here. 
Everyone has their own point of view and I, for example, haven't seen the "A 
LOT of community hurt" you're describing. I'm not implying that what you're 
saying is wrong, or that asyncio is perfect; the point is that it's just very 
subjective. The bug tracker is not the medium for these kind of remarks.

> That SQLAlchemy internally is not using this coding style, whether or not 
> that leads to new kinds of bugs, there are new kinds of bugs no matter what 
> kind of code a library uses, I don't think this hurts the user community.

You're free to use whatever approach you want in SQLAlchemy. We're here to 
share our advice and perspective (if we have any) and/or to discuss concrete 
proposals for API improvements or changes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

> With greenlets OTOH, it becomes possible for another task to observe 
> someobj.a == 1 without someobj.b == 2, in case someobj.__setattr__ internally 
> invoked an await_().

let me try this one more time.Basically if someone wrote this:

async def do_thing():
   someobj.a =1
   await do_io_setattr(someobj, "b", 2)

then in the async approach, you can again say, assuming "someobj" is global, 
that another task can observe "someobj.a == 1" without "someobj.b == 2".I 
suppose you are making the argument that because there's an "await" keyword 
there, now everything is OK because the reader of the code knows there's a 
context switch.

Whether or not one buys that, the point of my approach is that SQLAlchemy 
itself *will* publish async methods.  End user code *will not* ever context 
switch to another task without them explicitly calling using an await.  That 
SQLAlchemy internally is not using this coding style, whether or not that leads 
to new kinds of bugs, there are new kinds of bugs no matter what kind of code a 
library uses, I don't think this hurts the user community.  The community is 
hurting *A LOT* right now because asyncio is intentionally non-compatible with 
the traditional blocking approach that is not only still prevalent it's one 
that a lot of us think is *easier* to work with.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

> With greenlets OTOH, it becomes possible for another task to observe 
> someobj.a == 1 without someobj.b == 2, in case someobj.__setattr__ internally 
> invoked an await_(). Any operation can potentially invoke a context switch. 
> So debugging greenlets code is roughly as hard as debugging full-on 
> multithreaded code, and much harder than debugging async/await code.

I would invite you to look more closely at my approach.   The situation you 
describe above applies to a library like gevent, where IO means a context 
switch that can go anywhere.  My small recipe never breaks out of the asyncio 
event loop, and it only context switches within the scope of a single 
coroutine, not to any arbitrary coroutine.   So I don't think the above issue 
applies.

Additionally, we are here talking about *libraries* that are independently 
developed and tested distinct from end-user code.If there's a bug in 
SQLAlchemy, the end user isn't the person debugging that.   arguments over "is 
async or sync easier to debug" are IMO pretty subjective and at this point they 
are not relevant to what sync-based libraries should be doing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Yury Selivanov]

Yury Selivanov  added the comment:

> Yeah, writing a trivial "event loop" to drive actually-synchronous code is 
> easy. Try it out:

This is exactly the approach I used in edgedb-python.

> I guess there's technically some overhead, but it's tiny.

Correct, the overhead isn't even detectable in microbenchmarks. In most async 
programs regular function calls dominate awaits by a few factors of magnitude.

> I think dropping 'await' syntax has two major downsides:

For the extra context, in the case of using this approach for something like 
edgedb-python these downsides don't really apply, because we're adapting 
async/await implementation to be sync. The async/await code can handle 
cancellation etc. whereas the sync code only needs to support the general 
protocol parsing flow. FWIW I don't think it would be possible to apply my 
approach to SQLA without a very invasive rewrite, which isn't probably worth it.

> tl;dr: I think switching from async/await -> greenlets would make it much 
> easier to write programs that are 90% correct, and much harder to write 
> programs that are 100% correct. That might be a good tradeoff in some 
> situations, but it's a lot more complicated than it seems.

Yeah, this sums up my opinion on this topic.

Also, having spent a couple of years writing and debugging big and small 
greenlet-heavy code bases I wouldn't want to touch them ever again. Your 
mileage may vary, Mike.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Nathaniel Smith]

Nathaniel Smith  added the comment:

Yeah, writing a trivial "event loop" to drive actually-synchronous code is 
easy. Try it out:

-

async def f():
print("hi from f()")
await g()

async def g():
print("hi from g()")

# This is our event loop:
coro = f()
try:
coro.send(None)
except StopIteration:
pass

-

I guess there's technically some overhead, but it's tiny.

I think dropping 'await' syntax has two major downsides:

Downside 1: 'await' shows you where context switches can happen: As we know, 
writing correct thread-safe code is mind-bendingly hard, because data can 
change underneath your feet at any moment. With async/await, things are much 
easier to reason about, because any span of code that doesn't contain an 
'await' is automatically atomic:

---
async def task1():
# These two assignments happen atomically, so it's impossible for
# another task to see 'someobj' in an inconsistent state.
someobj.a = 1
someobj.b = 2
---

This applies to all basic operations like __getitem__ and __setitem__, 
arithmetic, etc. -- in the async/await world, any combination of these is 
automatically atomic.

With greenlets OTOH, it becomes possible for another task to observe someobj.a 
== 1 without someobj.b == 2, in case someobj.__setattr__ internally invoked an 
await_(). Any operation can potentially invoke a context switch. So debugging 
greenlets code is roughly as hard as debugging full-on multithreaded code, and 
much harder than debugging async/await code.

This first downside has been widely discussed (e.g. Glyph's "unyielding" blog 
post), but I think the other downside is more important:

- 'await' shows where cancellation can happen: Synchronous libraries don't have 
a concept of cancellation. OTOH async libraries *are* expected to handle 
cancellation cleanly and correctly. This is *not* trivial. With your 
sqlalchemy+greenlets code, you've introduced probably hundreds of extra 
unwinding paths that you've never tested or probably even thought about. How 
confident are you that they all unwind correctly (e.g. without corrupting 
sqlalchemy's internal state)? How do you plan to even find them, given that you 
can't see the cancellation points in your code? How can your users tell which 
operations could raise a cancelled exception?

AFAICT you can't reasonably build systems that handle cancellation correctly 
without some explicit mechanism to track where the cancellation can happen. 
There's a ton of prior art here and you see this conclusion over and over.

tl;dr: I think switching from async/await -> greenlets would make it much 
easier to write programs that are 90% correct, and much harder to write 
programs that are 100% correct. That might be a good tradeoff in some 
situations, but it's a lot more complicated than it seems.

--
nosy: +njs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Joshua Bronson]

Change by Joshua Bronson :


--
nosy: +jab

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

yes so if you have async/await all internal, are you saying you can make that 
work for synchronous code *without* running the event loop?  that is, some kind 
of container that just does the right thing?  my concern with that would still 
be performance.When asyncio was based on yield and exception throws, that 
was a lot of overhead to add to functions and that was what my performance 
testing some years back showed.   w/ async/await I'm sure things have been 
optimized, but in general when i have function a() -> b() -> c(), I am trying 
to iron as much Python overhead as I possibly can out of that and I'd be 
concerned that the machinery to work through async/await would add latency.   
additionally if it was async/await internally but then i need to access the 
majority of Python DBAPIs that are sync, I need a thread pool anyway, right?  
which is also another big barrier to jump over.

It seems you were involved with urllib3's approach to use a code rewriter 
rather than a runtime approach based on the discussion at 
https://github.com/urllib3/urllib3/issues/1323 , but it's not clear if Python 2 
compatibility was the only factor or if the concern of "writing a giant shim" 
was also.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Yury Selivanov]

Yury Selivanov  added the comment:

> I think this is a really critical technique to have so that libraries that 
> mediate between a user-facing facade and TCP based backends no longer have to 
> make a hard choice about if they are to support sync vs. async (or async with 
> an optional sync facade around it).

If this works for such a big and elaborate framework as SQLA, we can definitely 
highlight this as a valid approach and even add a link to a blog post from the 
docs. We'll need to add an asyncio specific FAQ page for that or something 
similar.

Another approach, which would probably be a nonstarter for SQLA, is to use 
async/await for literally everything internally, and provide a tiny synchronous 
facade on top.  Funny thing you don't even need an event loop for that, just 
the basic understanding of how coroutines work internally.  I used this to 
create the edgedb-python package which has both sync and async first-class 
support with one code base.  Sync is even faster there in simple throughput 
benchmarks (as expected).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

>  This recipe was one of the reasons why I added `loop.set_task_factory` 
> method to the spec, so that it's possible to implement this in an *existing* 
> event loop like uvloop. But ultimately asyncio is flexible enough to let 
> users use their own event loop which can be compatible with greenlets among 
> other improvements.

Right, when I sought to look at this, I know that my users want to use the 
regular event loop in asyncio or whatever system they are using.


> Ultimately, asyncio will probably never ship with greenlets integration 
> enabled by default, but we should definitely make it possible (if there are 
> some limitations now).  It doesn't seem to me that nested event loops are 
> needed for this, right?

So right, the approach I came up with does not need nested event loops and it's 
my vague understanding that nesting event loops is more difficult to debug, 
because you have these two separate loops handing off to each other.

What is most striking about my recipe is that it never even leaves the default 
event loop.  Originally my first iteration when I was trying to get it working, 
I had a separate thread going on, as it seemed intuitive that "of course you 
need a thread to bridge async and sync code" but then I erased the "Thread()" 
part around it and then it just worked anyway.   Like it's simple enough that 
shipping this as a third party library is almost not even worth it, you can 
just drop this in wherever.   If different libraries had their own drop-in of 
this, they would even work together.   greenlet is really like totally 
cheating.  

the philosophical thing here is, usually in my many twitter debates on the 
subject, folks get into how they like the explicit async and await keywords and 
they like that IO is explicit.   So I'm seeking to keep these people happy and 
give then "async def execute(sql)", and use an async DB driver, but then the 
library that these folks are using is internally not actually explicit IO.  But 
they don't have to see that, I've done all the work of the "implicit IO" stuff 
and in a library it's not the same kind of problem anyway. I think this is 
a really critical technique to have so that libraries that mediate between a 
user-facing facade and TCP based backends no longer have to make a hard choice 
about if they are to support sync vs. async (or async with an optional sync 
facade around it).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Yury Selivanov]

Yury Selivanov  added the comment:

Thanks for posting this, Mike.

> Vague claims of "framework X is faster because it's async" appear, impossible 
> to confirm as it is unknown how much of their performance gains come from the 
> "async" aspect and how much of it is that they happened to rewrite a new 
> framework from scratch in a completely different way (hint: it's the latter).

These kind of claims are not specific to async vs. sync. They are all over the 
place for every two pieces of comparable technologies. While novice users might 
base their technology choice purely on such benchmarks, it's less of an issue 
for startups/tech companies.

That said, I agree with most of your points so far.

> The asyncpg project, one of the few asyncio database drivers that exists, 
> notes in its FAQ "asyncpg uses asynchronous execution model and API, which is 
> fundamentally incompatible with SQLAlchemy" [2], yet we know this is not true 
>  because SQLAlchemy works just fine with gevent and eventlet, with no 
> architectural changes at all.

But it is true. Making asynchronous network requests in asyncio requires 
async/await or using callbacks and it's not possible to do them, say, from 
__getattr__ (you mention this yourself).  This is what that particular comment 
is about, nothing more. Using gevent and eventlet as examples in this 
particular context isn't helping you. Apologies for nitpicking, I know it's not 
the point of this discussion.

> A day later, someone took the same idea and got Flask to work in an asyncio 
> event loop at [5].  The general idea of using greenlet in this way is also 
> present at [6], so I won't be patenting this idea today as oremanj can claim 
> prior art.

Yes, this approach definitely works and I even did that in production myself a 
few years ago with https://github.com/1st1/greenio (it's terribly outdated now).

> The recipe is simple and so far appears to be very effective.

This recipe was one of the reasons why I added `loop.set_task_factory` method 
to the spec, so that it's possible to implement this in an *existing* event 
loop like uvloop. But ultimately asyncio is flexible enough to let users use 
their own event loop which can be compatible with greenlets among other 
improvements.

Ultimately, asyncio will probably never ship with greenlets integration enabled 
by default, but we should definitely make it possible (if there are some 
limitations now).  It doesn't seem to me that nested event loops are needed for 
this, right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[mike bayer]

mike bayer  added the comment:

hey there,

I seem to have two cents to offer so here it is.An obscure issue in the 
Python bug tracker is probably not the right place for this so consider this as 
an early draft of something that maybe I'll talk about more elsewhere.

> This basically divides code into two islands - async and non-async

yes, this is the problem, and at the bottom of this apparently somewhat ranty 
comment is a solution, and the good news is that it does not require Python or 
asyncio be modified.  My concern is kind of around how it is that everyone has 
been OK with the current state of affairs for so long, why it is that "asyncio 
is fundamentally incompatible with library X" is considered to be acceptable, 
and also how easy it was to find a workaround, this is not something I would 
have expected to come up with.  Kind of like you don't expect to invent Velcro 
or windshield wipers.

asyncio's approach is what those of us in the library/framework community call 
"explicit async", you have to mark functions that will be doing IO and the 
points at which IO occurs must also be marked.Long ago it was via callback 
functions, then asyncio turned it into decorators and yields, and finally 
pep492 turned it into async/await, and it is very nicely done.  It is of course 
a feature of asyncio that writing out async/await means your code can in theory 
be clearer as to where IO occurs and all that, and while I don't totally buy 
that myself, I'm of course in favor of that style of coding being available, it 
definitely has its own kind of self-satisfaction built in when you do it.  
That's all great.

But as those of us in the library/framework community also know, asyncio's 
approach essentially means, libraries like Flask, Django, my own SQLAlchemy, 
etc. are all automatically "non-workable" with the asyncio approach; while 
these libraries can certainly have asyncio endpoints added to them, the task as 
designed is not that simple, since to go from an asyncio endpoint all the way 
through library code that doesn't care about async and then down into a 
networking library that again has asyncio endpoints, the publishing of "async" 
and the "await" or yield approach must be wired all the way through every 
function and method.  This is all despite that when you're not at the 
endpoints, the points at which IO occurs is fully predictable such that 
libraries like gevent don't need you to write it.   So we are told that 
libraries have to have full end-to-end rewrites of all their code to work this 
way, or otherwise maintain two codebases, or something like that.

The side effect of this is that a whole bunch of library and framework authors 
now get to create all new libraries and frameworks, which do exactly the same 
thing as all the existing libraries and frameworks, except they sprinkle the 
"async/await" keywords throughout middle tiers as required.  Vague claims of 
"framework X is faster because it's async" appear, impossible to confirm as it 
is unknown how much of their performance gains come from the "async" aspect and 
how much of it is that they happened to rewrite a new framework from scratch in 
a completely different way (hint: it's the latter).

Or in other cases, as if to make it obvious how much the "async/await" keywords 
come down to being more or less boilerplate for the "middle" parts of 
libraries, the urllib3 project wrote the "unasync" project [1] so that they can 
simply maintain two separate codebases, one that has "async/await" and  the 
other which just search-and-replaced them out.

SQLAlchemy has not been "replaced" by this trend as asyncio database libraries 
have not really taken off in Python, and there are very few actual async 
drivers.   Some folks have written SQLAlchemy-async libraries that use 
SQLAlchemy's expression system while they have done the tedious, redundant and 
impossible-to-maintain work of replicating enough of SQLAlchemy's execution 
internals such that a modest "sqlalchemy-like" experience with asyncio can be 
reproduced. But these libraries are closed out from all of the fixes and 
improvements that occur to SQLAlchemy itself, as well as that these systems 
likely target a smaller subset of SQLAlchemy's behaviors and features in any 
case.They certainly can't get the ORM working as the ORM runs lots of SQL 
executions internally, all of which would have to propagate their "asyncness" 
outwards throughout hundreds of functions.

The asyncpg project, one of the few asyncio database drivers that exists, notes 
in its FAQ "asyncpg uses asynchronous execution model and API, which is 
fundamentally incompatible with SQLAlchemy" [2], yet we know this is not true  
because SQLAlchemy works just fine with gevent and eventlet, with no 
architectural changes at all.  Using libraries like SQLAlchemy or Django with a 
non-blocking IO, event-based model is commonplace.   It's the "explicit" part 
of it that is hard, which is because of 

[issue22239] asyncio: nested event loop

[Crusader Ky]

Change by Crusader Ky :


--
nosy: +Crusader Ky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Jesús Cea Avión]

Change by Jesús Cea Avión :


--
nosy: +jcea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Rokas K. (rku)]

Rokas K. (rku) added the comment:

I understand rationale for rejection of this issue but i beg to reconsider.

Unlike in traditional coroutines (windows fibers / setjmp|longjmp with stack 
switching) we can not yield from any point of execution. There must be full 
async-await chain preserved. This basically divides code into two islands - 
async and non-async. And there seems to be no way to schedule async call from 
non-async code and get a response. While suggestion to make custom `async def 
contains()` call is a valid one we can not always do that. Consider the case 
when we have to do some networking calls in a function that is invoked by 
non-async library. Naturally it would be simple non-awaited call from which we 
can not call a coroutine and get a response. And since it is a library calling 
into our code we can not easily change it. It might even be completely 
unsuitable change for library in question.

I see two solutions to this problem (if i am missing something please point it 
out):

1. Reentrant loops as suggested in this issue.
2. Allow awaited calls from non-coroutines provided execution is invoked from a 
coroutine somewhere up the call stack.

First one is certainly easier to implement.

--
nosy: +Rokas K. (rku)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Daniel Arbuckle]

Daniel Arbuckle added the comment:

All right.

However, for anyone who's interested, here is a patch that enables nested event 
loops in asyncio, and the accompanying unit tests

--
keywords: +patch
Added file: http://bugs.python.org/file36498/nested.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22239
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Guido van Rossum]

Guido van Rossum added the comment:

While I understand your problem, I really do not want to enable recursive event 
loops. While they are popular in some event systems (IIRC libevent relies 
heavily on the concept), I have heard some strong objections from other parts, 
and I am trying to keep the basic event loop functionality limited to encourage 
interoperability with other even loop systems (e.g. Tornado, Twisted).

In my own experience, the very programming technique that you are proposing has 
caused some very hard to debug problems that appeared as very infrequent and 
hard to predict stack overflows.

I understand this will make your code slightly less elegant in some cases, but 
I think in the end it is for the best if you are required to define an explicit 
method (declared to be a coroutine) for membership testing of a remote object.  
The explicit yield from will help the readers of your code understand that 
global state may change (due to other callbacks running while you are blocked), 
and potentially help a static analyzer find bugs in your code before they take 
down your production systems.

--
resolution:  - wont fix
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22239
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22239
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22239] asyncio: nested event loop

[Daniel Arbuckle]

New submission from Daniel Arbuckle:

It's occasionally necessary to invoke the asyncio event loop from code that was 
itself invoked within (although usually not directly by) the event loop.

For example, imagine you are writing a class that serves as a local proxy for a 
remote data structure. You can not make the __contains__ method of that class 
into a coroutine, because Python automatically converts the return value into a 
boolean. However, __contains__ must invoke coroutines in order to communicate 
over the network, and it must be invokable from within a coroutine to be at all 
useful.

If the event loop _run_once method were reentrant, addressing this problem 
would be simple. That primitive could be used to create a loop_until_complete 
function, which could be applied to the io tasks that __contains__ needs to 
invoke

So, making _run_once reentrant is one way of addressing this request.

Alternately, I've attached a decorator that sets aside some of the state of 
_run_once, runs a couroutine to completion in a nested event loop, restores the 
saved state, and returns the coroutine's result. This is merely a proof of 
concept, but it does work, at least in my experiments.

--
components: asyncio
files: nested.py
messages: 225578
nosy: djarb, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: asyncio: nested event loop
type: enhancement
Added file: http://bugs.python.org/file36422/nested.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22239
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com