Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 6, 2016, at 3:54 PM, David Beazley  wrote:
> 
> 
>> However, as far as I know curio doesn’t have the ability to schedule an 
>> operation in a synchronous manner by means of something like a Future. Is 
>> that correct? If there is no way in curio to spawn a task to occur later 
>> without having to await on it, then clearly there is no option but to allow 
>> coroutines in __aexit__ and finally: how else could curio operate?
>> 
> 
> Yes, Curio does not utilize Futures or callback functions for that matter.   
> However, I think the real issue at hand might be much more subtle and weird.
> 
> If I understand things correctly, the goal is to implement an asynchronous 
> generator for feeding asynchronous iteration.  This is a new kind of 
> generator that runs inside of a coroutine (so, imagine a coroutine inside of 
> another coroutine).   Without seeing much more, I'd guess it would look 
> something roughly akin to this:
> 
> async def agen():
>   ... some sort of async generator ...
>   async yield value #  Syntax 

I my current WIP branch I just use ‘yield’ inside ‘async def’.  That’s what I 
was going to propose to do in the PEP.

> 
> async def main():
>   async for x in agen():
>  ...
> 
> There's some kind of underlying protocol driving the async iteration, but 
> it's *different* than what is being used by the enclosing coroutines.   Yes, 
> there is a scheduler kernel (or event loop) that makes the outer coroutines 
> run, but that scheduler is not driving the underlying iteration protocol of 
> the async generator part.   So, things get weird when stuff like this happens:
> 
> async def main():
>   async for x in agen():
>   if x == STOP:
> break

Good catch.

[..]

> Since forgetting that last close() step would be easy, naturally an async 
> generator should support the asynchronous context-management protocol.
> 
> async def main():
>   async with agen() as items:
>  async for x in items:
>  if x == STOP:
>break

This is an interesting idea, but I wonder if instead of using ‘async with’ we 
can actually augment ‘async for’ to do the async finalization.

We can add an __aiter_close__ special method which will return an awaitable.  
In this case, ‘async for’ can always look for that method and call it at the 
end of the iteration.  Async generators will implement the method to make sure 
that ‘finally’ is always executed (any number of awaits in ‘finally’ is OK; 
‘yield’ expressions cannot be used).

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[David Beazley]

> However, as far as I know curio doesn’t have the ability to schedule an 
> operation in a synchronous manner by means of something like a Future. Is 
> that correct? If there is no way in curio to spawn a task to occur later 
> without having to await on it, then clearly there is no option but to allow 
> coroutines in __aexit__ and finally: how else could curio operate?
> 

Yes, Curio does not utilize Futures or callback functions for that matter.   
However, I think the real issue at hand might be much more subtle and weird.

If I understand things correctly, the goal is to implement an asynchronous 
generator for feeding asynchronous iteration.  This is a new kind of generator 
that runs inside of a coroutine (so, imagine a coroutine inside of another 
coroutine).   Without seeing much more, I'd guess it would look something 
roughly akin to this:

async def agen():
   ... some sort of async generator ...
   async yield value #  Syntax 

async def main():
   async for x in agen():
  ...

There's some kind of underlying protocol driving the async iteration, but it's 
*different* than what is being used by the enclosing coroutines.   Yes, there 
is a scheduler kernel (or event loop) that makes the outer coroutines run, but 
that scheduler is not driving the underlying iteration protocol of the async 
generator part.   So, things get weird when stuff like this happens:

async def main():
   async for x in agen():
   if x == STOP:
 break

Here, the asynchronous generator makes it to a point, but never to completion 
because of the break statement.  Instead, it gets garbage collected and all 
hell breaks loose back in the agen() function because what happens now?  
Especially if agen() uses finally or an async context manager:

async def agen():
   async with whatever:
...

Assuming that this is getting to the heart of the issue, I spent some time 
pondering it this morning and almost wonder if it could be solved by 
"underthinking" the solution so to speak.  For example, perhaps the __del__() 
method of an async-generator could just raise a RuntimeError if it's ever 
garbage collected before being properly terminated. Maybe you give asynchronous 
generators an async-close method to explicitly shut it down.  So, you'd have to 
do this.

async def main():
   items = agen()
   async for x in items:
 if x == STOP:
break
  await items.close()

Maybe the async-close() method would merely raise AsyncGeneratorExit in the 
generator and not enforce any other kind of semantics other than having it 
continue to run as a coroutine as it did before (with the understanding that 
the purpose of the exception is to terminate eventually). 

Since forgetting that last close() step would be easy, naturally an async 
generator should support the asynchronous context-management protocol.

async def main():
   async with agen() as items:
  async for x in items:
  if x == STOP:
break

Perhaps the only thing missing at this point is a metaclass---or possibly a 
codec.  I'm not sure. 

Yikes

Cheers,
Dave

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Nathaniel Smith]

On Wed, Jul 6, 2016 at 6:42 AM, Cory Benfield  wrote:
>
>> On 6 Jul 2016, at 13:09, David Beazley  wrote:
>>
>> Curio uses asynchronous context managers for much more than closing sockets 
>> (which frankly is the least interesting thing).   For example, they're used 
>> extensively with synchronization primitives such as Locks, Semaphores, 
>> Events, Queues, and other such things.   The ability to use coroutines in 
>> the __aexit__() method is an essential part of these primitives because it 
>> allows task scheduling decisions to be made in conjunction with 
>> synchronization events such as lock releases.   For example, you can 
>> implement fair-locking or various forms of priority scheduling.   Curio also 
>> uses asynchronous context managers for timeouts and other related 
>> functionality where coroutines have to be used in __aexit__.  I would expect 
>> coroutines in __aexit__ to also be useful in more advanced contexts such as 
>> working with databases, dealing with transactions, and other kinds of 
>> processing where asynchronous I/O might be involved.
>
> For my own edification, Dave, do you mind if I dive into this a little bit? 
> My suspicion is that this problem is rather unique to curio (or, at least, 
> does not so strongly effect event loop implementations), and I’d just like to 
> get a handle on it. It’s important to me that we don’t blow away curio, so 
> where it differs from event loops I’d like to understand what it’s doing.

The relevant difference between curio and asyncio here is that in
asyncio, there are two different mechanisms for accessing the event
loop: for some operations, you access it through its coroutine runner
interface using 'await', and for other operations, you get a direct
reference to it through some side-channel (loop= arguments, global
lookups) and then make direct method calls. To make the latter work,
asyncio code generally has to be written in a way that makes sure
loop= objects are always passed throughout the whole call stack, and
always stay in sync [no pun intended] with the coroutine runner
object. This has a number of downsides, but one upside is that it
means that the loop object is available from __del__, where the
coroutine runner isn't.

Curio takes the other approach, of standardizing on 'await' as the
single blessed mechanism for accessing the event loop (or kernel or
whatever you want to call it). So this eliminates all the tiresome
loop= tracking and the potential for out-of-sync bugs, but it means
you can't do *anything* event-loop-related from __del__.

However, I'm not convinced that asyncio really has an advantage here.
Imagine some code that uses asyncio internally, but exposes a
synchronous wrapper to the outside (e.g. as proposed here ;-) [1]):

def synchronous_wrapper(...):
# Use a new loop since we might not be in the same thread as the global loop
loop = asyncio.new_event_loop()
return loop.run_until_complete(work_asynchronously(..., loop=loop))

async def work_asynchronously(..., loop=loop):
stream = await get_asyncio_stream_writer(..., loop=loop)
stream.write(b"hello")

stream.write(...) queues some data to be sent, but doesn't send it.
Then stream falls out of scope, which triggers a call to
stream.__del__, which calls stream.close(), which does some calls on
loop requesting that it flush the buffer and then close the underlying
socket. So far so good.

...but then immediately after this, the loop itself falls out of
scope, and you lose. AFAICT from a quick skim of the asyncio code, the
data will not be sent and the socket will not be closed (depending on
kernel buffering etc.).

(And even if you wrap everything with proper 'with' blocks, this is
still true... asyncio event loops don't seem to have any API for
"complete all work and then shut down"? Maybe I'm just missing it --
if not then this is possibly a rather serious bug in actual
currently-existing asyncio. But for the present purposes, the point is
that you really do need something like 'async with' around everything
here to force the I/O to complete before handing things over to the gc
-- you can't rely on the gc to do your I/O.)

-n

[1] https://github.com/kennethreitz/requests/issues/1390#issuecomment-225361421

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/