[Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Matthias Urlichs]

The following code has a problem: the generator returned by .wait() has a 
finally: section. When self.stopped is set, it still needs to run. As it is 
asynchronous (it needs to re-acquire the lock), I need to come up with a 
reliable way to wait for it. If I don't, .release() will throw an exception 
because the lock is still unlocked.

The best method to do this that I've come up with is the five marked lines.
I keep thinking there must be a better way to do this (taking into account 
that I have no idea whether the 'await r' part of this is even necessary).


```
class StopMe(BaseException):
pass
class Foo:
async dev some_method(self):
self.uptodate = asyncio.Condition()
self.stopped = asyncio.Future()
…
await self.uptodate.acquire()
try:
while self.some_condition():
w = self.uptodate.wait()
await asyncio.wait([w,self.stopped], loop=self.conn._loop, 
return_when=asyncio.FIRST_COMPLETED)
with contextlib.suppress(StopMe):  # FIXME?
r = w.throw(StopMe())  # FIXME?
if r is not None:  # FIXME?
await r# FIXME?
await w# FIXME?
finally:
self.uptodate.release()
```
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Guido van Rossum]

Perhaps you can add a check for a simple boolean 'stop' flag to your
condition check, and when you want to stop the loop you set that flag and
then call notify() on the condition. Then you can follow the standard
condition variable protocol instead of all this nonsense. :-)

class Foo:
async def some_method(self):
self.uptodate = asyncio.Condition()
self.stopped = False
…
await self.uptodate.acquire()
try:
while (not self.stopped) and self.some_condition():
await self.uptodate.wait()
finally:
self.uptodate.release()
def stop_it(self):
self.stopped = True
self.uptodate.notify()

On Sat, Dec 19, 2015 at 7:43 AM, Matthias Urlichs 
wrote:

> The following code has a problem: the generator returned by .wait() has a
> finally: section. When self.stopped is set, it still needs to run. As it is
> asynchronous (it needs to re-acquire the lock), I need to come up with a
> reliable way to wait for it. If I don't, .release() will throw an exception
> because the lock is still unlocked.
>
> The best method to do this that I've come up with is the five marked lines.
> I keep thinking there must be a better way to do this (taking into account
> that I have no idea whether the 'await r' part of this is even necessary).
>
>
> ```
> class StopMe(BaseException):
> pass
> class Foo:
> async dev some_method(self):
> self.uptodate = asyncio.Condition()
> self.stopped = asyncio.Future()
> …
> await self.uptodate.acquire()
> try:
> while self.some_condition():
> w = self.uptodate.wait()
> await asyncio.wait([w,self.stopped], loop=self.conn._loop,
> return_when=asyncio.FIRST_COMPLETED)
> with contextlib.suppress(StopMe):  # FIXME?
> r = w.throw(StopMe())  # FIXME?
> if r is not None:  # FIXME?
> await r# FIXME?
> await w# FIXME?
> finally:
> self.uptodate.release()
> ```
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Matthias Urlichs]

On 19.12.2015 20:25, Guido van Rossum wrote:
> Perhaps you can add a check for a simple boolean 'stop' flag to your
> condition check, and when you want to stop the loop you set that flag
> and then call notify() on the condition. Then you can follow the
> standard condition variable protocol instead of all this nonsense. :-)
Your example does not work.

>def stop_it(self):
>self.stopped = True
>self.uptodate.notify()

self.uptodate needs to be locked before I can call .notify() on it.
Creating a new task just for that seems like overkill, and I'd have to
add a generation counter to prevent a race condition. Doable, but ugly.

However, this doesn't fix the generic problem; Condition.wait() was just
what bit me today.
When a non-async generator goes out of scope, its finally: blocks will
execute. An async procedure call whose refcount reaches zero without
completing simply goes away; finally: blocks are *not* called and there
is *no* warning.
I consider that to be a bug.

-- 
-- Matthias Urlichs

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Matthias Urlichs]

On 20.12.2015 01:26, Kevin Conway wrote:
> async def coroutine():
> try:
> await Awaitable()
> await Awaitable()
> finally:
> print('finally')
Try adding another "await Awaitable()" after the "finally:".

I have to take back my "doesn't print an error" comment, however;
there's another reference to the Condition.wait() generator (the task
asyncio.wait() creates to wrap the generator in), and the "Task was
destroyed but it is pending!" message got delayed sufficiently that I
missed it. (Dying test cases tend to spew many of these.)

Testcase:

import asyncio
import gc
cond = asyncio.Condition()
loop = asyncio.get_event_loop()

async def main():
async with cond:
# asyncio.wait() does this, if we don't
w = asyncio.ensure_future(cond.wait())
await asyncio.wait([w],timeout=1)
print(gc.get_referrers(w))
loop.run_until_complete(main())

Time to refactor my code to do the wait/timeout outside the "async with
cond".

-- 
-- Matthias Urlichs

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Guido van Rossum]

On Sat, Dec 19, 2015 at 1:40 PM, Matthias Urlichs 
wrote:

> On 19.12.2015 20:25, Guido van Rossum wrote:
> > Perhaps you can add a check for a simple boolean 'stop' flag to your
> > condition check, and when you want to stop the loop you set that flag
> > and then call notify() on the condition. Then you can follow the
> > standard condition variable protocol instead of all this nonsense. :-)
> Your example does not work.
>
> >def stop_it(self):
> >self.stopped = True
> >self.uptodate.notify()
>
> self.uptodate needs to be locked before I can call .notify() on it.
>

Fair enough.


> Creating a new task just for that seems like overkill, and I'd have to
> add a generation counter to prevent a race condition. Doable, but ugly.
>

I guess that's due to some application logic, but whatever. You don't
really seem to care about finding a solution for this problem anyways:


> However, this doesn't fix the generic problem; Condition.wait() was just
> what bit me today.
> When a non-async generator goes out of scope, its finally: blocks will
> execute. An async procedure call whose refcount reaches zero without
> completing simply goes away; finally: blocks are *not* called and there
> is *no* warning.
> I consider that to be a bug.
>

If that's so, can you demonstrate that without invoking all these other
things? Other traffic in this thread seems to indicate it may be not as
simple as that.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Gustavo Carneiro]

I tried to reproduce the problem you describe, but failed.  Here's my test
program (forgive the awful tab indentation, long story):

--
import asyncio

async def foo():
print("resource acquire")
try:
await asyncio.sleep(100)
finally:
print("resource release")


async def main():
task = asyncio.ensure_future(foo())
print("task created")
await asyncio.sleep(0)
print("about to cancel task")
task.cancel()
print("task cancelled, about to wait for it")
try:
await task
except asyncio.CancelledError:
pass
print("waited for cancelled task")


if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
---

I get this output:


10:54:28 ~/Documents$ python3.5 foo.py
task created
resource acquire
about to cancel task
task cancelled, about to wait for it
resource release
waited for cancelled task


Which seems to indicate that the finally clause is correctly executed when
the task is waited for, after being cancelled.

But maybe I completely misunderstood your problem...


On 19 December 2015 at 21:40, Matthias Urlichs  wrote:

> On 19.12.2015 20:25, Guido van Rossum wrote:
> > Perhaps you can add a check for a simple boolean 'stop' flag to your
> > condition check, and when you want to stop the loop you set that flag
> > and then call notify() on the condition. Then you can follow the
> > standard condition variable protocol instead of all this nonsense. :-)
> Your example does not work.
>
> >def stop_it(self):
> >self.stopped = True
> >self.uptodate.notify()
>
> self.uptodate needs to be locked before I can call .notify() on it.
> Creating a new task just for that seems like overkill, and I'd have to
> add a generation counter to prevent a race condition. Doable, but ugly.
>
> However, this doesn't fix the generic problem; Condition.wait() was just
> what bit me today.
> When a non-async generator goes out of scope, its finally: blocks will
> execute. An async procedure call whose refcount reaches zero without
> completing simply goes away; finally: blocks are *not* called and there
> is *no* warning.
> I consider that to be a bug.
>
> --
> -- Matthias Urlichs
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>



-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] asyncio: how to interrupt an async def w/ finally: ( e.g. Condition.wait() )

[Kevin Conway]

> An async procedure call whose refcount reaches zero without completing
simply goes away; finally: blocks are *not* called and there is *no*
warning.

I believe OP is looking at these two scenarios:

def generator():
try:
yield None
yield None
finally:
print('finally')


gen = generator()
gen.send(None)
del gen
# prints finally on GC


class Awaitable:
def __await__(self):
return self
def __next__(self):
return self


async def coroutine():
try:
await Awaitable()
await Awaitable()
finally:
print('finally')


coro = coroutine()
coro.send(None)
del coro
# prints finally on GC

I don't see any difference in the behaviour between the two. My guess is
that OP's code is not hitting a zero refcount.

On Sat, Dec 19, 2015 at 5:00 PM Gustavo Carneiro 
wrote:

> I tried to reproduce the problem you describe, but failed.  Here's my test
> program (forgive the awful tab indentation, long story):
>
> --
> import asyncio
>
> async def foo():
> print("resource acquire")
> try:
> await asyncio.sleep(100)
> finally:
> print("resource release")
>
>
> async def main():
> task = asyncio.ensure_future(foo())
> print("task created")
> await asyncio.sleep(0)
> print("about to cancel task")
> task.cancel()
> print("task cancelled, about to wait for it")
> try:
> await task
> except asyncio.CancelledError:
> pass
> print("waited for cancelled task")
>
>
> if __name__ == '__main__':
> loop = asyncio.get_event_loop()
> loop.run_until_complete(main())
> loop.close()
> ---
>
> I get this output:
>
> 
> 10:54:28 ~/Documents$ python3.5 foo.py
> task created
> resource acquire
> about to cancel task
> task cancelled, about to wait for it
> resource release
> waited for cancelled task
> 
>
> Which seems to indicate that the finally clause is correctly executed when
> the task is waited for, after being cancelled.
>
> But maybe I completely misunderstood your problem...
>
>
> On 19 December 2015 at 21:40, Matthias Urlichs 
> wrote:
>
>> On 19.12.2015 20:25, Guido van Rossum wrote:
>> > Perhaps you can add a check for a simple boolean 'stop' flag to your
>> > condition check, and when you want to stop the loop you set that flag
>> > and then call notify() on the condition. Then you can follow the
>> > standard condition variable protocol instead of all this nonsense. :-)
>> Your example does not work.
>>
>> >def stop_it(self):
>> >self.stopped = True
>> >self.uptodate.notify()
>>
>> self.uptodate needs to be locked before I can call .notify() on it.
>> Creating a new task just for that seems like overkill, and I'd have to
>> add a generation counter to prevent a race condition. Doable, but ugly.
>>
>> However, this doesn't fix the generic problem; Condition.wait() was just
>> what bit me today.
>> When a non-async generator goes out of scope, its finally: blocks will
>> execute. An async procedure call whose refcount reaches zero without
>> completing simply goes away; finally: blocks are *not* called and there
>> is *no* warning.
>> I consider that to be a bug.
>>
>> --
>> -- Matthias Urlichs
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>>
>
>
>
> --
> Gustavo J. A. M. Carneiro
> Gambit Research
> "The universe is always one step beyond logic." -- Frank Herbert
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/kevinjacobconway%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com