Re: [Async-sig] async generator confusion or bug?

[Nathaniel Smith]

I actually thought that async generators already guarded against this using
their ag_running attribute. If I try running Dima's example with
async_generator, I get:

sending user-1
received user-1
sending user-2
sending user-0
Traceback (most recent call last):
[...]
ValueError: async generator already executing

The relevant code is here:
https://github.com/njsmith/async_generator/blob/e303e077c9dcb5880c0ce9930d560b282f8288ec/async_generator/impl.py#L273-L279

But I added this in the first place because I thought it was needed for
compatibility with native async generators :-)

-n

On Jun 26, 2017 6:54 PM, "Yury Selivanov"  wrote:

> (Posting here, rather than to the issue, because I think this actually
> needs more exposure).
>
> I looked at the code (genobject.c) and I think I know what's going on
> here.  Normally, when you work with an asynchronous generator (AG) you
> interact with it through "asend" or "athrow" *coroutines*.
>
> Each AG has its own private state, and when you await on "asend" coroutine
> you are changing that state.  The state changes on each "asend.send" or
> "asend.throw" call.  The normal relation between AGs and asends is 1 to 1.
>
>   AG - asend
>
> However, in your example you change that to 1 to many:
>
>  asend
> /
>   AG - asend
> \
>  asend
>
> Both 'ensure_future' and 'gather' will wrap each asend coroutine into an
> 'asyncio.Task'. And each Task will call "asend.send(None)" right in its
> '__init__', which changes the underlying *shared* AG instance completely
> out of order.
>
> I don't see how this can be fixed (or that it even needs to be fixed), so
> I propose to simply raise an exception if an AG has more than one asends
> changing it state *at the same time*.
>
> Thoughts?
>
> Yury
>
> > On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> >
> > Hi group,
> >
> > I'm trying to cross-use an sync generator across several async functions.
> > Is it allowed or a completely bad idea? (if so, why?)
> >
> > Here's MRE:
> >
> > import asyncio
> >
> >
> > async def generator():
> >while True:
> >x = yield
> >print("received", x)
> >await asyncio.sleep(0.1)
> >
> >
> > async def user(name, g):
> >print("sending", name)
> >await g.asend(name)
> >
> >
> > async def helper():
> >g = generator()
> >await g.asend(None)
> >
> >await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> >
> >
> > if __name__ == "__main__":
> >asyncio.get_event_loop().run_until_complete(helper())
> >
> >
> > And the output it produces when ran (py3.6.1):
> >
> > sending user-1
> > received user-1
> > sending user-2
> > sending user-0
> > received None
> > received None
> >
> >
> > Where are those None's coming from in the end?
> > Where did "user-0" and "user-1" data go?
> >
> > Is this a bug, or am I hopelessly confused?
> > Thanks!
> > ___
> > Async-sig mailing list
> > Async-sig@python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Guido van Rossum]

It does look complicated. The crux of the problem seems to be that helper()
is essentially

async def helper():
  g = generator()
  await g.asend(None)
  await asyncio.gather(user("user-0", g), user("user-1", g), user("user-2",
g))

which means that it is attempting to wait for three calls to user()
concurrently. This feels to me similar to three threads attempting to call
next() or send() on the same generator in parallel, which AFAIR is
explicitly guarded against somewhere. So detecting disallowing a similar
situation for async generators makes sense (and can be considered a bugfix).

On Mon, Jun 26, 2017 at 6:54 PM, Yury Selivanov 
wrote:

> (Posting here, rather than to the issue, because I think this actually
> needs more exposure).
>
> I looked at the code (genobject.c) and I think I know what's going on
> here.  Normally, when you work with an asynchronous generator (AG) you
> interact with it through "asend" or "athrow" *coroutines*.
>
> Each AG has its own private state, and when you await on "asend" coroutine
> you are changing that state.  The state changes on each "asend.send" or
> "asend.throw" call.  The normal relation between AGs and asends is 1 to 1.
>
>   AG - asend
>
> However, in your example you change that to 1 to many:
>
>  asend
> /
>   AG - asend
> \
>  asend
>
> Both 'ensure_future' and 'gather' will wrap each asend coroutine into an
> 'asyncio.Task'. And each Task will call "asend.send(None)" right in its
> '__init__', which changes the underlying *shared* AG instance completely
> out of order.
>
> I don't see how this can be fixed (or that it even needs to be fixed), so
> I propose to simply raise an exception if an AG has more than one asends
> changing it state *at the same time*.
>
> Thoughts?
>
> Yury
>
> > On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> >
> > Hi group,
> >
> > I'm trying to cross-use an sync generator across several async functions.
> > Is it allowed or a completely bad idea? (if so, why?)
> >
> > Here's MRE:
> >
> > import asyncio
> >
> >
> > async def generator():
> >while True:
> >x = yield
> >print("received", x)
> >await asyncio.sleep(0.1)
> >
> >
> > async def user(name, g):
> >print("sending", name)
> >await g.asend(name)
> >
> >
> > async def helper():
> >g = generator()
> >await g.asend(None)
> >
> >await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> >
> >
> > if __name__ == "__main__":
> >asyncio.get_event_loop().run_until_complete(helper())
> >
> >
> > And the output it produces when ran (py3.6.1):
> >
> > sending user-1
> > received user-1
> > sending user-2
> > sending user-0
> > received None
> > received None
> >
> >
> > Where are those None's coming from in the end?
> > Where did "user-0" and "user-1" data go?
> >
> > Is this a bug, or am I hopelessly confused?
> > Thanks!
> > ___
> > Async-sig mailing list
> > Async-sig@python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

(Posting here, rather than to the issue, because I think this actually needs 
more exposure).

I looked at the code (genobject.c) and I think I know what's going on here.  
Normally, when you work with an asynchronous generator (AG) you interact with 
it through "asend" or "athrow" *coroutines*.

Each AG has its own private state, and when you await on "asend" coroutine you 
are changing that state.  The state changes on each "asend.send" or 
"asend.throw" call.  The normal relation between AGs and asends is 1 to 1.

  AG - asend

However, in your example you change that to 1 to many:

 asend
/
  AG - asend
\
 asend

Both 'ensure_future' and 'gather' will wrap each asend coroutine into an 
'asyncio.Task'. And each Task will call "asend.send(None)" right in its 
'__init__', which changes the underlying *shared* AG instance completely out of 
order.

I don't see how this can be fixed (or that it even needs to be fixed), so I 
propose to simply raise an exception if an AG has more than one asends changing 
it state *at the same time*.

Thoughts?

Yury

> On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> 
> Hi group,
> 
> I'm trying to cross-use an sync generator across several async functions.
> Is it allowed or a completely bad idea? (if so, why?)
> 
> Here's MRE:
> 
> import asyncio
> 
> 
> async def generator():
>while True:
>x = yield
>print("received", x)
>await asyncio.sleep(0.1)
> 
> 
> async def user(name, g):
>print("sending", name)
>await g.asend(name)
> 
> 
> async def helper():
>g = generator()
>await g.asend(None)
> 
>await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> 
> 
> if __name__ == "__main__":
>asyncio.get_event_loop().run_until_complete(helper())
> 
> 
> And the output it produces when ran (py3.6.1):
> 
> sending user-1
> received user-1
> sending user-2
> sending user-0
> received None
> received None
> 
> 
> Where are those None's coming from in the end?
> Where did "user-0" and "user-1" data go?
> 
> Is this a bug, or am I hopelessly confused?
> Thanks!
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Dima Tisnek]

Thanks Yuri for quick reply.
http://bugs.python.org/issue30773 created :)


On 26 June 2017 at 19:55, Yury Selivanov  wrote:
>
>> On Jun 26, 2017, at 1:53 PM, Andrew Svetlov  wrote:
>>
>> IIRC gather collects coroutines in arbitrary order, maybe it's the source of 
>> misunderstanding?
>
> Yes, but that does not explain "receiving None" messages. Let's move this 
> discussion to the bug tracker.
>
> Yury
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

> On Jun 26, 2017, at 1:53 PM, Andrew Svetlov  wrote:
> 
> IIRC gather collects coroutines in arbitrary order, maybe it's the source of 
> misunderstanding?

Yes, but that does not explain "receiving None" messages. Let's move this 
discussion to the bug tracker.

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Andrew Svetlov]

IIRC gather collects coroutines in arbitrary order, maybe it's the source
of misunderstanding?

On Mon, Jun 26, 2017 at 8:48 PM Yury Selivanov  wrote:

> Hi Dima,
>
> > On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> >
> > Hi group,
> >
> > I'm trying to cross-use an sync generator across several async functions.
> > Is it allowed or a completely bad idea? (if so, why?)
>
> It is allowed, but leads to complex code.
>
> >
> > Here's MRE:
> >
> > import asyncio
> >
> >
> > async def generator():
> >while True:
> >x = yield
> >print("received", x)
> >await asyncio.sleep(0.1)
> >
> >
> > async def user(name, g):
> >print("sending", name)
> >await g.asend(name)
> >
> >
> > async def helper():
> >g = generator()
> >await g.asend(None)
> >
> >await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> >
> >
> > if __name__ == "__main__":
> >asyncio.get_event_loop().run_until_complete(helper())
> >
> >
> > And the output it produces when ran (py3.6.1):
> >
> > sending user-1
> > received user-1
> > sending user-2
> > sending user-0
> > received None
> > received None
> >
> >
> > Where are those None's coming from in the end?
> > Where did "user-0" and "user-1" data go?
>
>
> Interesting.  If I replace "gather" with three consecutive awaits of
> "asend", everything works as expected.  So there's some weird interaction
> of asend/gather, or maybe you did find a bug.  Need more time to
> investigate.
>
> Would you mind to open an issue on bugs.python?
>
> Thanks,
> Yury
>
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-- 
Thanks,
Andrew Svetlov
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

Hi Dima,

> On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> 
> Hi group,
> 
> I'm trying to cross-use an sync generator across several async functions.
> Is it allowed or a completely bad idea? (if so, why?)

It is allowed, but leads to complex code.

> 
> Here's MRE:
> 
> import asyncio
> 
> 
> async def generator():
>while True:
>x = yield
>print("received", x)
>await asyncio.sleep(0.1)
> 
> 
> async def user(name, g):
>print("sending", name)
>await g.asend(name)
> 
> 
> async def helper():
>g = generator()
>await g.asend(None)
> 
>await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> 
> 
> if __name__ == "__main__":
>asyncio.get_event_loop().run_until_complete(helper())
> 
> 
> And the output it produces when ran (py3.6.1):
> 
> sending user-1
> received user-1
> sending user-2
> sending user-0
> received None
> received None
> 
> 
> Where are those None's coming from in the end?
> Where did "user-0" and "user-1" data go?


Interesting.  If I replace "gather" with three consecutive awaits of "asend", 
everything works as expected.  So there's some weird interaction of 
asend/gather, or maybe you did find a bug.  Need more time to investigate.

Would you mind to open an issue on bugs.python?

Thanks,
Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/