Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 8:34 PM, Ian Kelly  wrote:
> To be pedantic, it should be more like:
>
> return type(aiter).__dict__['__anext__']()

And of course, if you don't find it there then to be proper you also
have to walk the MRO and check all of those class dicts as well.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 5:05 PM, Steve D'Aprano
 wrote:
> On Wed, 30 Nov 2016 07:51 pm, Ian Kelly wrote:
>
>> On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman  wrote:
>
>>> But I found it easy to write my own -
>>>
>>> async def anext(aiter):
>>>return await aiter.__anext__()
>>
>> Even simpler:
>>
>> def anext(aiter):
>> return aiter.__anext__()
>
>
> With very few exceptions, you shouldn't be calling dunder methods directly.
>
> Ideally, you should have some function or operator that performs the call
> for you, e.g. next(x) not x.__next__().

Yes, that's what the purpose of this function is.

> One important reason for that is that the dunder method may be only *part*
> of the protocol, e.g. the + operator can call either __add__ or __radd__;
> str(x) may end up calling either __repr__ or __str__.
>
> If such a function doesn't exist, then it's best to try to match Python's
> usual handling of dunders as closely as possible. That means, you shouldn't
> do the method lookup on the instance, but on the class instead:
>
> return type(aiter).__anext__()
>
> That matches the behaviour of the other dunder attributes, which normally
> bypass the instance attribute lookup.

To be pedantic, it should be more like:

return type(aiter).__dict__['__anext__']()

The difference between this and the above is that the above would try
the metaclass if it didn't find the method in the class dict, and it
might also call the metaclass's __getattr__ or __getattribute__.

What difference does it really make, though? That dunder methods are
looked up directly on the class is primarily an optimization. It's not
critical to the inner workings of the language.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Steve D'Aprano]

On Wed, 30 Nov 2016 07:51 pm, Ian Kelly wrote:

> On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman  wrote:

>> But I found it easy to write my own -
>>
>> async def anext(aiter):
>>return await aiter.__anext__()
> 
> Even simpler:
> 
> def anext(aiter):
> return aiter.__anext__()


With very few exceptions, you shouldn't be calling dunder methods directly.

Ideally, you should have some function or operator that performs the call
for you, e.g. next(x) not x.__next__().

One important reason for that is that the dunder method may be only *part*
of the protocol, e.g. the + operator can call either __add__ or __radd__;
str(x) may end up calling either __repr__ or __str__.

If such a function doesn't exist, then it's best to try to match Python's
usual handling of dunders as closely as possible. That means, you shouldn't
do the method lookup on the instance, but on the class instead:

return type(aiter).__anext__()

That matches the behaviour of the other dunder attributes, which normally
bypass the instance attribute lookup.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Marko Rauhamaa]

Ian Kelly :

> On Wed, Nov 30, 2016 at 2:28 AM, Marko Rauhamaa  wrote:
>> Each "await" in a program is a (quasi-)blocking state. In each state,
>> the program needs to be ready to process different input events.
>
> Well, that's why you can have multiple different coroutines awaiting
> at any given time.

At the very least, the programmer needs to actively consider
CancelledError for every "async" statement, even in the middle of an
"async for".


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 2:28 AM, Marko Rauhamaa  wrote:
> "Frank Millman" :
>
>> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k@elektro.pacujo.net...
>>>   I don't think bulk iteration in asynchronous programming is ever that
>>>   great of an idea. You want to be prepared for more than one possible
>>>   stimulus in any given state. IOW, a state machine matrix might be
>>>   sparse but it is never diagonal.
>>
>> [...]
>>
>> I use 'bulk iteration' a lot in my app. It is a client/server
>> multi-user business/accounting app.
>>
>> If a user wants to view the contents of a large table, or I want to
>> print statements for a lot of customers, I can request the data and
>> process it as it arrives, without blocking the other users.
>
> Each "await" in a program is a (quasi-)blocking state. In each state,
> the program needs to be ready to process different input events.

Well, that's why you can have multiple different coroutines awaiting
at any given time.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Marko Rauhamaa]

"Frank Millman" :

> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k@elektro.pacujo.net...
>>   I don't think bulk iteration in asynchronous programming is ever that
>>   great of an idea. You want to be prepared for more than one possible
>>   stimulus in any given state. IOW, a state machine matrix might be
>>   sparse but it is never diagonal.
>
> [...]
>
> I use 'bulk iteration' a lot in my app. It is a client/server
> multi-user business/accounting app.
>
> If a user wants to view the contents of a large table, or I want to
> print statements for a lot of customers, I can request the data and
> process it as it arrives, without blocking the other users.

Each "await" in a program is a (quasi-)blocking state. In each state,
the program needs to be ready to process different input events.

I suppose there are cases where a coroutine can pursue an objective
single-mindedly (in which case the only secondary input event is the
cancellation of the operation). I have done quite a bit of asynchronous
programming but never run into that scenario yet.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Peter Otten]

Steve D'Aprano wrote:

> On Wed, 30 Nov 2016 07:07 am, Marko Rauhamaa wrote:
> 
>> Terry Reedy :
>> 
>>> On 11/29/2016 9:25 AM, Frank Millman wrote:
>>>
 Is there any technical reason for this, or is it just that no-one has
 got around to writing an asynchronous version yet?
>>>
>>> Google's first hit for 'aenumerate' is
>>>
> https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html
>> 
>> Ok, so how about:
>> 
>>aall(aiterable)
>>aany(aiterable)
>>class abytearray(aiterable[, encoding[, errors]])
> [...]
> 
> 
> What about them? What's your question?

Well, my questions as someone who hasn't touched the async stuff so far 
would be:

Is there a viable approach to provide

(a) support for async without duplicating the stdlib
(b) a reasonably elegant way to access the async versions

I hope we can agree that prepending an "a" to the name should be the last 
resort.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Frank Millman]

"Ian Kelly"  wrote in message 
news:CALwzid=hrijtv4p1_6frkqub25-o1i8ouquxozd+aujgl7+...@mail.gmail.com...


On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman  wrote:
>
> async def anext(aiter):
>return await aiter.__anext__()

Even simpler:

def anext(aiter):
return aiter.__anext__()

As a general rule, if the only await in a coroutine is immediately
prior to the return, then it doesn't need to be a coroutine. Just
return the thing it's awaiting so that the caller can be rid of the
middle man and await it directly.


Fascinating!

Now I will have to go through all my code looking for similar occurrences. I 
am sure I will find some!


Frank


--
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman  wrote:
> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k@elektro.pacujo.net...
>>
>>
>> One of the more useful ones might be:
>>
>> o = await anext(ait)
>>
>
> Definitely!
>
> But I found it easy to write my own -
>
> async def anext(aiter):
>return await aiter.__anext__()

Even simpler:

def anext(aiter):
return aiter.__anext__()

As a general rule, if the only await in a coroutine is immediately
prior to the return, then it doesn't need to be a coroutine. Just
return the thing it's awaiting so that the caller can be rid of the
middle man and await it directly.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 1:20 AM, Chris Angelico  wrote:
> Hmm. The thing is, comprehensions and generators are implemented with
> their own nested functions. So I would expect that their use of async
> is independent of the function they're in. But maybe we have a bug
> here?
>
 async def spam():
> ... def ham():
> ... async for i in x:
> ... pass
> ...
 def ham():
> ... async for i in x:
>   File "", line 2
> async for i in x:
> ^
> SyntaxError: invalid syntax
 def ham():
> ... async def spam():
> ... async for i in x:
> ... pass
> ...

>
> Clearly the second one is correct to throw SyntaxError, and the third
> is correctly acceptable. But the first one, ISTM, should be an error
> too.

Yeah, that looks like a bug to me. Note that 'await' results in a
clear error in the first case:

>>> async def ham():
... def spam():
... await foo
...
  File "", line 3
SyntaxError: 'await' outside async function


>> Yeah, that's what I would expect. (x async for x in foo) is
>> essentially a no-op, just like its synchronous equivalent; it takes an
>> asynchronous iterator and produces an equivalent asynchronous
>> iterator. Meanwhile, list() can't consume an async iterator because
>> the list constructor isn't a coroutine. I don't think it's generally
>> possible to "synchronify" an async iterator other than to materialize
>> it. E.g.:
>>
>> def alist(aiterable):
>> result = []
>> async for value in aiterable:
>> result.append(value)
>> return result
>>
>> And I find it a little disturbing that I actually can't see a better
>> way to build a list from an async iterator than that.
>
> Oh. Oops. That materialization was exactly what I intended to happen
> with the comprehension. Problem: Your version doesn't work either,
> although I think it probably _does_ work if you declare that as "async
> def alist".

Yes, that's what I meant.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Frank Millman]

"Marko Rauhamaa"  wrote in message news:87d1hd4d5k@elektro.pacujo.net...


One of the more useful ones might be:

o = await anext(ait)



Definitely!

But I found it easy to write my own -

async def anext(aiter):
   return await aiter.__anext__()

[...]


  I don't think bulk iteration in asynchronous programming is ever that
  great of an idea. You want to be prepared for more than one possible
  stimulus in any given state. IOW, a state machine matrix might be
  sparse but it is never diagonal.


I am not familiar with your terminology here, so my comment may be way 
off-track.


I use 'bulk iteration' a lot in my app. It is a client/server multi-user 
business/accounting app.


If a user wants to view the contents of a large table, or I want to print 
statements for a lot of customers,  I can request the data and process it as 
it arrives, without blocking the other users.


I find that very powerful.

Frank


--
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Wed, Nov 30, 2016 at 12:53 AM, Marko Rauhamaa  wrote:
> I have a couple of points to make with my question:
>
>  * We are seeing the reduplication of a large subset of Python's
>facilities. I really wonder if the coroutine fad is worth the price.

I don't think there's any technical reason why functions like zip
can't support both synchronous and asynchronous iterators. For
example:

_zip = __builtins__.zip
_azip = (some asynchronous zip implementation)

class zip:
def __init__(self, *args):
self._args = args
def __iter__(self):
return _zip(*self._args)
def __aiter__(self):
return _azip(*self._args)

Now I can do "for x, y in zip(a, b)" or "async for x, y in
zip(async_a, async_b)" and either will work as expected. Of course, if
you use the wrong construct then you'll probably get an error since
the underlying iterators don't support the protocol. But it keeps the
builtin namespace simple and clean.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Chris Angelico]

On Wed, Nov 30, 2016 at 7:10 PM, Ian Kelly  wrote:
> On Tue, Nov 29, 2016 at 8:22 PM, Chris Angelico  wrote:
>> Interestingly, I can't do that in a list comp:
>>
> [x async for x in aiterable]
>>   File "", line 1
>> [x async for x in aiterable]
>>^
>> SyntaxError: invalid syntax
>>
>> Not sure why.
>
> Because you tried to use an async comprehension outside of a coroutine.
>
> py> [x async for x in y]
>   File "", line 1
> [x async for x in y]
>^
> SyntaxError: invalid syntax
> py> async def foo():
> ... [x async for x in y]
> ...
>
> The same is true for async generator expressions. The documentation is
> clear that this is illegal for the async for statement:
>
> https://docs.python.org/3.6/reference/compound_stmts.html#the-async-for-statement

Hmm. The thing is, comprehensions and generators are implemented with
their own nested functions. So I would expect that their use of async
is independent of the function they're in. But maybe we have a bug
here?

>>> async def spam():
... def ham():
... async for i in x:
... pass
...
>>> def ham():
... async for i in x:
  File "", line 2
async for i in x:
^
SyntaxError: invalid syntax
>>> def ham():
... async def spam():
... async for i in x:
... pass
...
>>>

Clearly the second one is correct to throw SyntaxError, and the third
is correctly acceptable. But the first one, ISTM, should be an error
too.

> Yeah, that's what I would expect. (x async for x in foo) is
> essentially a no-op, just like its synchronous equivalent; it takes an
> asynchronous iterator and produces an equivalent asynchronous
> iterator. Meanwhile, list() can't consume an async iterator because
> the list constructor isn't a coroutine. I don't think it's generally
> possible to "synchronify" an async iterator other than to materialize
> it. E.g.:
>
> def alist(aiterable):
> result = []
> async for value in aiterable:
> result.append(value)
> return result
>
> And I find it a little disturbing that I actually can't see a better
> way to build a list from an async iterator than that.

Oh. Oops. That materialization was exactly what I intended to happen
with the comprehension. Problem: Your version doesn't work either,
although I think it probably _does_ work if you declare that as "async
def alist".

Shows you just how well I understand Python's asyncness, doesn't it?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Tue, Nov 29, 2016 at 8:22 PM, Chris Angelico  wrote:
> Interestingly, I can't do that in a list comp:
>
 [x async for x in aiterable]
>   File "", line 1
> [x async for x in aiterable]
>^
> SyntaxError: invalid syntax
>
> Not sure why.

Because you tried to use an async comprehension outside of a coroutine.

py> [x async for x in y]
  File "", line 1
[x async for x in y]
   ^
SyntaxError: invalid syntax
py> async def foo():
... [x async for x in y]
...

The same is true for async generator expressions. The documentation is
clear that this is illegal for the async for statement:

https://docs.python.org/3.6/reference/compound_stmts.html#the-async-for-statement

I don't see anything about async comprehensions or async generators
outside of the "what's new" section, but it stands to reason that the
same would apply.

On Tue, Nov 29, 2016 at 11:06 PM, Frank Millman  wrote:
 async def main():
>
> ...   print(list(x async for x in gen(5)))
>
> ...

 loop.run_until_complete(main())
>
> Traceback (most recent call last):
>  File "", line 1, in 
>  File
> "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py",
> line 466, in run_until_complete
>return future.result()
> TypeError: 'async_generator' object is not iterable

Yeah, that's what I would expect. (x async for x in foo) is
essentially a no-op, just like its synchronous equivalent; it takes an
asynchronous iterator and produces an equivalent asynchronous
iterator. Meanwhile, list() can't consume an async iterator because
the list constructor isn't a coroutine. I don't think it's generally
possible to "synchronify" an async iterator other than to materialize
it. E.g.:

def alist(aiterable):
result = []
async for value in aiterable:
result.append(value)
return result

And I find it a little disturbing that I actually can't see a better
way to build a list from an async iterator than that.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Marko Rauhamaa]

Chris Angelico :

> On Wed, Nov 30, 2016 at 7:07 AM, Marko Rauhamaa  wrote:
> Any of these that depend on pumping the entire iterable can simply
> synchronify [1] the iterable:

One of the more useful ones might be:

o = await anext(ait)

> list(x async for x in aiterable)
>
> Interestingly, I can't do that in a list comp:

I have a couple of points to make with my question:

 * We are seeing the reduplication of a large subset of Python's
   facilities. I really wonder if the coroutine fad is worth the price.

 * I don't think bulk iteration in asynchronous programming is ever that
   great of an idea. You want to be prepared for more than one possible
   stimulus in any given state. IOW, a state machine matrix might be
   sparse but it is never diagonal.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Frank Millman]

"Frank Millman"  wrote in message news:o1k355$da5$1...@blaine.gmane.org...


Hi all

Python 3.6 has introduced Asynchronous Generators, which work very well.


[...]


However, it does not allow you to enumerate over the generator output -


[...]


Is there any technical reason for this, or is it just that no-one has got 
around to writing an asynchronous version yet?


Thanks for the replies.

@Ian
Thanks for your explanation and example.
The example is perfect for my simple requirement.

@Terry
I should have googled for aenumerate - didn't think of it.

However, this recipe is based on Python 3.5 and earlier.
Asynchronous Generators, introduced in 3.6, make it much easier.
See PEP 525 for the details -
   https://www.python.org/dev/peps/pep-0525/

@Chris
I agree, there is a big difference between functions which consume the 
entire iterable before returning the result, and those which behave 
asynchronously and return the value on-the-fly.


I happened upon enumerate. You mentioned zip and map, which are also likely 
to be useful in the right circumstances.


I did not quite follow your example, as I get the opposite result -

Python 3.6.0b4 (default, Nov 22 2016, 05:30:12) [MSC v.1900 64 bit (AMD64)] 
on win32

Type "help", "copyright", "credits" or "license" for more information.

import asyncio
loop = asyncio.get_event_loop()



async def gen(n):

...   for i in range(n):
... yield i
...

async def main():

...   print([x async for x in gen(5)])
...

loop.run_until_complete(main())

[0, 1, 2, 3, 4]


async def main():

...   print(list(x async for x in gen(5)))
...

loop.run_until_complete(main())

Traceback (most recent call last):
 File "", line 1, in 
 File 
"C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", 
line 466, in run_until_complete

   return future.result()
TypeError: 'async_generator' object is not iterable




Frank


--
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Chris Angelico]

On Wed, Nov 30, 2016 at 7:07 AM, Marko Rauhamaa  wrote:
> Ok, so how about:
>
>aall(aiterable)
>aany(aiterable)
>class abytearray(aiterable[, encoding[, errors]])
>class adict(aiterable, **kwarg)
>class afilter(coro, aiterable)
>class afrozenset(aiterable)
>aiter(object[, sentinel])
>class alist(aiterable)
>amap(coro, aiterable, ...)
>amax(aiterable, *[, key, default])
>amin(aiterable, *[, key, default])
>anext(aiterator[, default])
>class aset(aiterable)
>asorted(aiterable[, key][, reverse])
>asum(aiterable[, start])
>atuple(aiterable)
>azip(*aiterables)
>
> to name a few...
>
> How about awaitable comprehensions?

Any of these that depend on pumping the entire iterable can simply
synchronify [1] the iterable:

list(x async for x in aiterable)

Interestingly, I can't do that in a list comp:

>>> [x async for x in aiterable]
  File "", line 1
[x async for x in aiterable]
   ^
SyntaxError: invalid syntax

Not sure why. Anyhow. That removes the need for alist, atuple, amax,
etc. All you would need are the ones that process an iterable and
yield values individually, like azip and amap. Those can be provided
as recipes, third-party modules, or stdlib modules, until they prove
their worth as potential builtins.

ChrisA

[1] is that even a word?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Steve D'Aprano]

On Wed, 30 Nov 2016 07:07 am, Marko Rauhamaa wrote:

> Terry Reedy :
> 
>> On 11/29/2016 9:25 AM, Frank Millman wrote:
>>
>>> Is there any technical reason for this, or is it just that no-one has
>>> got around to writing an asynchronous version yet?
>>
>> Google's first hit for 'aenumerate' is
>>
https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html
> 
> Ok, so how about:
> 
>aall(aiterable)
>aany(aiterable)
>class abytearray(aiterable[, encoding[, errors]])
[...]


What about them? What's your question?





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Marko Rauhamaa]

Terry Reedy :

> On 11/29/2016 9:25 AM, Frank Millman wrote:
>
>> Is there any technical reason for this, or is it just that no-one has
>> got around to writing an asynchronous version yet?
>
> Google's first hit for 'aenumerate' is
> https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html

Ok, so how about:

   aall(aiterable)
   aany(aiterable)
   class abytearray(aiterable[, encoding[, errors]])
   class adict(aiterable, **kwarg)
   class afilter(coro, aiterable)
   class afrozenset(aiterable)
   aiter(object[, sentinel])
   class alist(aiterable)
   amap(coro, aiterable, ...)
   amax(aiterable, *[, key, default])
   amin(aiterable, *[, key, default])
   anext(aiterator[, default])
   class aset(aiterable)
   asorted(aiterable[, key][, reverse])
   asum(aiterable[, start])
   atuple(aiterable)
   azip(*aiterables)

to name a few...

How about awaitable comprehensions?


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Terry Reedy]

On 11/29/2016 9:25 AM, Frank Millman wrote:


Is there any technical reason for this, or is it just that no-one has
got around to writing an asynchronous version yet?


Google's first hit for 'aenumerate' is
https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html

Note that updated 3.5.2+ code is in response to my comment.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: async enumeration - possible?

[Ian Kelly]

On Tue, Nov 29, 2016 at 7:25 AM, Frank Millman  wrote:
> However, it does not allow you to enumerate over the generator output -
>
 async def main():
>
> ...   c = counter(5)
> ...   async for j, k in enumerate(c):
> ... print(j, k)
> ...   print('done')
> ...

 loop.run_until_complete(main())
>
> Traceback (most recent call last):
>  File "", line 1, in 
>  File
> "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py",
> line 466, in run_until_complete
>return future.result()
> TypeError: 'async_generator' object is not iterable


>
> Is there any technical reason for this, or is it just that no-one has got
> around to writing an asynchronous version yet?

No one has written an async enumerate afaik. You may be interested in
the aitertools package in PyPI:

https://pypi.python.org/pypi/aitertools/0.1.0

It also doesn't have an async enumerate. It does have an aiter
function that can wrap an ordinary iterable into an async iterable, so
one way to write that would be aiter(enumerate(c)), with the caveat
that the wrapped iterator is still running synchronously.

Otherwise, you can write one yourself. This doesn't support all the
features of enumerate, but it serves as demonstration:

from itertools import count

async def aenumerate(aiterable):
counter = count()
async for x in aiterable:
 yield next(counter), x
-- 
https://mail.python.org/mailman/listinfo/python-list