[issue26221] asynco run_in_executor swallows StopIteration

[Ian Kelly]

Ian Kelly added the comment:

The place I'd expect to find it is in 
https://docs.python.org/3/library/asyncio-task.html#coroutines, in the list of 
"things a coroutine can do". The first two bullets in the list say that any 
exceptions raised will be propagated. Maybe there should be a note after the 
bullet list to the effect that "StopIteration carries special meaning to 
coroutines and will not be propagated if raised by an awaited coroutine or 
future."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Guido van Rossum]

Guido van Rossum added the comment:

Can you suggest a sentence to insert into the docs and a place where
to insert it? (As you can imagine I'm pretty blind for such issues
myself.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Ian Kelly]

New submission from Ian Kelly:

I was playing around with this class for adapting regular iterators to async 
iterators using BaseEventLoop.run_in_executor:


import asyncio

class AsyncIteratorWrapper:

def __init__(self, iterable, loop=None, executor=None):
self._iterator = iter(iterable)
self._loop = loop or asyncio.get_event_loop()
self._executor = executor

async def __aiter__(self):
return self

async def __anext__(self):
try:
return await self._loop.run_in_executor(
self._executor, next, self._iterator)
except StopIteration:
raise StopAsyncIteration


Unfortunately this fails because when next raises StopIteration, 
run_in_executor swallows the exception and just returns None back to the 
coroutine, resulting in an infinite iterator of Nones.

--
components: asyncio
messages: 259036
nosy: gvanrossum, haypo, ikelly, yselivanov
priority: normal
severity: normal
status: open
title: asynco run_in_executor swallows StopIteration
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[STINNER Victor]

Changes by STINNER Victor :


--
nosy:  -haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Guido van Rossum]

Guido van Rossum added the comment:

What are you trying to do here? Can you post a simple example of an iterator 
that you would like to use with this? Without that it just raises my hackles -- 
it seems totally wrong to run an iterator in another thread. (Or is the 
iterator really a coroutine/future?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Ian Kelly]

Ian Kelly added the comment:

The idea is that the wrapped iterator is something potentially blocking, like a 
database cursor that doesn't natively support asyncio. Usage would be something 
like this:


async def get_data():
cursor.execute('select * from stuff')
async for row in AsyncIteratorWrapper(cursor):
process(row)


Investigating this further, I think the problem is actually in await, not 
run_in_executor:

>>> async def test():
... fut = asyncio.Future()
... fut.set_exception(StopIteration())
... print(await fut)
... 
>>> loop.run_until_complete(test())
None

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Guido van Rossum]

Guido van Rossum added the comment:

StopIteration has a special meaning. Don't use set_exception() with it.

You probably need a more roundabout way to do this.

Instead of submitting each __next__() call to the executor separately, you 
should submit something to the executor that pulls the items from the iterator 
and sticks them into a queue; then on the asyncio side you pull them out of the 
queue.

You can use an asyncio.Queue as the queue, and use loop.call_soon_threadsafe() 
to put things into that queue from the tread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26221] asynco run_in_executor swallows StopIteration

[Ian Kelly]

Ian Kelly added the comment:

Fair enough. I think there should be some documentation though to the effect 
that coroutines aren't robust to passing StopIteration across coroutine 
boundaries. It's particularly surprising with PEP-492 coroutines, since those 
aren't even iterators and intuitively should ignore StopIteration like normal 
functions do.

As it happens, this variation (moving the try-except into the executor thread) 
does turn out to work but is probably best avoided for the same reason. I don't 
think it's obviously bad code though:


class AsyncIteratorWrapper:

def __init__(self, iterable, loop=None, executor=None):
self._iterator = iter(iterable)
self._loop = loop or asyncio.get_event_loop()
self._executor = executor

async def __aiter__(self):
return self

async def __anext__(self):
def _next(iterator):
try:
return next(iterator)
except StopIteration:
raise StopAsyncIteration
return await self._loop.run_in_executor(
self._executor, _next, self._iterator)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com