[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[Matthew]

Matthew  added the comment:

Let me preface this by declaring that I am very new to Python async so it is 
very possible that I am missing something seemingly obvious. That being said, 
I've been looking at various resources to try to understand the internals of 
asyncio and it hasn't led to any insights on this problem thus far.
-

This all sounds quite similar to an experience I am dealing with. I'm working 
with pub sub within aioredis which internally uses a StreamReader with a 
function equivalent to readexactly. This all started from debugging "Task was 
destroyed but it is pending!" to which attempted fixes led to multiple 
"RuntimeError: aclose(): asynchronous generator is already running" errors.

I did the same thing, adding try excepts everywhere in my code to understand 
what was happening and this led me to identifying that a regular async function 
would raise GeneratorExit during await. However, even if I suppress this, the 
caller awaiting on this function would also raise a GeneratorExit. Suppressing 
this exception at the top level leads to an unsuspecting (to me) error 
"coroutine ignored GeneratorExit".

I understand that GeneratorExit is raised in unfinished generators when garbage 
collected to handle cleanup. And I understand that async functions are 
essentially a generator in the sense that they yield when they await. So, if 
the entire coroutine were garbage collected this might trigger GeneratorExit in 
each nested coroutine. However, from all of my logging I am sure that prior to 
the GeneratorExit, nothing returns  upwards so there should still be valid 
references to every object.

I'll include some errors below, in case they may be of relevance:

=== Exception in await of inner async function ===
Traceback (most recent call last):
  File ".../site-packages/uvicorn/protocols/http/httptools_impl.py", line 165, 
in data_received
self.parser.feed_data(data)
  File "httptools/parser/parser.pyx", line 196, in 
httptools.parser.parser.HttpParser.feed_data
httptools.parser.errors.HttpParserUpgrade: 858

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../my_code.py", line 199, in wait_for_update
return await self.waiter.wait_for_value()
GeneratorExit

=== Exception when suppressing GeneratorExit on the top level ===
Exception ignored in: 
Traceback (most recent call last):
  File ".../site-packages/websockets/protocol.py", line 229, in __init__
self.reader = asyncio.StreamReader(limit=read_limit // 2, loop=loop)
RuntimeError: coroutine ignored GeneratorExit

--
nosy: +matthew

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[twisteroid ambassador]

twisteroid ambassador  added the comment:

Well this is unexpected, the same code running on Linux is throwing 
GeneratorExit-related mysterious exceptions as well. I'm not sure whether this 
is the same problem, but this one has a clearer traceback. I will attach the 
full error log, but the most pertinent part seems to be this:


During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/contextlib.py", line 662, in __aexit__
cb_suppress = await cb(*exc_details)
  File "/usr/lib/python3.8/contextlib.py", line 189, in __aexit__
await self.gen.athrow(typ, value, traceback)
  File "/opt/prettysocks/prettysocks.py", line 332, in closing_writer
await writer.wait_closed()
  File "/usr/lib/python3.8/asyncio/streams.py", line 376, in wait_closed
await self._protocol._get_close_waiter(self)
RuntimeError: cannot reuse already awaited coroutine


closing_writer() is an async context manager that calls close() and await 
wait_closed() on the given StreamWriter. So it looks like wait_closed() can 
occasionally reuse a coroutine?

--
Added file: https://bugs.python.org/file49534/error_log_on_linux_python38.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[twisteroid ambassador]

twisteroid ambassador  added the comment:

I have attached a script that should be able to reproduces this problem. It's 
not a minimal reproduction, but hopefully easy enough to trigger.

The script is a SOCKS5 proxy server listening on localhost:1080. In its current 
form it does not need any external dependencies. Run it on Windows 10 + Python 
3.9, set a browser to use the proxy server, and browse a little bit, it should 
soon start printing mysterious errors involving GeneratorExit.

--
Added file: https://bugs.python.org/file49532/prettysocks.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[twisteroid ambassador]

Change by twisteroid ambassador :


--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[twisteroid ambassador]

twisteroid ambassador  added the comment:

This problem still exists on Python 3.9 and latest Windows 10.

I tried to catch the GeneratorExit and turn it into a normal Exception, and 
things only got weirder from here. Often several lines later another await 
statement would raise another GeneratorExit, such as writer.write() or even 
asyncio.sleep(). Doesn't matter whether I catch the additional GeneratorExit or 
not, once code exits this coroutine a RuntimeError('coroutine ignored 
GeneratorExit') is raised. And it doesn't matter what I do with this 
RuntimeError, the outermost coroutine's Task always generates an 'asyncio Task 
was destroyed but it is pending!' error message.

Taking a step back from this specific problem. Does a "casual" user of asyncio 
need to worry about handling GeneratorExits? Can I assume that I should not see 
GeneratorExits in user code?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[Guilherme Salgado]

Change by Guilherme Salgado :


--
nosy: +salgado

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39116] StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop

[twisteroid ambassador]

New submission from twisteroid ambassador :

I have been getting these strange exception since Python 3.8 on my Windows 10 
machine. The external symptoms are many errors like "RuntimeError: aclose(): 
asynchronous generator is already running" and "Task was destroyed but it is 
pending!".

By adding try..except..logging around my code, I found that my StreamReaders 
would raise GeneratorExit on readexactly(). Digging deeper, it seems like the 
following line in StreamReader._wait_for_data():

await self._waiter

would raise a GeneratorExit.

There are only two other methods on StreamReader that actually does anything to 
_waiter, set_exception() and _wakeup_waiter(), but neither of these methods 
were called before GeneratorExit is raised. In fact, both these methods sets 
self._waiter to None, so normally after _wait_for_data() does "await 
self._waiter", self._waiter is None. However, after GeneratorExit is raised, I 
can see that self._waiter is not None. So it seems the GeneratorExit came from 
nowhere.

I have not been able to reproduce this behavior in other code. This is with 
Python 3.8.1 on latest Windows 10 1909, using ProactorEventLoop. I don't 
remember seeing this ever on Python 3.7.

--
components: asyncio
messages: 358774
nosy: asvetlov, twisteroid ambassador, yselivanov
priority: normal
severity: normal
status: open
title: StreamReader.readexactly() raises GeneratorExit on ProactorEventLoop
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com