[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[robbiecares]

Change by robbiecares :


--
nosy: +robbiecares

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 40e2444c364eede59f193979df5a02ed958f56e9 by Miss Islington (bot) 
in branch '3.8':
bpo-39010: Improve test shutdown (GH-22066) (#22083)
https://github.com/python/cpython/commit/40e2444c364eede59f193979df5a02ed958f56e9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 8d68e59f11267ded765d4af6d71a49784a12fad5 by Miss Islington (bot) 
in branch '3.9':
bpo-39010: Improve test shutdown (GH-22066) (#22082)
https://github.com/python/cpython/commit/8d68e59f11267ded765d4af6d71a49784a12fad5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[miss-islington]

Change by miss-islington :


--
pull_requests: +21169
pull_request: https://github.com/python/cpython/pull/22083

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[miss-islington]

Change by miss-islington :


--
pull_requests: +21168
pull_request: https://github.com/python/cpython/pull/22082

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset a986b061a3cd4661eb9f857e2936291f1847bd15 by Miss Islington (bot) 
in branch '3.8':
bpo-39010: Fix errors logged on proactor loop restart (GH-22017) (#22035)
https://github.com/python/cpython/commit/a986b061a3cd4661eb9f857e2936291f1847bd15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 49571c0b0e57e20d85727c738d9a0fe342dd2938 by Miss Islington (bot) 
in branch '3.9':
bpo-39010: Fix errors logged on proactor loop restart (GH-22017) (#22034)
https://github.com/python/cpython/commit/49571c0b0e57e20d85727c738d9a0fe342dd2938


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset be435ae2b064dc64f04475bec632862e1dbf605f by Ben Darnell in branch 
'master':
bpo-39010: Improve test shutdown (#22066)
https://github.com/python/cpython/commit/be435ae2b064dc64f04475bec632862e1dbf605f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Ben Darnell  added the comment:

I've fixed the test and added some commentary about the different levels of 
clean shutdown we must do to quiet all the warnings: 
https://github.com/python/cpython/pull/22066

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Change by Ben Darnell :


--
pull_requests: +21156
pull_request: https://github.com/python/cpython/pull/22066

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[David Bolen]

David Bolen  added the comment:

I'm guessing the warning appears odd as we're seeing a thread shutdown data 
race.  The message is produced by threading_cleanup in 
support/threading_helper.py, and it appears that between the first and second 
lines one of the initially dangling threads goes away, so the only one left to 
be enumerated is the main thread, at which point the function is simply listing 
all threads currently alive.  But by that point it's just the main thread 
remaining.

I notice in the test that you have a comment about needing to wait for f to 
complete or you get a warning about thread not shutting down cleanly.  Was that 
a similar warning?  The run_until_complete(f) line seems to have no effect on 
the buildbot.

If I added a small sleep at the end of the test the warnings go away on the 
buildbot.  The buildbot is a fairly fast machine, so perhaps the test just 
needs to wait somehow for the event loop to fully shut down or something.

The most direct cause of the warnings seems to be the self.loop.close() call - 
if I just comment that out the test runs warning-free without any extra delay 
needed.

I don't know much about asyncio tests, but it would appear the close call in 
the test defeats some of the logic in the close_loop teardown code that runs 
under TestCase (in utils.py), which under Windows is just a call to 
run_until_complete(loop.shutdown_default_executor()).  If I add that same call 
to the test prior to the close it also passes cleanly.

So if closing the loop in the test itself is crucial, I'd suggest including the 
extra run_until_complete call.  If closing isn't crucial to the test, simply 
removing it seems to address the issue.  I'm not sure if its removal then has 
any implications for the extra run_until_complete(f) call in the test, as I 
can't see any impact from that on the buildbot.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Ben Darnell  added the comment:

I can confirm that those warnings appear to be coming from the test I added 
here. I'm not sure how to interpret them, though - what does it mean for the 
main thread to be dangling?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[David Bolen]

David Bolen  added the comment:

I've been seeing failures on the Win10 buildbot 3.x branch that seem to 
correlate with the timing of this change - could there be some further work 
needed on Windows?  Or, if it's a test-only artifact and the warnings are 
innocuous, something to ignore the output during the tests?

Specifically, the following warnings occur during test_asyncio:
   Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, 
dangling: 2)
   Warning -- Dangling thread: <_MainThread(MainThread, started 5220)>

See https://buildbot.python.org/all/#/builders/146/builds/23 for the first 
failing build.

--
nosy: +db3l

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Steve Dower]

Steve Dower  added the comment:

Thanks, Ben!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset ea5a6363c3f8cc90b7c0cc573922b10f296073b6 by Ben Darnell in branch 
'master':
bpo-39010: Fix errors logged on proactor loop restart (#22017)
https://github.com/python/cpython/commit/ea5a6363c3f8cc90b7c0cc573922b10f296073b6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[miss-islington]

Change by miss-islington :


--
pull_requests: +21133
pull_request: https://github.com/python/cpython/pull/22035

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 11.0 -> 12.0
pull_requests: +21132
pull_request: https://github.com/python/cpython/pull/22034

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Ben Darnell  added the comment:

I've posted a pull request with a test and fix: 
https://github.com/python/cpython/pull/22017. It's a more targeted fix than 
cmeyer's PR (which I didn't even notice until now due to unfamiliarity with the 
BPO UI)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Change by Ben Darnell :


--
pull_requests: +21117
pull_request: https://github.com/python/cpython/pull/22017

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Steve Dower]

Steve Dower  added the comment:

Any input from the asyncio experts?

I don't have an issue with handling the exception in this case, and hopefully 
when someone is up to the task of dealing with the range of edge cases 
throughout this loop implementation, hopefully they can get the ordering of 
waits right.

--
nosy: +steve.dower
versions: +Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Chris Meyer]

Change by Chris Meyer :


--
keywords: +patch
pull_requests: +19769
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/20525

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Burak Yiğit Kaya]

Change by Burak Yiğit Kaya :


--
nosy: +Burak Yiğit Kaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Chris Meyer]

Chris Meyer  added the comment:

Here is another way to reproduce this (or an extremely similar) error without a 
loop. Since may be a race condition, I'm not sure this works 100% of the time 
on all machines - but it did on several machines I tried.

```
import asyncio

loop = asyncio.get_event_loop()

def func():
pass

f = loop.run_in_executor(None, func)
loop.stop()
loop.run_forever()
loop.stop()
loop.run_forever()
loop.stop()
loop.run_forever()
```

```
Error on reading from the event loop self pipe
loop: 
Traceback (most recent call last):
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 453, in 
finish_recv
return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a 
thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\envs\py38\lib\asyncio\proactor_events.py", line 768, in 
_loop_self_reading
f.result()  # may raise
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 808, in 
_poll
value = callback(transferred, key, ov)
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 457, in 
finish_recv
raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because 
of either a thread exit or an application request
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Segev Finer]

Change by Segev Finer :


--
nosy: +Segev Finer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Chris Meyer]

Change by Chris Meyer :


--
nosy: +cmeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Peter Lovett]

Change by Peter Lovett :


--
nosy: +PeterL777

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Rustam S.]

Rustam S.  added the comment:

Please take a look at this as well:
(ipython #12049 'Unhandled exception in event loop' (WinError 995))
https://github.com/ipython/ipython/issues/12049#issuecomment-586544339 and below

--
nosy: +Rustam S.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Ben Darnell]

Ben Darnell  added the comment:

I just spent some time digging into this. Each call to `run_forever` starts a 
call to `_loop_self_reading`, then attempts to cancel it before returning:

https://github.com/python/cpython/blob/1ed61617a4a6632905ad6a0b440cd2cafb8b6414/Lib/asyncio/windows_events.py#L312-L325

The comment at line 321 is not entirely accurate: the future will not resolve 
in the future, but it may have *already* resolved, and added its callback to 
the call_soon queue. This callback will run if the event loop is restarted 
again. Since `_loop_self_reading` calls itself, this results in two copies of 
the "loop" running concurrently and stepping on each other's 
`_self_reading_futures`. 

This appears to be fairly harmless except for noise in the logs when only one 
of the loops is stopped cleanly.

I believe the simplest fix is for `_loop_self_reading` to compare its argument 
to `self._self_reading_future` to determine if it is the "current" loop and if 
not, don't reschedule anything.

--
nosy: +Ben.Darnell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Caleb Hattingh]

Change by Caleb Hattingh :


--
nosy: +cjrh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Jonathan Slenders]

Jonathan Slenders  added the comment:

Even simpler, the following code will crash after so many iterations:


```
import asyncio
loop = asyncio.get_event_loop()

while True:
loop.call_soon_threadsafe(loop.stop)
loop.run_forever()
```

Adding a little sleep of 0.01s after `run_forever()` prevents the issue.

So, to me it looks like the cancellation of the `_OverlappedFuture` that wraps 
around the `_recv()` call from the self-pipe did not complete before we start 
`_recv()` again in the next `run_forever()` call. No idea if that makes sense...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Jonathan Slenders]

Jonathan Slenders  added the comment:

It looks like the following code will reproduce the issue:

```
import asyncio
import threading

loop = asyncio.get_event_loop()

while True:
def test():
loop.call_soon_threadsafe(loop.stop)

threading.Thread(target=test).start()
loop.run_forever()

```

Leave it running on Windows, in Python 3.8 for a few seconds, then it starts 
spawning `ConnectionResetError`s.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Jonathan Slenders]

Jonathan Slenders  added the comment:

Thanks Victor for the reply.

It looks like it's the self-socket in the BaseProactorEventLoop that gets 
closed. It's exactly this FD for which the exception is raised.

We don't close the event loop anywhere. I also don't see `_close_self_pipe` 
being called anywhere.

Debug logs don't provide any help. I'm looking into a reproducer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[STINNER Victor]

STINNER Victor  added the comment:

ConnectionResetError means that the pipe socket is closed. Was the event loop 
closed? Can you provide a reproducer? Can you try to get debug logs to see 
what's going on?
https://docs.python.org/dev/library/asyncio-dev.html#debug-mode

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Jonathan Slenders]

Jonathan Slenders  added the comment:

Suppressing `ConnectionResetError` in 
`BaseProactorEventLoop._loop_self_reading`, like we do with `CancelledError` 
seems to fix it.

Although I'm not sure what it causing the error, and whether we need to handle 
it somehow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39010] ProactorEventLoop raises unhandled ConnectionResetError

[Jonathan Slenders]

New submission from Jonathan Slenders :

We have a snippet of code that runs perfectly fine using the 
`SelectorEventLoop`, but crashes *sometimes* using the `ProactorEventLoop`.

The traceback is the following. The exception cannot be caught within the 
asyncio application itself (e.g., it is not attached to any Future or 
propagated in a coroutine). It probably propagates in `run_until_complete()`.

  File "C:\Python38\lib\asyncio\proactor_events.py", line 768, in 
_loop_self_reading
f.result()  # may raise
  File "C:\Python38\lib\asyncio\windows_events.py", line 808, in _poll
value = callback(transferred, key, ov)
  File "C:\Python38\lib\asyncio\windows_events.py", line 457, in finish_recv
raise ConnectionResetError(*exc.args)

I can see that in `IocpProactor._poll`, `OSError` is caught and attached to the 
future, but not `ConnectionResetError`. I would expect that 
`ConnectionResetError` too will be attached to the future.

In order to reproduce, run the following snippet on Python 3.8:

from prompt_toolkit import prompt  # pip install prompt_toolkit
while 1:
prompt('>')

Hold down the enter key, and it'll trigger quickly.

See also: https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1023

--
components: asyncio
messages: 358140
nosy: Jonathan Slenders, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: ProactorEventLoop raises unhandled ConnectionResetError
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com