[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

> ResourceWarning?

That's a small bug in the test, but the main issue is that 
test_huge_content_recvinto() has a race condition.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[Andrew Svetlov]

Change by Andrew Svetlov :


--
assignee:  -> asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[Andrew Svetlov]

Andrew Svetlov  added the comment:

ResourceWarning?
I'll take a look

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

Recent failure on AMD64 Windows7 SP1 3.8:
https://buildbot.python.org/all/#/builders/208/builds/268

test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... 
C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\support\__init__.py:1637: 
ResourceWarning: unclosed 
ResourceWarning: Enable tracemalloc to get the object allocation traceback

ERROR: test_huge_content 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests)

Re-running failed tests in verbose mode
C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\support\__init__.py:1637: 
ResourceWarning: unclosed 
ResourceWarning: Enable tracemalloc to get the object allocation traceback
FAIL: test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

The failing test was added by:

commit 74387926072abf338a4c1cec1bf0501fc65bbee5
Author: Andrew Svetlov 
Date:   Mon Nov 12 19:00:22 2018 +0200

bpo-30064: Refactor sock_* asyncio API (#10419)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

We get a ConnectionResetError exception thanks to IocpProactor.recv() callback:

def finish_recv(trans, key, ov):
try:
return ov.getresult()
except OSError as exc:
if exc.winerror in (_overlapped.ERROR_NETNAME_DELETED,
_overlapped.ERROR_OPERATION_ABORTED):
raise ConnectionResetError(*exc.args)
else:
raise

Would it be legit to convert ConnectionResetError to returning b'' in 
sock_recv()? Example with the test:

while True:
data = await self.loop.sock_recv(sock, DATA_SIZE)
if not data:
break
expected = bytes(islice(checker, len(data)))
self.assertEqual(data, expected)
size -= len(data)
self.assertEqual(size, 0)

"if not data:" is a common test to check if we reached the end of file or if a 
socket has been closed by the peer, no?

Proposed patch:

diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index 9b8ae064a8..14b7f10729 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -687,10 +687,16 @@ class BaseProactorEventLoop(base_events.BaseEventLoop):
 super().close()
 
 async def sock_recv(self, sock, n):
-return await self._proactor.recv(sock, n)
+try:
+return await self._proactor.recv(sock, n)
+except (ConnectionResetError, ConnectionAbortedError):
+return b''
 
 async def sock_recv_into(self, sock, buf):
-return await self._proactor.recv_into(sock, buf)
+try:
+return await self._proactor.recv_into(sock, buf)
+except (ConnectionResetError, ConnectionAbortedError):
+return 0
 
 async def sock_sendall(self, sock, data):
 return await self._proactor.send(sock, data)


I'm not sure about this change.

I'm not sure about sock_recv_info(): are you suppose to truncate buf to 0 bytes 
in this case?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[Andrew Svetlov]

Andrew Svetlov  added the comment:

There is an interesting discussion: 
https://stackoverflow.com/questions/14499593/can-the-infamous-error-netname-deleted-error-be-considered-an-error-at-all

"[WinError 64] The specified network name is no longer available" is raised on 
pending reads from closed socket.

Looks like we can just drop these reads without error reporting, it is safe and 
easy.

Victor, what do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

New failure on AMD64 Windows7 SP1 3.8:
https://buildbot.python.org/all/#/builders/208/builds/75

==
ERROR: test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests)
--
Traceback (most recent call last):
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 225, in test_huge_content_recvinto
self.loop.run_until_complete(
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\base_events.py", line 
608, in run_until_complete
return future.result()
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 211, in _basetest_huge_content_recvinto
nbytes = await self.loop.sock_recv_into(sock, buf)
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\selector_events.py", 
line 400, in sock_recv_into
return await fut
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\selector_events.py", 
line 409, in _sock_recv_into
nbytes = sock.recv_into(buf)
ConnectionResetError: [WinError 10054] An existing connection was forcibly 
closed by the remote host

(... test_asyncio re-run in verbose mode ...)

==
ERROR: test_huge_content 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests)
--
Traceback (most recent call last):
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\windows_events.py", 
line 453, in finish_recv
return ov.getresult()
OSError: [WinError 64] The specified network name is no longer available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 170, in test_huge_content
self.loop.run_until_complete(
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\base_events.py", line 
608, in run_until_complete
return future.result()
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 157, in _basetest_huge_content
data = await self.loop.sock_recv(sock, DATA_SIZE)
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\proactor_events.py", 
line 690, in sock_recv
return await self._proactor.recv(sock, n)
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\windows_events.py", 
line 808, in _poll
value = callback(transferred, key, ov)
  File 
"C:\buildbot.python.org\3.8.kloth-win64\build\lib\asyncio\windows_events.py", 
line 457, in finish_recv
raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 64] The specified network name is no longer 
available

--
resolution: out of date -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

I didn't see this issue recently. I close it.

--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36732] test_asyncio: test_huge_content_recvinto() fails randomly

[STINNER Victor]

New submission from STINNER Victor :

Failure on AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/2053

...
test_start_unix_server_1 
(test.test_asyncio.test_server.SelectorStartServerTests) ... skipped 'no Unix 
sockets'
test_create_connection_sock 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... ok
test_huge_content (test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) 
... ok
test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... ERROR
test_sock_accept (test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) 
... 
C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\support\__init__.py:1627: 
ResourceWarning: unclosed 
  gc.collect()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ok
test_sock_client_fail 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... ok
test_sock_client_ops 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... ok
test_unix_sock_client_ops 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests) ... skipped 'No 
UNIX Sockets'
test_create_connection_sock 
(test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests) ... ok
test_huge_content (test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests) 
... ok
test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.SelectEventLoopTests) ... ok
...

==
ERROR: test_huge_content_recvinto 
(test.test_asyncio.test_sock_lowlevel.ProactorEventLoopTests)
--
Traceback (most recent call last):
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py", 
line 474, in finish_recv
return ov.getresult()
OSError: [WinError 64] The specified network name is no longer available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 225, in test_huge_content_recvinto
self.loop.run_until_complete(
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\base_events.py", line 
590, in run_until_complete
return future.result()
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_asyncio\test_sock_lowlevel.py",
 line 211, in _basetest_huge_content_recvinto
nbytes = await self.loop.sock_recv_into(sock, buf)
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\proactor_events.py", 
line 551, in sock_recv_into
return await self._proactor.recv_into(sock, buf)
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py", 
line 760, in _poll
value = callback(transferred, key, ov)
  File 
"C:\buildbot.python.org\3.x.kloth-win64\build\lib\asyncio\windows_events.py", 
line 478, in finish_recv
raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 64] The specified network name is no longer 
available

--
components: Tests, asyncio
messages: 340892
nosy: asvetlov, vstinner, yselivanov
priority: normal
severity: normal
status: open
title: test_asyncio: test_huge_content_recvinto() fails randomly
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com