[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[miss-islington]

miss-islington  added the comment:


New changeset 9d37ae0bee25692572c201378cd0692df22fa2ac by Miss Islington (bot) 
in branch '3.8':
bpo-36918: Fix "Exception ignored in" in test_urllib (GH-13996)
https://github.com/python/cpython/commit/9d37ae0bee25692572c201378cd0692df22fa2ac


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[miss-islington]

Change by miss-islington :


--
pull_requests: +13861
pull_request: https://github.com/python/cpython/pull/13998

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

STINNER Victor  added the comment:


New changeset eb976e47e261760330c1bed224019b073b05e994 by Victor Stinner in 
branch 'master':
bpo-36918: Fix "Exception ignored in" in test_urllib (GH-13996)
https://github.com/python/cpython/commit/eb976e47e261760330c1bed224019b073b05e994


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13859
pull_request: https://github.com/python/cpython/pull/13996

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

STINNER Victor  added the comment:

I proposed PR 13955: a fix in 2 lines.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

STINNER Victor  added the comment:

Well, maybe Python GC can be enhanced. In the meanwhile, I would like to make 
these warnings quiet since they are very annoying when I have to analyze 
buildbot warnings. Moreover, this issues blocks my bpo-37069.

Maybe open a separated issue to propose to enhance the GC, if you have an idea 
how to change it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13823
pull_request: https://github.com/python/cpython/pull/13955

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests:  -13428

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13428

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

My analysis was that close was called on fakesocket which is internally closed 
when it's counter resets to zero and the destructor was trying to flush on a 
closed object during destructor call.

Comment from https://bugs.python.org/issue18748#msg341106

The ValueError warnings in test_urllib noted in msg340059  feels like an issue 
with the test where FakeSocket calls close by itself once it's internal io_refs 
counter is 0 and during destructor again close is called on an already closed 
object causing the ValueError.

* The test uses fakehttp() creating FakeSocket starting with io_refs as 1 [1]
* During the test in urlopen, sock.makefile() (io_refs += 1) is called 
increasing the io_refs to be 2.
* HTTPConnection.close calls sock.close (fakesocket.close) [3] calling io_refs 
-= 1 and to fakesocket.io_ref to be 1.
* This test uses raises status 302 with http_error_302 which calls 
self.redirect_internal where self.close is again called causing fp.io_refs == 0 
and subsequently calling io.BytesIO.close(self) to close the object. [4]
* During the end of the test destructor is called on the above closed 
fakesocket object and trying to flush on a closed object causes the ValueError 
warnings .

Maybe a check could be added during flush to make sure the object is not closed 
by testing for self.closed but I am not sure if closed attribute is guaranteed 
to be present. Removing the manual call to close in fakesocket could help since 
the destructor should be taking care of it and I could see no test failures or 
warnings removing the close as io_refs gets to 0.

[1] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L61
[2] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/test/test_urllib.py#L67
[3] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/http/client.py#L919
[4] 
https://github.com/python/cpython/blob/be6dbfb43b89989ccc83fbc4c5234f50f44c47ad/Lib/urllib/request.py#L2145

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

No, BasicIO.close() is correct.

This is a bug in the garbage collector: the underlying file is closed before 
closing HTTPResponse. The HTTPResponse instance has a reference to the file 
object, the file object does not have a reference to the HTTPResponse instance, 
therefore the HTTPResponse instance should be destroyed first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I think this is a bug in BasicIO. close() should be idempotent. Calling it on 
the closed file should have no effect.

--
components: +IO -Tests
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
keywords: +patch
pull_requests: +13236
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36918] ValueError warning in test_urllib due to io.IOBase destructor

[Karthikeyan Singaravelan]

New submission from Karthikeyan Singaravelan :

Issue for https://bugs.python.org/issue18748#msg340059.

comment : 

Is there someone interested to debug remaining "Exception ignored:" logs in 
test_urllib?

test_invalid_redirect (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_read_bogus (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok

test_redirect_limit_independent (test.test_urllib.urlopen_HttpTests) ...
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
Exception ignored in: 
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 402, in 
close
super().close() # set "closed" flag
  File "/home/vstinner/prog/python/master/Lib/http/client.py", line 415, in 
flush
self.fp.flush()
ValueError: I/O operation on closed file.
ok


It's unclear to be if it's a bug in http.client, a bug in the test... or 
something else.

--
components: Tests
messages: 342492
nosy: vstinner, xtreak
priority: normal
severity: normal
status: open
title: ValueError warning in test_urllib due to io.IOBase destructor
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com