[issue21069] test_fileno of test_urllibnet intermittently fails

[Martin Panter]

Changes by Martin Panter :


--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Roundup Robot]

Roundup Robot added the comment:

New changeset 00240ddce1d0 by Martin Panter in branch '3.5':
Issue #21069: Move test_fileno() from test_urllibnet and rewrite it
https://hg.python.org/cpython/rev/00240ddce1d0

New changeset 4c19396bd4a0 by Martin Panter in branch 'default':
Issue #21069: Merge test_fileno() from 3.5
https://hg.python.org/cpython/rev/4c19396bd4a0

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Berker Peksag]

Berker Peksag added the comment:

I saw test_fileno failure again on the Gentoo buildbot: 
http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/459/steps/test/logs/stdio

rewrite-fileno.patch looks good to me.

--
nosy: +berker.peksag
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Martin Panter]

Martin Panter added the comment:

Yes using select() would be another way to fix the immediate problem of read() 
returning None. In the long term I was hoping to use something like my patch to 
avoid the problems with reading the HTTP response (already discussed in this 
report) at the same time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[STINNER Victor]

STINNER Victor added the comment:

I added a timeout because test_urllibnet fails with a timeout after 15 minutes 
on the ARM buildbot and other tests of the same file also use a timeout.

I tried but failed to reproduce the timeout.

I ran the test after my change and it passes so I considered that it was ok. I 
understand your rationale for timeout, non blocking socket and read() returning 
None.

Can we modify the test to use a selector to wait until the socket is ready and 
only read available bytes, rather than reading all data?

Using a local server would also avoid blocking too long (take less than 15 min).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Martin Panter]

Martin Panter added the comment:

Here is a patch implementing my ideas. Let me know what you think of the ideas 
and/or the patch :)

If people think this change is too much for 3.5, I could try making a more 
minimal patch that calls something like socket.getpeername() rather than 
read(). That should also avoid the non-blocking problem.

--
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file42351/rewrite-fileno.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails

[Martin Panter]

Martin Panter added the comment:

Mark: My understanding is on Windows, winsock file descriptors and C library 
file descriptors are different beasts; see 
. Perhaps 
the test should call socket functions like socket.recv() on the FD rather than 
C library functions via os.fdopen().

Victor: The test in this bug has started failing again, very likely due to your 
revision 7bd4736195ce enabling a timeout on the HTTP request. I guess this 
causes the socket to be in non-blocking mode, and read() to return None. This 
is what Issue 10119 tried to fix. Example:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3404/steps/test/logs/stdio
==
FAIL: test_fileno (test.test_urllibnet.urlopenNetworkTests)
--
Traceback (most recent call last):
  File 
"/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_urllibnet.py", 
line 110, in test_fileno
self.assertTrue(f.read(), "reading from file created using fd "
AssertionError: None is not true : reading from file created using fd returned 
by fileno failed

A less serious and long-standing problem with the test is that it attempts to 
close the socket twice. We are just lucky that socket.close() is called second, 
which does not raise any errors: 
.

Regarding the purpose and use cases of fileno(), I agree with Senthil that 
using it to read the HTTP response behind the HTTPResponse object’s back in 
Python 3 is a bad idea, and I don’t think it is practical to make this work 
without losing the benefits of buffering. But there are probably other valid 
use cases such as calling getsockname() on the socket, or sending and receiving 
non-HTTP data after setting up a CONNECT tunnel.

Proposals:

1. Change the test to do use socket(fileno=...), rather than os.fdopen(...), so 
that it will be usable on Windows.

2. Ensure that the secondary socket object is not closed; use socket.detach()

3. Rewrite the test to test http.client directly, rather than indirectly 
through urlopen(). As far as I can see the purpose is only to test 
HTTPResponse.fileno(), not urlopen().

4. Rewrite the test to test a local server run in a background thread, rather 
than relying external web sites (currently Google, previously IETF, and 
Python). This would eliminate the need for setting a timeout.

5. Rewrite to the test for a more realistic use case that does not depend on 
specific internal HTTPResponse buffering and the HTTP protocol. I suggest 
mocking a CONNECT request, and uploading some non-HTTP data through the proxy.

--
components: +Tests
keywords: +buildbot -patch
nosy: +haypo, martin.panter
stage: test needed -> needs patch
title: test_fileno of test_urllibnet intermittently fails when using 
www.example.com -> test_fileno of test_urllibnet intermittently fails
type:  -> behavior
versions: +Python 3.6 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Mark Lawrence]

Mark Lawrence added the comment:

I tried reproducing this on Windows 8.1 with the code snippet from msg216429 
but got

Traceback (most recent call last):
  File "C:\Users\Mark\Documents\MyPython\mytest.py", line 9, in 
with os.fdopen(fd, 'rb') as f:
  File "c:\python34\lib\os.py", line 980, in fdopen
return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor

Is there something else that I can try?  Is this still an issue that needs 
looking into?

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Ned Deily]

Ned Deily added the comment:

Senthil, thanks for looking into this.  Since it is turning out to be more of a 
urllib design issue, I'm going to deassign myself from it.

--
assignee: ned.deily -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Senthil Kumaran]

Senthil Kumaran added the comment:

Ned Deily had done the correct analysis in the msg214947 and has this question 
- 

> I don't know whether the file descriptor read is expected to be meaningful 
> for urllib2/urllib.request.

I can see that this test case was for the old behavior where we created a file 
like object to read using addinfourl and fileno /fp was explicitly set.  We 
have to determine if this expectation that we can access the socket's fp using 
HTTPResponse object is a right one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Senthil Kumaran]

Senthil Kumaran added the comment:

This is turning out be trickier than I ever thought.

fileno() returning b'' is just random error. The underlying issue is, 
*directly* reading from the fp of the socket() is returning a incomplete output 
at all times. The correct way to read the output is by using HTTPResponse 
read() method only it seems.

Here is a small snippet that will fail for every url.

from urllib.request import urlopen

import os

web = "http://www.example.org";

open_url = urlopen(web)
fd = open_url.fileno()
with os.fdopen(fd, 'rb') as f:
file_read_len = len(f.read())

req = urlopen(web)
res_len = len(req.read())

assert file_read_len == res_len

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Senthil Kumaran]

Senthil Kumaran added the comment:

This is the best way I found to reproduce the failure.

I changed the resource to www.example.com and then ran this.

$ ./python.exe -m test -m "*fileno*" -u all -v -F test_urllibnet

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Senthil Kumaran]

Senthil Kumaran added the comment:

I am getting to this late.

>> I don't know whether the file descriptor read is expected to be meaningful 
>> for urllib2/urllib.request.
>> Senthil, what do you think?

It should be meaningful no matter what the length is. I am looking further into 
this now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +Arfrever

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Ned Deily]

Ned Deily added the comment:

After looking at why the 2.7 version of the test does not fail, the problem 
became apparent.  In 2.7, test_errno tests urlopen() of the original deprecated 
urllib module.  In 3.x, the test was ported over but now uses urlopen() of 
urllib.request which is based on urllib2() of 2.x.

2.7:
>>> x = urllib.urlopen("http://www.example.com";)
[79234 refs]
>>> x
>
[79234 refs]
>>> os.fdopen(x.fileno()).read()
'\n\n\nExample Domain\n\n
\n\n\n\n 
   body {\nbackground-color: #f0f0f2;\nmargin: 0;\n
padding: 0;\nfont-family: "Open Sans", "Helvetica Neue", Helvetica, 
Arial, sans-serif;\n\n}\ndiv {\nwidth: 600px;\n
margin: 5em auto;\npadding: 50px;\nbackground-color: #fff;\n
border-radius: 1em;\n}\na:link, a:visited {\ncolor: 
#38488f;\ntext-decoration: none;\n}\n@media (max-width: 700px) 
{\nbody {\nbackground-color: #fff;\n}\ndiv 
{\nwidth: auto;\nmargin: 0 auto;\n
border-radius: 0;\npadding: 1em;\n}\n}\n
\n\n\n\n
 \nExample Domain\nThis domain is established to be used 
for illustrative examples in documents. You may use this\ndomain in 
examples without prior coordination or asking for permission.\nhttp://www.iana.org/domains/example";>More 
information...\n\n\n\n'
[79234 refs]

3.4 (when the read doesn't fail):
>>> x = urllib.request.urlopen("http://www.example.com";)
>>> x

>>> os.fdopen(x.fileno()).read()
__main__:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=4 mode='r' 
encoding='UTF-8'>
' without prior coordination or asking for permission.\nhttp://www.iana.org/domains/example";>More 
information...\n\n\n\n'

In the 3.x case (and the 2.7 urllib2 case), the read from the file descriptor 
starts at mid-response or at the end (returning an empty byte string).  In the 
past, the test passed because of the amount of data returned by the previous 
test URL.  Now, with the short response from www.example.com, it's clear that 
the file descriptor read is not returning the whole response.  I don't know 
whether the file descriptor read is expected to be meaningful for 
urllib2/urllib.request.

Senthil, what do you think?

--
nosy: +orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21069] test_fileno of test_urllibnet intermittently fails when using www.example.com

[Ned Deily]

Ned Deily added the comment:

After pushing the changes for Issue20939, many of the buildbots started 
experiencing the test_fileno failure using www.example.com.  The interesting 
thing is that not all of them do, including my primary development system (OS X 
10.9) which is why I didn't see a problem during the initial testing.  On 
another dev system (Debian Linux on a VM), the test fails intermittently. So, 
yes, something funky *is* going on.  One obvious difference is that the read 
from www.python.org returns 44000+ byes while www.example.com returns only 162 
bytes.  There could be a race condition with the TCP connection close.  I've 
temporarily changed the test to use www.google.com pending resolution. Thanks 
for the report and investigation, Daniel.

--
priority: normal -> high
stage: needs patch -> test needed
title: urllib unit tests failing without ssl module -> test_fileno of 
test_urllibnet intermittently fails when using www.example.com

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com