[issue34629] Python3 regression for urllib(2).urlopen(...).fp for chunked http responses

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34629] Python3 regression for urllib(2).urlopen(...).fp for chunked http responses

[Jonathan Schweder]

Jonathan Schweder  added the comment:

Hello @tkruse, I have made some research and found that when using the Chunked 
transfer encoding [1], each chunk is preceded by its size in bytes, something 
that really happen if you check the content of one downloaded file from the 
example you provided [2]. So far, I would say that this is not a bug, it is 
just how the transfer encoding works.

[1]: https://en.wikipedia.org/wiki/Chunked_transfer_encoding
[2]: https://gist.github.com/jaswdr/95b2adc519d986c00b17f6572d470f2a

--
nosy: +jaswdr

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34629] Python3 regression for urllib(2).urlopen(...).fp for chunked http responses

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34629] Python3 regression for urllib(2).urlopen(...).fp for chunked http responses

[Thibault Kruse]

New submission from Thibault Kruse :

We had a problem running code that downloads files from github when porting 
from python2.7 to python3.[3-7]. Not sure if a bug or not.

With the given code, in python3 a file downloaded in chunks will contain the 
size of chunks when using the undocumented fp from urlopen(...).fp. In python2, 
only the chunk payload would make it into the file.

We assume that we can just use the urlopen response directly as a fix (without 
'.fp'), but though it might still be nice to report the difference.

Short code:
resp = urlopen('http://someurl')
fhand = os.fdopen(fdesc, "wb")
shutil.copyfileobj(resp.fp, fhand)   # using .fp here is the dodgy part
fhand.close()

The attached script demonstrates the difference:

$ python --version
Python 2.7.15rc1
$ python urllib_issue.py 
127.0.0.1 - - [12/Sep/2018 01:27:28] "GET /downloads/1.0.tar.gz HTTP/1.1" 200 -

$ python3 --version
Python 3.6.5
$ python3 urllib_issue.py 
127.0.0.1 - - [12/Sep/2018 01:27:37] "GET /downloads/1.0.tar.gz HTTP/1.1" 200 -
Traceback (most recent call last):
  File "urllib_issue.py", line 87, in 
assert data == FILE_CONTENT, '%s, %s'%(len(FILE_CONTENT), len(data))
AssertionError: 10, 100493
!!! BASH reports ERROR: shell returned 1

--
components: Library (Lib)
files: urllib_issue.py
messages: 325025
nosy: tkruse
priority: normal
severity: normal
status: open
title: Python3 regression for urllib(2).urlopen(...).fp for chunked http 
responses
type: behavior
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47796/urllib_issue.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com