[issue1508475] transparent gzip compression in urllib

[Demian Brecht]

Changes by Demian Brecht :


--
nosy: +demian.brecht

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Martin Panter]

Martin Panter added the comment:

I suggest resolving Issue 15955 first, then the GzipFile API could be used 
without fear of decompression bombs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +berker.peksag

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Martin Panter]

Martin Panter added the comment:

The Lib/xmlrpc/client.py file appears to already support compression using 
“Content-Encoding: gzip”. Perhaps it could be leveraged for any work on this 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Martin Panter]

Martin Panter added the comment:

Related: Issue 1243678, which includes a patch for “httplib” (now known as 
“http.client”?). That patch looks like it sets Accept-Encoding and decodes 
according to Content-Encoding. However I suspect it is also trying to be too 
“transparent” at the wrong level and would have many of the problems already 
mentioned here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Martin Panter]

Martin Panter added the comment:

I think the patch is indeed a bit short, for instannce it looks like calling 
read() without a size limit could bypass the decoding.

Also, I wonder if Content-Encoding handling is better done at a higher level. 
What if someone wants to download a *.tar.gz file? They may not expect the tar 
file to be transparently decompressed. And I suspect this would blow up if you 
tried a partial range request.

Transfer-Encoding is meant to be the proper way to transparently compress HTTP 
messages at a low level, but it doesn’t seem to be used as much in the real 
world.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Berker Peksag]

Changes by Berker Peksag :


--
versions: +Python 3.5 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Jim Jewett]

Jim Jewett added the comment:

This is an enhancement, so I am changing the affected version from 3.3 to 3.5.  

It is python-only, which works well with the cheeseshop.

That said, the patch is truly short; if that is really sufficient, it could 
almost go into the documentation as a recipe.  But I would prefer some more 
assurances that it actually does work; a quick skim suggests that it relies on 
a superclass happening to implement read via readinto.

Needs tests and documentation change.

--
nosy: +Jim.Jewett

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Raymond Hettinger]

Raymond Hettinger added the comment:

Victor, the patch looks good and would be a welcome enhancement.

There should be an option for turning this on and off (perhaps, I want the 
zipped content and want to unzip later or in a different thread).

Consider adding support for "deflate" as well.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Martin Panter]

Martin Panter added the comment:

I have code that already handles an “gzip” encoded response from urlopen(). All 
three patches leave the Content-Encoding header intact, so I suspect my code 
would try to decompress the body a second time. Deleting this header (as 
already suggested) would work for me.

--
nosy: +vadmium

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[STINNER Victor]

STINNER Victor added the comment:

I updated issue1508475.diff for Python 3.4 and removed the change in urllib: 
http_client_gzip.patch. The patch only changes http.client to support server 
sending gzip data.

For example, the new python.org website serves gzip data even if the 
Accept-Encoding header is not sent by the client: see the issue #20719.

--
keywords: +patch
nosy: +haypo
Added file: http://bugs.python.org/file34177/http_client_gzip.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[John O'Connor]

Changes by John O'Connor :


--
nosy: +jcon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Stephen Thorne]

Changes by Stephen Thorne :


--
nosy: +jerub

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

I think, the transparent compression should work at http.client level. I also 
agree with other points made by Serhiy:

- transparent decompression should delete headers Content-Encoding and 
Content-Length (this is as per RFC too)

- Should not do another compression if the user has a explicit specified intent 
of using Content-Encoding: gzip and is ready to do decompression himself.

- This transparent compression/decompression would require the availability 
gzip module, if not then the feature may be disabled and normal 
request-response cycle would proceed.

- I think, having it 'ON' with a flag to switch 'OFF' would be more desirable 
than having this feature via Handler. The reason being it can help in 
performance of any requests on servers that support it and browsers have 
adopted similar approach too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The first step is to answer on the fundamental question: on what level 
transparent decompression will work? On http.client level or on urllib level? 
Patch for first case will be much more difficult, but will benefit from 
compression in other http-based protocols.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
priority: normal -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


--
nosy: +jcea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Éric Araujo]

Éric Araujo  added the comment:

Enabled by default with a knob to turn it off sounds good.  Maybe the original 
headers could be preserved in some object.

--
keywords:  -easy, patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Tom Pinckney]

Tom Pinckney  added the comment:

What if this gzip decompression was optional and controlled via a flag or 
handler instead of making it automagic?

It's not entirely trivial to implement so it is nice to have the option of this 
happening automatically if one wishes.

Then, the caller would be aware that Content-length / Accept-encoding / 
Content-encoding etc have been modified iff they requested gzip decompression.

--
nosy: +thomaspinckney3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The patch for py3k also has the disadvantage that the content is decoded even 
if the user has defined a Content-Encoding and he is going to process 
compressed response himself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

In that case, transparent decompression should not be available. (
Request header should not be sent and response wont be compressed).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

What if the gzip module is not available?

I think, with transparent decompression should delete headers Content-Encoding 
(to free the user from re-decompression) and Content-Length (which is wrong).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +storchaka
versions: +Python 3.3 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Evgeny Kapun]

Changes by Evgeny Kapun :


--
nosy: +abacabadabacaba

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Patch for py3k.

--
stage: unit test needed -> patch review
Added file: http://bugs.python.org/file19811/issue1508475.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Senthil Kumaran]

Senthil Kumaran  added the comment:

The transparent gzip Content-Encoding support should be done at the
http.client level code.

Before adding this feature, a question needs to be sorted out.

If we support the transparent gzip and wrap the file pointer to a
GzipFile filepointer, should reset the Content-Length value?

What if a user of urllib is relying on the Content-Length of response
to do something further?

I observed that google-chrome returns the uncompressed output (which
is correct for a browser), but has the Content-Length set the
compressed output length.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Nadeem Vawda]

Changes by Nadeem Vawda :


--
nosy: +nvawda

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[MunSic JEONG]

Changes by MunSic JEONG :


--
nosy: +ruseel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1508475] transparent gzip compression in urllib

[Éric Araujo]

Changes by Éric Araujo :


--
components:  -Extension Modules
nosy: +eric.araujo -BreamoreBoy
title: transparent gzip compression in liburl2 -> transparent gzip compression 
in urllib

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com