Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On Wed, 6 Feb 2002 11:22:23 -0500, Brad Clements [EMAIL PROTECTED] wrote: Also, RESPONSE.setBody really should have access to REQUEST.headers. What's the clean way to do that? Just pass the request object to response object's init method? RESPONSE objects have a REQUEST attribute Are you sure? I know that request objects have a response object. But looking at publish.py doesn't look like it goes the other way. D'Oh, you are right. A better recommendation is to look at make_request in ZServer/HTTPResponse. It probably does make sense for the request headers to be stored in the response object, and I suspect this would be the place to do it. I am using Apache with mod_rewrite. Sure, it'd be great to compression there, but Apache doesn't cache, you need squid for that, right? I know it can be done with apache, but I prefer squid. I agree, Transfer Encoding is the way to go, but based on remarks at: http://www.iol.ie/~alank/python/httpcomp.html#encoding I stuck with the simpler to understand content-encoding. I think those remarks are misleading, and I dont find rfc2616 at all unclear in *this* area. Content-Encoding is a property of the entity being transferred. It is an end-to-end property (that is, from origin server to browser) and intermediate proxies or caches are not permitted to tamper with it. Transfer-Encoding is a property of the HTTP conversation (that is the conversation from browser to cache, from cache to another cache, from cache to front-end-proxy, and front-end-proxy to back-end-zope). If both ends of a conversation implement the same transfer-encoding (gzip, rsync, etc) then they could transparently apply the encoding as the document travels over the wire. My conclusions: 1. Adding Transfer-Encoding support to Zope is unnecessary if you are using a front-end proxy (there are many reasons why most people should, but some important exceptions where you need not). 2. Adding Content-Encoding support to Zope is a good thing only if your front-end proxy does not support on-the-fly Transfer-Encoding. I think apache does, but squid does not. Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
I don't have much useful to add - I just wanted to mention that I know
there are people out there who have succesfully used mod_gzip with Zope;
and that I *like* the name dogzip :-)
seb
On Tue, 2002-02-05 at 22:34, Brad Clements wrote:
I'm looking for architectural suggestions for adding gzip compression to
HTTPResponse for text types.
First, I just wanted to compress xml-rpc output, since I'm returing lots of table
data as
XML text (not objects), then loading that text/xml into a DOM for XSLT processing.
I hacked the attached code into HTTPResponse, at the end of setBody. It works for
xml-rpc responses and I suppose any text output, so long as the response object has
a
header named dogzip set.
I know dogzip is a stupid name, but this is just a testing thing.
Representative compressions:
compress oldlen 150366 new len 11926
compress oldlen 204382 new len 14170
compress oldlen 12746 new len 1364
As you can see, very useful compressions for xml-rpc output.
But for HTML output, what's really needed is I think a special kind of Cache Object.
One that combines HTTP Caching with Ram caching to keep gzip compressed objects
in memory.
Some HTML pages are really quite large, and gzip compression can make a noticable
difference. Just the javascript code sizes themselves are .. really big.
For xml-rpc, obviously every response must be compressed if it's worth it, and I
can
see that having to set a response property on a per request basis is appropriate for
xml-rpc.
But for text file objects, Page Templates and stuff.. How does setBody work with Ram
Cache objects? I have some ideas...
Anyone think this is worthwhile?
Also, RESPONSE.setBody really should have access to REQUEST.headers. What's
the clean way to do that? Just pass the request object to response object's init
method?
Here's quick gzip compression hack-in, based on code posted by Neil Schemenauer
Thanks Neil.
Added about line 265 in HTTPResponse.py in Zope 2.5 B3
try:
dogzip = self.headers['dogzip']
del self.headers['dogzip']
if dogzip and split(content_type,'/')[0] == 'text':
body = self.body
startlen = len(body)
import zlib, struct
_gzip_header = (\037\213 # magic
\010 # compression method
\000 # flags
\000\000\000\000 # time
\002
\377)
co = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS,
zlib.DEF_MEM_LEVEL,0)
chunks = [_gzip_header, co.compress(body),
co.flush(),struct.pack(ll,zlib.crc32(body),startlen)]
z = join(chunks,)
newlen = len(z)
print compress oldlen ,startlen,new len,newlen
if newlen startlen:
self.body = z
self.setHeader('content-length', newlen)
self.setHeader('content-encoding','gzip')
except:
pass
Brad Clements,[EMAIL PROTECTED] (315)268-1000
http://www.murkworks.com (315)268-9812 Fax
netmeeting: ils://ils.murkworks.com AOL-IM: BKClements
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On Tue, 5 Feb 2002 17:34:26 -0500, Brad Clements [EMAIL PROTECTED] wrote: I hacked the attached code into HTTPResponse, at the end of setBody. It works for xml-rpc responses and I suppose any text output, so long as the response object has a header named dogzip set. I think you also need to check the accept-encoding header, to allow for clients that do not know how to gunzip. That also means you should set caching headers to prevent the compressed and uncompressed responses getting delivered to the wrong clients by a cache. Also, RESPONSE.setBody really should have access to REQUEST.headers. What's the clean way to do that? Just pass the request object to response object's init method? RESPONSE objects have a REQUEST attribute Anyone think this is worthwhile? I looked at this roughly 18 months ago and came to the conclusion that (at the time) adding content-encoding support in Zope was the wrong way to do it. It you are using Zope behind a front-end proxy and you really should be. then it seems like a better idea to deliver the message to that proxy in an uncompressed form, and let it negotiate a transfer-encoding on its own. (Note that is transfer-, not content-encoding) The advantages of this scheme come from the fact that transfer-encoding is a hop-by-hop property. Two downstream caches can negotiate the best compression for that hop. Pushing everything downstream takes load away from zope, and making it a local choice means that the choice is often a better one. At the time, this type of auto-compressing proxies looked like they were just coming of age (http://rproxy.samba.org looked good at the time too). Unfortunately nothing has changed since. Today I think only Apache can do this (and has done for ages). Support in squid has stalled (http://devel.squid-cache.org/projects.html#te). Although I still think this is the way of the future, I suspect the short-term advantage of content-encoding the way you implemented it may be an advanatge for longer than I originally thought. I hope this helps, Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On 6 Feb 2002 at 10:02, seb bacon wrote: I don't have much useful to add - I just wanted to mention that I know there are people out there who have succesfully used mod_gzip with Zope; and that I *like* the name dogzip :-) That's my dog, zip! Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On 6 Feb 2002 at 10:40, Toby Dickenson wrote: I think you also need to check the accept-encoding header, to allow for clients that do not know how to gunzip. That also means you should set caching headers to prevent the compressed and uncompressed responses getting delivered to the wrong clients by a cache. Right, but I couldn't figure out how to see the request headers from response.setBody Also, RESPONSE.setBody really should have access to REQUEST.headers. What's the clean way to do that? Just pass the request object to response object's init method? RESPONSE objects have a REQUEST attribute Are you sure? I know that request objects have a response object. But looking at publish.py doesn't look like it goes the other way. At the time, this type of auto-compressing proxies looked like they were just coming of age (http://rproxy.samba.org looked good at the time too). Unfortunately nothing has changed since. Today I think only Apache can do this (and has done for ages). Support in squid has stalled (http://devel.squid-cache.org/projects.html#te). Although I still think this is the way of the future, I suspect the short-term advantage of content-encoding the way you implemented it may be an advanatge for longer than I originally thought. I am using Apache with mod_rewrite. Sure, it'd be great to compression there, but Apache doesn't cache, you need squid for that, right? For non-xmlrpc responses I'd want the stuff cached. I agree, Transfer Encoding is the way to go, but based on remarks at: http://www.iol.ie/~alank/python/httpcomp.html#encoding I stuck with the simpler to understand content-encoding. -- Shouldn't downstream caching proxies ungzip a response if they get a connection from a client that doesn't support gzip? Or will they only do this if its Transfer-Encoded? Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
I have been working on a compression patch as a component of my PatchKit
product for some time now, but I have been hitting a few problems, and had
put it to one side while I worked on a couple of other products. If people
still want to see something like this, I can bring the code out of
moth-balls and start working on the problems.
Adrian...
--
The difficulty of tactical maneuvering consists in turning the devious into
the direct, and misfortune into gain.
- Sun Tzu
- Original Message -
From: Brad Clements [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: Tuesday, February 05, 2002 10:34 PM
Subject: [Zope-dev] Adding gzip compression to HTTPResponse.py
I'm looking for architectural suggestions for adding gzip compression to
HTTPResponse for text types.
First, I just wanted to compress xml-rpc output, since I'm returing lots
of table data as
XML text (not objects), then loading that text/xml into a DOM for XSLT
processing.
I hacked the attached code into HTTPResponse, at the end of setBody. It
works for
xml-rpc responses and I suppose any text output, so long as the response
object has a
header named dogzip set.
I know dogzip is a stupid name, but this is just a testing thing.
Representative compressions:
compress oldlen 150366 new len 11926
compress oldlen 204382 new len 14170
compress oldlen 12746 new len 1364
As you can see, very useful compressions for xml-rpc output.
But for HTML output, what's really needed is I think a special kind of
Cache Object.
One that combines HTTP Caching with Ram caching to keep gzip compressed
objects
in memory.
Some HTML pages are really quite large, and gzip compression can make a
noticable
difference. Just the javascript code sizes themselves are .. really big.
For xml-rpc, obviously every response must be compressed if it's worth
it, and I can
see that having to set a response property on a per request basis is
appropriate for
xml-rpc.
But for text file objects, Page Templates and stuff.. How does setBody
work with Ram
Cache objects? I have some ideas...
Anyone think this is worthwhile?
Also, RESPONSE.setBody really should have access to REQUEST.headers.
What's
the clean way to do that? Just pass the request object to response
object's init
method?
Here's quick gzip compression hack-in, based on code posted by Neil
Schemenauer
Thanks Neil.
Added about line 265 in HTTPResponse.py in Zope 2.5 B3
try:
dogzip = self.headers['dogzip']
del self.headers['dogzip']
if dogzip and split(content_type,'/')[0] == 'text':
body = self.body
startlen = len(body)
import zlib, struct
_gzip_header = (\037\213 # magic
\010 # compression method
\000 # flags
\000\000\000\000 # time
\002
\377)
co = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS,
zlib.DEF_MEM_LEVEL,0)
chunks = [_gzip_header, co.compress(body),
co.flush(),struct.pack(ll,zlib.crc32(body),startlen)]
z = join(chunks,)
newlen = len(z)
print compress oldlen ,startlen,new len,newlen
if newlen startlen:
self.body = z
self.setHeader('content-length', newlen)
self.setHeader('content-encoding','gzip')
except:
pass
Brad Clements,[EMAIL PROTECTED] (315)268-1000
http://www.murkworks.com (315)268-9812 Fax
netmeeting: ils://ils.murkworks.com AOL-IM: BKClements
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
