Re: "Streaming" compression of output from mod_perl handler?
> Can I find the Apache::Dynagzip handler anywhere - Google returns nothing... As far as I know the Apache::Dynagzip is not on CPAN yet. Currently, it's a property of OLT. Let me know if you are doing the Open Source Project and I will try to negotiate your needs with OLT. In case of commercial project you have to contact Tom Evans [EMAIL PROTECTED] yourself. > > Is your handler Apache::Filter Chain compatible? > Not at the moment. But as it is only a content producer, isn't the only requirement, that I insert "$r->filter_register()" somewhere near the top of the handler, before returning any output? At least that is what I did, when I tried to test the Apache::Compress handler (which I didn't succeed in getting to work properly). Right, it's not a big deal when you have the source... Thanks, Slava
Re: "Streaming" compression of output from mod_perl handler?
On Wed, 20 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > Sorry again. I've just checked sources and found that if mod_deflate > > received flush then it flushes both zlib and Apache. > > > > You can try to set autoflush in perl module with $|=1; > > or call $r->rflush; when you like to flush output. > > I just tried using $r->rflush in my handler and it works perfectly :-) > The output gets rendered on the fly and it barely hurts the compression ratio. The >430 KB gets compressed to 26 KB instead of 24.5 KB :-) :-) What browsers did you test ? > But wouldn't it be nice to have some mod_deflate option where you could specify that >mod_deflate should flush and send the currently compressed output every time it had >received a certain amount of input or every time it had generated a certain amount of >output. It's complicates things. Besides Apache never flushes output on timeout or amount of content - it flushes only if you ask it. > We are, for instance, using a template module to generate the output. We just give >the template module a data structure and a template, and it then goes away and fills >in the template, outputting HTML. This means that we don't have any easy way of >calling flush at certain So we don't have any easy way of calling rflush once in a >while. > Of course we might just modify or substitute the template module, but it seems like >a more "automatic" kind of "streaming" deflating (like described above) would be nice >to have. You can set $|=1 as Eagle book says: This method [r->flush()] is also called automatically after each print() if the Perl global variable $| is nonzero. Igor Sysoev
Re: "Streaming" compression of output from mod_perl handler?
> Sorry again. I've just checked sources and found that if mod_deflate > received flush then it flushes both zlib and Apache. > > You can try to set autoflush in perl module with $|=1; > or call $r->rflush; when you like to flush output. I just tried using $r->rflush in my handler and it works perfectly :-) The output gets rendered on the fly and it barely hurts the compression ratio. The 430 KB gets compressed to 26 KB instead of 24.5 KB :-) :-) But wouldn't it be nice to have some mod_deflate option where you could specify that mod_deflate should flush and send the currently compressed output every time it had received a certain amount of input or every time it had generated a certain amount of output. We are, for instance, using a template module to generate the output. We just give the template module a data structure and a template, and it then goes away and fills in the template, outputting HTML. This means that we don't have any easy way of calling flush at certain So we don't have any easy way of calling rflush once in a while. Of course we might just modify or substitute the template module, but it seems like a more "automatic" kind of "streaming" deflating (like described above) would be nice to have. Nicholas Oxhøj
Re: "Streaming" compression of output from mod_perl handler?
On Wed, 20 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > I'm not sure that lynx can handle compressed response on the fly - > > it uses gzip in pipe. > > The best way to test it using netcat. > > Well, lynx didn't decompress it, it just output the gzip compressed content to >stdout. As I didn't have netcat readily available on the machine, I instead put an >strace on lynx, to be absolutely sure, that it didn't receive any output until the >very end - and it didn't :-( > > > I you like to test I can make patch for mod_deflate to flush Apache. > > But if major browsers can not handle compressed content on the fly > > it's not valuable. > > That would be an interesting patch, but with approx 450KB of uncompressed HTML, I >would expect mod_deflate to receive compressible input, regardless if the content >producer specifically flushes or not. But I might have misunderstood something. > > Regarding the browsers ability to handle compressed content on the fly, we probably >won't know until I find a module that is able to produce such output. Sorry again. I've just checked sources and found that if mod_deflate received flush then it flushes both zlib and Apache. You can try to set autoflush in perl module with $|=1; or call $r->rflush; when you like to flush output. Igor Sysoev
Re: "Streaming" compression of output from mod_perl handler?
> I'm not sure that lynx can handle compressed response on the fly - > it uses gzip in pipe. > The best way to test it using netcat. Well, lynx didn't decompress it, it just output the gzip compressed content to stdout. As I didn't have netcat readily available on the machine, I instead put an strace on lynx, to be absolutely sure, that it didn't receive any output until the very end - and it didn't :-( > I you like to test I can make patch for mod_deflate to flush Apache. > But if major browsers can not handle compressed content on the fly > it's not valuable. That would be an interesting patch, but with approx 450KB of uncompressed HTML, I would expect mod_deflate to receive compressible input, regardless if the content producer specifically flushes or not. But I might have misunderstood something. Regarding the browsers ability to handle compressed content on the fly, we probably won't know until I find a module that is able to produce such output. Nicholas Oxhøj
Re: "Streaming" compression of output from mod_perl handler?
> I've developed an Apache::Dynagzip handler for Outlook > Technologies, Inc. to > serve the dynamic content with the option to control the size of the > chunk(s). It works fine (as standing along, and within the > Apache::Filter > Chain). Using the Apache::Dynagzip you have several options > to control your > chunk size(s). You can even control the size of every chunk > from the source > generator: Just send the mask of the end of the chunk to the > Apache::Dynagzip within the outgoing content/stream. (You > might wish to send > your header to the client browser while creating the rest of > the body...) > Otherwise, it is buffering the outgoing stream up to the length of the > chunk's minimum size , which is not less then declared (2K default). It certainly sounds very promising. Can I find the Apache::Dynagzip handler anywhere - Google returns nothing... > Is your handler Apache::Filter Chain compatible? Not at the moment. But as it is only a content producer, isn't the only requirement, that I insert "$r->filter_register()" somewhere near the top of the handler, before returning any output? At least that is what I did, when I tried to test the Apache::Compress handler (which I didn't succeed in getting to work properly). Nicholas Oxhøj
Re: "Streaming" compression of output from mod_perl handler?
I've developed an Apache::Dynagzip handler for Outlook Technologies, Inc. to serve the dynamic content with the option to control the size of the chunk(s). It works fine (as standing along, and within the Apache::Filter Chain). Using the Apache::Dynagzip you have several options to control your chunk size(s). You can even control the size of every chunk from the source generator: Just send the mask of the end of the chunk to the Apache::Dynagzip within the outgoing content/stream. (You might wish to send your header to the client browser while creating the rest of the body...) Otherwise, it is buffering the outgoing stream up to the length of the chunk's minimum size , which is not less then declared (2K default). Is your handler Apache::Filter Chain compatible? Thanks, Slava Bizyayev - Original Message - From: "Igor Sysoev" <[EMAIL PROTECTED]> To: "Nicholas Oxhoj" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Tuesday, February 19, 2002 4:12 AM Subject: Re: "Streaming" compression of output from mod_perl handler? > On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > > I am looking for an Apache module which will allow me to compress the output of my mod_perl handler (a "native" handler, i.e. not running under Apache::Registry). But since my handler can potentially take a long time to finish, the output has to be compressed in a "streaming" fashion (or in small blocks) so that the browser will start receiving data before my handler has completely finished. > > > > I have been experimenting with all the different Apache compression modules I have been able to find, but have not been able to get the desired result. I have tried Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with different results. One I cannot get to work at all. Most work, but seem to collect all the output before compressing it and sending it to the browser. > > > > There also seems to be an issue about the new HTTP/1.1 "chunked" transfer-encoding. For instance, mod_gzip will not compress chunked output, unless you allow it to "dechunk" it by collecting all the output and compressing it as one big block. > > > > So I am basically looking for anyone who has had any success in achieving this kind of "streaming" compression, who could direct me at an appropriate Apache module. > > What mod_deflate did you try ? My or Apache 2.0 ? > I can comment my mod_deflate. > First, mod_deflate did not collect all output before compressing - it > compress it on the fly. But it emits compressed content in 8K block. > It's Apache's HUGE_STRING_LEN #define and it can be changed in sources. > Besides if some module flushes output then mod_deflate would flushed it too. > > mod_deflate did not have problems with chunked transfer-encoding > because it compress content before Apache start to make chunks. > mod_deflate remove Content-Length header so compressed content > would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). > > Igor Sysoev > >
Re: "Streaming" compression of output from mod_perl handler?
On Tue, 2002-02-19 at 06:11, Igor Sysoev wrote: > On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > > > if mod_deflate will receive flush request it will flush > > > deflate encoding > > > and will write compressed content to Apache buffer. But it does not > > > flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS > > > usually should send data if in OS's buffer there is more then 2K. > > > So it's probably MSIE feature as well as NC4 can not render tables > > > until it receive ''. > > > > If it was a "bug" in MSIE, it must be something specifically related to receiving >compressed content, since the same data sent uncompressed, gets rendered as they >arrive. > > > > Anyway, I just tried getting the same data using lynx, and this made it evident >that *without* mod_deflate, the data gets sent by Apache as they are ready, whereas >*with* mod_deflate, all the compressed data are sent as one big block at the end. > > I'm not sure that lynx can handle compressed response on the fly - > it uses gzip in pipe. > The best way to test it using netcat. > > > So it seems that I am still unable to get the functionality I am looking for. > > I you like to test I can make patch for mod_deflate to flush Apache. > But if major browsers can not handle compressed content on the fly > it's not valueable. I did some experiments using Ethereal to capture the IP stream between my browser and rambler.ru. On an example request, the timing was: 0.000 Initiate connection 0.245 Acknowledge connection 0.245 Request sent 0.530 Response receieved 0.540 Response continued 0.823 Response continued 0.826 Response finished It is difficult to tell from these timings whether the response was sent in steps or not. A useful test script would do something like for 50 times: sleep one second; print 1KB; -jwb
Re: "Streaming" compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > if mod_deflate will receive flush request it will flush > > deflate encoding > > and will write compressed content to Apache buffer. But it does not > > flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS > > usually should send data if in OS's buffer there is more then 2K. > > So it's probably MSIE feature as well as NC4 can not render tables > > until it receive ''. > > If it was a "bug" in MSIE, it must be something specifically related to receiving >compressed content, since the same data sent uncompressed, gets rendered as they >arrive. > > Anyway, I just tried getting the same data using lynx, and this made it evident that >*without* mod_deflate, the data gets sent by Apache as they are ready, whereas *with* >mod_deflate, all the compressed data are sent as one big block at the end. I'm not sure that lynx can handle compressed response on the fly - it uses gzip in pipe. The best way to test it using netcat. > So it seems that I am still unable to get the functionality I am looking for. I you like to test I can make patch for mod_deflate to flush Apache. But if major browsers can not handle compressed content on the fly it's not valueable. Igor Sysoev
Re: "Streaming" compression of output from mod_perl handler?
> if mod_deflate will receive flush request it will flush > deflate encoding > and will write compressed content to Apache buffer. But it does not > flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS > usually should send data if in OS's buffer there is more then 2K. > So it's probably MSIE feature as well as NC4 can not render tables > until it receive ''. If it was a "bug" in MSIE, it must be something specifically related to receiving compressed content, since the same data sent uncompressed, gets rendered as they arrive. Anyway, I just tried getting the same data using lynx, and this made it evident that *without* mod_deflate, the data gets sent by Apache as they are ready, whereas *with* mod_deflate, all the compressed data are sent as one big block at the end. So it seems that I am still unable to get the functionality I am looking for. Nicholas Oxhøj
Re: "Streaming" compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > > What mod_deflate did you try ? My or Apache 2.0 ? > Yours > > > I can comment my mod_deflate. > > First, mod_deflate did not collect all output before compressing - it > > compress it on the fly. But it emits compressed content in 8K block. > > It's Apache's HUGE_STRING_LEN #define and it can be changed > > in sources. > > Besides if some module flushes output then mod_deflate would > > flushed it too. > > > > mod_deflate did not have problems with chunked transfer-encoding > > because it compress content before Apache start to make chunks. > > mod_deflate remove Content-Length header so compressed content > > would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). > > At the moment I have made my mod_perl handler extra slow, so that it is easier to >see the effects of the different compression modules. > The handler returns approx. 430KB of uncompressed HTML. > > With no compression module, the HTML slowly trickles to the browser, and the data >gets displayed/rendered as it arrives. > With mod_deflate enabled, nothing gets displayed in the browser until all 24KB of >the gzip compressed HTML has arrived at the browser. > > Unfortunately I don't really have the tools to see when data are received at the >client side, I can only judge by the time they are displayed/rendered in the browser. >So I can't really tell if the problem is, that all the data are received in one big >batch or if they are actually received in 8K blocks, but some bug-like feature in IE >5.5 makes it unable to decode the gzipped data in a "streaming" fashion as they >arrive. Sorry, if mod_deflate will receive flush request it will flush deflate encoding and will write compressed content to Apache buffer. But it does not flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS usually should send data if in OS's buffer there is more then 2K. So it's probably MSIE feature as well as NC4 can not render tables until it receive ''. Igor Sysoev
Re: "Streaming" compression of output from mod_perl handler?
> What mod_deflate did you try ? My or Apache 2.0 ? Yours > I can comment my mod_deflate. > First, mod_deflate did not collect all output before compressing - it > compress it on the fly. But it emits compressed content in 8K block. > It's Apache's HUGE_STRING_LEN #define and it can be changed > in sources. > Besides if some module flushes output then mod_deflate would > flushed it too. > > mod_deflate did not have problems with chunked transfer-encoding > because it compress content before Apache start to make chunks. > mod_deflate remove Content-Length header so compressed content > would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). At the moment I have made my mod_perl handler extra slow, so that it is easier to see the effects of the different compression modules. The handler returns approx. 430KB of uncompressed HTML. With no compression module, the HTML slowly trickles to the browser, and the data gets displayed/rendered as it arrives. With mod_deflate enabled, nothing gets displayed in the browser until all 24KB of the gzip compressed HTML has arrived at the browser. Unfortunately I don't really have the tools to see when data are received at the client side, I can only judge by the time they are displayed/rendered in the browser. So I can't really tell if the problem is, that all the data are received in one big batch or if they are actually received in 8K blocks, but some bug-like feature in IE 5.5 makes it unable to decode the gzipped data in a "streaming" fashion as they arrive. Nicholas Oxhøj
Re: "Streaming" compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: > I am looking for an Apache module which will allow me to compress the output of my >mod_perl handler (a "native" handler, i.e. not running under Apache::Registry). But >since my handler can potentially take a long time to finish, the output has to be >compressed in a "streaming" fashion (or in small blocks) so that the browser will >start receiving data before my handler has completely finished. > > I have been experimenting with all the different Apache compression modules I have >been able to find, but have not been able to get the desired result. I have tried >Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with different >results. One I cannot get to work at all. Most work, but seem to collect all the >output before compressing it and sending it to the browser. > > There also seems to be an issue about the new HTTP/1.1 "chunked" transfer-encoding. >For instance, mod_gzip will not compress chunked output, unless you allow it to >"dechunk" it by collecting all the output and compressing it as one big block. > > So I am basically looking for anyone who has had any success in achieving this kind >of "streaming" compression, who could direct me at an appropriate Apache module. What mod_deflate did you try ? My or Apache 2.0 ? I can comment my mod_deflate. First, mod_deflate did not collect all output before compressing - it compress it on the fly. But it emits compressed content in 8K block. It's Apache's HUGE_STRING_LEN #define and it can be changed in sources. Besides if some module flushes output then mod_deflate would flushed it too. mod_deflate did not have problems with chunked transfer-encoding because it compress content before Apache start to make chunks. mod_deflate remove Content-Length header so compressed content would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). Igor Sysoev
"Streaming" compression of output from mod_perl handler?
Hi I am looking for an Apache module which will allow me to compress the output of my mod_perl handler (a "native" handler, i.e. not running under Apache::Registry). But since my handler can potentially take a long time to finish, the output has to be compressed in a "streaming" fashion (or in small blocks) so that the browser will start receiving data before my handler has completely finished. I have been experimenting with all the different Apache compression modules I have been able to find, but have not been able to get the desired result. I have tried Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with different results. One I cannot get to work at all. Most work, but seem to collect all the output before compressing it and sending it to the browser. There also seems to be an issue about the new HTTP/1.1 "chunked" transfer-encoding. For instance, mod_gzip will not compress chunked output, unless you allow it to "dechunk" it by collecting all the output and compressing it as one big block. So I am basically looking for anyone who has had any success in achieving this kind of "streaming" compression, who could direct me at an appropriate Apache module. Regards, Nicholas Oxhøj
