Re: Limiting response body length
Hi list,
thanks for the replies. Looks like squid in case Content-Length
response header is missing, does it's limitation in a hard way (snip
from src/client_side.c):
} else if (clientReplyBodyTooLarge(http, http-out.offset - 4096)) {
/* 4096 is a margin for the HTTP headers included in out.offset */
comm_close(fd);
} else {
However this seems to be the only way in case we want to avoid content
buffering. mod_security also relies on Content-Length an if is not
present, output buffering (and I suppose the limitation as well) stops
(snip from apache2/apache2_io.c:output_filter):
case 0 :
/* We do not want to observe this response body
* but we need to remain attached to observe
* when it is completed so that we can run
* the RESPONSE_BODY phase.
*/
msr-of_skipping = 1;
msr-resbody_status = RESBODY_STATUS_NOT_READ;
break;
On 2/13/07, Nick Kew [EMAIL PROTECTED] wrote:
On Mon, 12 Feb 2007 23:35:24 +0100
Henrik Nordstrom [EMAIL PROTECTED] wrote:
mån 2007-02-12 klockan 21:55 + skrev Nick Kew:
Because the chunking filter is equipped to discard the chunk that
takes it over the limit, and substitute end-of-chunking.
If we do that in a new filter, we have to reinvent that wheel.
Not sure substitue end-of-chunking is a reasonable thing here. It's
an abort condition, not an EOF condition. Imho you'd better abort the
flow, that way telling the client that the request failed instead of
silently truncating the response.
How would you abort it other than by truncating it?
Don't forget, the headers are long gone.
If you don't send an end-marker, the client will
sit there waiting for more.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
--
Dziugas
Re: Limiting response body length
On 2/13/07, Dziugas Baltrunas [EMAIL PROTECTED] wrote:
Hi list,
thanks for the replies. Looks like squid in case Content-Length
response header is missing, does it's limitation in a hard way (snip
from src/client_side.c):
} else if (clientReplyBodyTooLarge(http, http-out.offset - 4096)) {
/* 4096 is a margin for the HTTP headers included in out.offset */
comm_close(fd);
} else {
However this seems to be the only way in case we want to avoid content
buffering. mod_security also relies on Content-Length an if is not
present, output buffering (and I suppose the limitation as well) stops
(snip from apache2/apache2_io.c:output_filter):
No. If there's no C-L ModSecurity will count the bytes as they arrive.
If there are too many the entire response will be blocked with 500
(and the error page sent to the client).
case 0 :
/* We do not want to observe this response body
* but we need to remain attached to observe
* when it is completed so that we can run
* the RESPONSE_BODY phase.
*/
msr-of_skipping = 1;
msr-resbody_status = RESBODY_STATUS_NOT_READ;
break;
The above happens when ModSecurity decides it is not interested in the
content (e.g. if it is an image, or some other opaque file).
On 2/13/07, Nick Kew [EMAIL PROTECTED] wrote:
On Mon, 12 Feb 2007 23:35:24 +0100
Henrik Nordstrom [EMAIL PROTECTED] wrote:
mån 2007-02-12 klockan 21:55 + skrev Nick Kew:
Because the chunking filter is equipped to discard the chunk that
takes it over the limit, and substitute end-of-chunking.
If we do that in a new filter, we have to reinvent that wheel.
Not sure substitue end-of-chunking is a reasonable thing here. It's
an abort condition, not an EOF condition. Imho you'd better abort the
flow, that way telling the client that the request failed instead of
silently truncating the response.
How would you abort it other than by truncating it?
Don't forget, the headers are long gone.
If you don't send an end-marker, the client will
sit there waiting for more.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
--
Dziugas
--
Ivan Ristic
Re: Limiting response body length
On Tue, 13 Feb 2007 11:30:32 + Ivan Ristic [EMAIL PROTECTED] wrote: No. If there's no C-L ModSecurity will count the bytes as they arrive. If there are too many the entire response will be blocked with 500 (and the error page sent to the client). That's a tradeoff you make against performance. I would consider it unacceptable to buffer entire requests or responses at a proxy. At best it's a big performance hit; at worst it's a DoS-magnet. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: Limiting response body length
On 2/13/07, Nick Kew [EMAIL PROTECTED] wrote: On Tue, 13 Feb 2007 11:30:32 + Ivan Ristic [EMAIL PROTECTED] wrote: No. If there's no C-L ModSecurity will count the bytes as they arrive. If there are too many the entire response will be blocked with 500 (and the error page sent to the client). That's a tradeoff you make against performance. Of course it's a tradeoff. Isn't everything? I would consider it unacceptable to buffer entire requests or responses at a proxy. That depends entirely on system's security requirements. Some people require the screening/prevention functionality. Some people, such as yourself, don't. It's for everyone to consider what they want, along with the implications, and make their decisions accordingly. At best it's a big performance hit; at worst it's a DoS-magnet. Don't be so dramatic :) Every single new feature added to a web server is a performance hit and a DoS magnet. And yet there's plenty of sites that moved on from static files! The ingredients matter but it's how you build it that counts. I have made it a point to document everything there is to know about ModSecurity. It's what it is. I built it because it was fun and because I could. People should make their own minds. I am fine either way. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/ -- Ivan Ristic
Re: Limiting response body length
mån 2007-02-12 klockan 12:41 +0200 skrev Dziugas Baltrunas: To illustrate, squid for this purpose has reply_body_max_size [1] parameter. Looks like it is only Content-Length response header (if any) dependent, It also terminates requests when the amount of data transferred hits the specified limit if not known ahead by content-length. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Limiting response body length
Dziugas Baltrunas [EMAIL PROTECTED] writes: Hi list, I'm wondering if there any plans to implement implement a response body length limitation inside mod_proxy? For now we have only a global LimitRequestBody and what I'm looking is an analog LimitResponseBody. In case Apache HTTP works as a reverse/forward proxy, it's usual case for a proxy to block downloading files of a certain size (huge MP3 files as an example). To illustrate, squid for this purpose has reply_body_max_size [1] parameter. Looks like it is only Content-Length response header (if any) dependent, but IMHO with Apache filtering mechanism it should be possible to cancel a fetch transaction when output byte counter reaches some boundary. I've also tried to look at exiting third party modules like mod_throttle [2] (deprecated), mod_cband [3] and mod_bw [4], but all of them seem to be tied on bandwidth limitations and not a limits of a single request. mod_security may do the trick. Thanks in advance. [1] http://www.visolve.com:81/squid/squid30/accesscontrols.php#reply_body_max_size [2] http://www.snert.com/Software/apache.html [3] http://cband.linux.pl/ [4] http://bwmod.sourceforge.net/, http://svn.apache.org/viewvc/httpd/mod_bw/ -- Dziugas Baltrunas
Re: Limiting response body length
On 2/12/07, Ding Deng [EMAIL PROTECTED] wrote: Dziugas Baltrunas [EMAIL PROTECTED] writes: Hi list, I'm wondering if there any plans to implement implement a response body length limitation inside mod_proxy? For now we have only a global LimitRequestBody and what I'm looking is an analog LimitResponseBody. In case Apache HTTP works as a reverse/forward proxy, it's usual case for a proxy to block downloading files of a certain size (huge MP3 files as an example). To illustrate, squid for this purpose has reply_body_max_size [1] parameter. Looks like it is only Content-Length response header (if any) dependent, but IMHO with Apache filtering mechanism it should be possible to cancel a fetch transaction when output byte counter reaches some boundary. I've also tried to look at exiting third party modules like mod_throttle [2] (deprecated), mod_cband [3] and mod_bw [4], but all of them seem to be tied on bandwidth limitations and not a limits of a single request. mod_security may do the trick. It would do the trick, but only for those transactions that have response buffering enabled. When buffering is not enabled limits are not enforced. Thanks in advance. [1] http://www.visolve.com:81/squid/squid30/accesscontrols.php#reply_body_max_size [2] http://www.snert.com/Software/apache.html [3] http://cband.linux.pl/ [4] http://bwmod.sourceforge.net/, http://svn.apache.org/viewvc/httpd/mod_bw/ -- Dziugas Baltrunas -- Ivan Ristic
Re: Limiting response body length
Dziugas Baltrunas schreef: Hi list, I'm wondering if there any plans to implement implement a response body length limitation inside mod_proxy? Output filter which checks the size and returns an error when it's larger than a predefined size? Joost
Re: Limiting response body length
On Mon, 12 Feb 2007 18:26:54 +0100 Joost de Heer [EMAIL PROTECTED] wrote: Dziugas Baltrunas schreef: Hi list, I'm wondering if there any plans to implement implement a response body length limitation inside mod_proxy? Output filter which checks the size and returns an error when it's larger than a predefined size? Joost 1. Where there's a Content-Length, it can be done earlier and more efficiently by just reading that - with the added bonus of being able to send a sensible errordocument to the client. 2. Where there's chunked encoding, the check would best be implemented in the chunking filter. 3. A simple count/abort filter is then a last resort. And it won't be able to tell the client what's happened, because the header has already been sent (unless it buffers the entire response, which is horribly inefficient). Not too hard to implement. Alas, I have no time to look into it myself right now. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: Limiting response body length
mån 2007-02-12 klockan 17:51 + skrev Nick Kew: 2. Where there's chunked encoding, the check would best be implemented in the chunking filter. 3. A simple count/abort filter is then a last resort. And it won't be able to tell the client what's happened, because the header has already been sent (unless it buffers the entire response, which is horribly inefficient). Why differing 2 and 3? What's the benefit of doing it in the chunking filter? Just to avoid having yet another filter in the chain or something besides that? Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Limiting response body length
On Mon, 12 Feb 2007 22:09:57 +0100 Henrik Nordstrom [EMAIL PROTECTED] wrote: mån 2007-02-12 klockan 17:51 + skrev Nick Kew: 2. Where there's chunked encoding, the check would best be implemented in the chunking filter. 3. A simple count/abort filter is then a last resort. And it won't be able to tell the client what's happened, because the header has already been sent (unless it buffers the entire response, which is horribly inefficient). Why differing 2 and 3? What's the benefit of doing it in the chunking filter? Just to avoid having yet another filter in the chain or something besides that? Because the chunking filter is equipped to discard the chunk that takes it over the limit, and substitute end-of-chunking. If we do that in a new filter, we have to reinvent that wheel. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
Re: Limiting response body length
mån 2007-02-12 klockan 21:55 + skrev Nick Kew: Because the chunking filter is equipped to discard the chunk that takes it over the limit, and substitute end-of-chunking. If we do that in a new filter, we have to reinvent that wheel. Not sure substitue end-of-chunking is a reasonable thing here. It's an abort condition, not an EOF condition. Imho you'd better abort the flow, that way telling the client that the request failed instead of silently truncating the response. But yes, the earlier you know the limit is going to be hit the better. Just not sure you will find many where the chunk size is of such size it really makes a difference, but I may be wrong.. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: Limiting response body length
On Mon, 12 Feb 2007 23:35:24 +0100 Henrik Nordstrom [EMAIL PROTECTED] wrote: mån 2007-02-12 klockan 21:55 + skrev Nick Kew: Because the chunking filter is equipped to discard the chunk that takes it over the limit, and substitute end-of-chunking. If we do that in a new filter, we have to reinvent that wheel. Not sure substitue end-of-chunking is a reasonable thing here. It's an abort condition, not an EOF condition. Imho you'd better abort the flow, that way telling the client that the request failed instead of silently truncating the response. How would you abort it other than by truncating it? Don't forget, the headers are long gone. If you don't send an end-marker, the client will sit there waiting for more. -- Nick Kew Application Development with Apache - the Apache Modules Book http://www.apachetutor.org/
