RE: Reading of input after headers sent and 100-continue.
-Original Message-
From: Graham Dumpleton [mailto:[EMAIL PROTECTED]
Sent: Tuesday, January 29, 2008 4:29 PM
To: modules-dev@httpd.apache.org
Subject: Reading of input after headers sent and 100-continue.
The HTTP output filter will send a 100 result back to a
client when the first attempt to read input occurs and an
Except header with 100-continue was received. Ie., from
http_filters.c we have:
if ((ctx-state == BODY_CHUNK ||
(ctx-state == BODY_LENGTH ctx-remaining 0))
f-r-expecting_100 f-r-proto_num = HTTP_VERSION(1,1)) {
This is from ap_http_filter(). If you look at http_core.c, you can see
that it is registered as an input filter, not an output filter. So, if
you never read from the input brigade, the 100 continue will never be
sent. I'm not sure if the module needs to just ignore the input brigade,
or actively throw it away, though.
The problem then is if only after having sent some response
content and triggering the response headers to be sent one
actually goes to read the input, then the HTTP output filter
above is still sending the 100 status response string. In
other words, the 100 response status string is appearing in
the middle of the actual response content.
Doctor, it hurts when I do this! :)
If a module is sending a response before a 100 continue has been sent,
then it shouldn't read from the input brigade, because it is going
against the HTTP spec.
My question then is, what should a handler do if it is trying
to generate response content (non buffered), before having
attempted to read any input, ie., what is the correct way to
stop Apache still sending the 100 status response for the
100-continue header? I know that setting r-expecting_100 to
0 at time that first response content is being sent will
prevent it, but is there something else that should be done
instead?
Since ap_http_filter is an input filter only, it should be enough to
just avoid reading from the input brigade. (AFAICT, anyway.)
BTW, this is partly theoretical in that have no actual code
that is doing this, but technically in systems like
mod_python or mod_wsgi where one doesn't know what the Python
application code running on top is doing, a user could
trigger this situation.
The module can provide an interface to the input and output brigades
that prevents the application from doing this. mod_wsgi is doing this
already. As I mentioned on the Web-SIG list, it is difficult to have an
uniform, automatic mechanism for doing this for all request methods, or
even a uniform way of doing it for a particular method. So, it basically
has to be left up to the handler/application.
- Brian
Re: Reading of input after headers sent and 100-continue.
On 31/01/2008, Brian Smith [EMAIL PROTECTED] wrote:
-Original Message-
From: Graham Dumpleton [mailto:[EMAIL PROTECTED]
Sent: Tuesday, January 29, 2008 4:29 PM
To: modules-dev@httpd.apache.org
Subject: Reading of input after headers sent and 100-continue.
The HTTP output filter will send a 100 result back to a
client when the first attempt to read input occurs and an
Except header with 100-continue was received. Ie., from
http_filters.c we have:
if ((ctx-state == BODY_CHUNK ||
(ctx-state == BODY_LENGTH ctx-remaining 0))
f-r-expecting_100 f-r-proto_num = HTTP_VERSION(1,1)) {
This is from ap_http_filter(). If you look at http_core.c, you can see
that it is registered as an input filter, not an output filter.
I knew what I meant, it just didn't come out right. I blame the keyboard. :-)
So, if
you never read from the input brigade, the 100 continue will never be
sent. I'm not sure if the module needs to just ignore the input brigade,
or actively throw it away, though.
The problem then is if only after having sent some response
content and triggering the response headers to be sent one
actually goes to read the input, then the HTTP output filter
above is still sending the 100 status response string. In
other words, the 100 response status string is appearing in
the middle of the actual response content.
Doctor, it hurts when I do this! :)
If a module is sending a response before a 100 continue has been sent,
then it shouldn't read from the input brigade, because it is going
against the HTTP spec.
Can you point to the specific bit of the HTTP specification which says that.
Section 8.2.3 would to me appear to have slightly conflicting statements.
In particular it says:
Because of the presence of older implementations, the protocol
allows ambiguous situations in which a client may send Expect: 100-
continue without receiving either a 417 (Expectation Failed) status
or a 100 (Continue) status. Therefore, when a client sends this header
field to an origin server (possibly via a proxy) from which it has
never seen a 100 (Continue) status, the client SHOULD NOT wait for an
indefinite period before sending the request body.
Effectively, if a 200 response came back, it seems to suggest that the
client still should send the request body, just that it 'SHOULD NOT
wait for an indefinite period'. It doesn't say explicitly for the
client that it shouldn't still send the request body if another
response code comes back.
This is what I have seen with curl as a client. If one sends back a
200 response without reading any input, curl still sends the request
content, but one does notice a slight pause as some timeout occurs
only at which point it sends the request content. In other words, curl
doesn't send it as soon as it sees the 200 response, but it does still
send it.
So technically, if the client has to still send the request content,
something could still read it. It would not be ideal that there is a
delay depending on what the client does, but would still be possible
from what I read of this section.
But then, later it says:
Upon receiving a request which includes an Expect request-header
field with the 100-continue expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response. If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request. It MUST NOT
perform the requested method if it returns a final status code.
The critical bit here I guess is:
If it responds with a final status
code, it MAY close the transport connection or it MAY continue
to read and discard the rest of the request.
This suggests that the server can discard the request body if handler
didn't try and read it before returning a response. What it means by:
It MUST NOT
perform the requested method if it returns a final status code.
I am not quite sure because if the response headers was returned by
the handler you are already in the process of performing the requested
method, so how can you not now do it.
What is also a bit worrying to me is that what might be allowed by a
handler for a request can be changed based on the presence of
100-continue, something which is out of the control of the handler and
the web server receiving the request.
Specifically, if 100-continue is not present and the client therefore
sent the request body anyway, then technically nothing to stop the
handler reading the input after the response headers have been sent.
For example, the handler may generate response headers for same
content length and only then starting reading input and returning it
as the response body.
RE: Reading of input after headers sent and 100-continue.
Graham Dumpleton wrote: Effectively, if a 200 response came back, it seems to suggest that the client still should send the request body, just that it 'SHOULD NOT wait for an indefinite period'. It doesn't say explicitly for the client that it shouldn't still send the request body if another response code comes back. This behavior is to support servers that don't understand the Expect: header. Basically, if the server responds with a 100, the client must send the request body. If the server responds with a 4xx or 5xx, the client must not send the request body. If the server responds with a 2xx or a 3xx, then the client should must send (the rest of) the request body, on the assumption that the server doesn't understand Expect:. To be completely compliant, a server should always respond with a 100 in front of a 2xx or 3xx, I guess. Thanks for clarifying that for me. I guess the rules make sense after all. So technically, if the client has to still send the request content, something could still read it. It would not be ideal that there is a delay depending on what the client does, but would still be possible from what I read of this section. You are right. To avoid confusion, you should probably force mod_wsgi to send a 100-continue in front of any 2xx or 3xx response. It MUST NOT perform the requested method if it returns a final status code. The implication is that the only time it will avoid sending a 100 is when it is sending a 4xx, and it should never perform the requested method if it already said the method failed. The only excuse for not sending a 100 is that you don't know about Expect: 100-continue. But, that can't be true if you are reading this part of the spec! If it responds with a final status code, it MAY close the transport connection or it MAY continue to read and discard the rest of the request. If the client receives a 2xx or 3xx without a 100 first, it has to send the request body (well, depending on which 3xx it is, that is not true). But, the server doesn't have to read it! But, again, the assumption is that the server will only send a response without a 100 if it is a 4xx or 5xx. It seems by what you are saying that if 100-continue is present this wouldn't be allowed, and that to ensure correct behaviour the handler would have to read at least some of the request body before sending back the response headers. You are right, I was wrong. Since ap_http_filter is an input filter only, it should be enough to just avoid reading from the input brigade. (AFAICT, anyway.) In other words block the handler from reading, potentially raise an error in the process. Except to be fair and consistent, you would have to apply the same rule even if 100-continue isn't present. Whether that would break some existing code in doing that is the concern I have, even if it is some simple test program that just echos back the request body as the response body. Technically, even if the server returns a 4xx, it can still read the request body, but it might not get anything or it might only get part of it. I guess, the change to the WSGI spec that is needed is to say that the gateway must not send the 100 continue if it has already sent some headers, and that it should send a 100 continue before any 2xx or 3xx code, which is basically what James Knight suggested (sorry James). The gateway must indicate EOF if only a partial request body was received. I don't think the gateway should be required to provide any of the partial request content on a 4xx, though. - Brian
Re: Reading of input after headers sent and 100-continue.
For those on the Python web sig who might be thinking they missed part of the conversation, you have. This is the second half of a conversation started on Apache modules-dev list about Apache 100-continue processing. If interested, you can see the first half of the conversation at: http://mail-archives.apache.org/mod_mbox/httpd-modules-dev/200801.mbox/browser Graham On 31/01/2008, Brian Smith [EMAIL PROTECTED] wrote: Graham Dumpleton wrote: Effectively, if a 200 response came back, it seems to suggest that the client still should send the request body, just that it 'SHOULD NOT wait for an indefinite period'. It doesn't say explicitly for the client that it shouldn't still send the request body if another response code comes back. This behavior is to support servers that don't understand the Expect: header. Basically, if the server responds with a 100, the client must send the request body. If the server responds with a 4xx or 5xx, the client must not send the request body. If the server responds with a 2xx or a 3xx, then the client should must send (the rest of) the request body, on the assumption that the server doesn't understand Expect:. To be completely compliant, a server should always respond with a 100 in front of a 2xx or 3xx, I guess. Thanks for clarifying that for me. I guess the rules make sense after all. So technically, if the client has to still send the request content, something could still read it. It would not be ideal that there is a delay depending on what the client does, but would still be possible from what I read of this section. You are right. To avoid confusion, you should probably force mod_wsgi to send a 100-continue in front of any 2xx or 3xx response. It MUST NOT perform the requested method if it returns a final status code. The implication is that the only time it will avoid sending a 100 is when it is sending a 4xx, and it should never perform the requested method if it already said the method failed. The only excuse for not sending a 100 is that you don't know about Expect: 100-continue. But, that can't be true if you are reading this part of the spec! If it responds with a final status code, it MAY close the transport connection or it MAY continue to read and discard the rest of the request. If the client receives a 2xx or 3xx without a 100 first, it has to send the request body (well, depending on which 3xx it is, that is not true). But, the server doesn't have to read it! But, again, the assumption is that the server will only send a response without a 100 if it is a 4xx or 5xx. It seems by what you are saying that if 100-continue is present this wouldn't be allowed, and that to ensure correct behaviour the handler would have to read at least some of the request body before sending back the response headers. You are right, I was wrong. Since ap_http_filter is an input filter only, it should be enough to just avoid reading from the input brigade. (AFAICT, anyway.) In other words block the handler from reading, potentially raise an error in the process. Except to be fair and consistent, you would have to apply the same rule even if 100-continue isn't present. Whether that would break some existing code in doing that is the concern I have, even if it is some simple test program that just echos back the request body as the response body. Technically, even if the server returns a 4xx, it can still read the request body, but it might not get anything or it might only get part of it. I guess, the change to the WSGI spec that is needed is to say that the gateway must not send the 100 continue if it has already sent some headers, and that it should send a 100 continue before any 2xx or 3xx code, which is basically what James Knight suggested (sorry James). The gateway must indicate EOF if only a partial request body was received. I don't think the gateway should be required to provide any of the partial request content on a 4xx, though. - Brian
