Hi,
I'd like to take the discussion to nginx's mailing list, you will see my
post there soon.
I don't think buffering is necessary; HTTP already knows how to handle a 200
response that was cut in the middle, in case the connection was closed
before the whole content length was transmitted or before the last chunk. A
500 response is not necessary, just to close the connection on the 200
response.
The problem we're having here is that nginx will not close the downstream
connection when its upstream connection is closed prematurely; nginx will
simply keep the downstream connection open ("keepalived") even though the
upstream has died and there's no chance this connection will complete
correctly. When gzip compression is enabled, the situation is even worse -
nginx will take the chopped response, strip away the content-length header,
and chunk-transfer the response downstream as if nothing bad happened.
- Yaniv
On Fri, Sep 30, 2011 at 10:28 AM, Evgeny Turnaev <[email protected]>wrote:
> 2011/9/30 C Anthony Risinger <[email protected]>:
> > On Thu, Sep 29, 2011 at 3:25 PM, Evgeny Turnaev <[email protected]>
> wrote:
> >> 2011/9/29 Yaniv Aknin <[email protected]>:
> >>>
> >>> I'm happy you understand me now.
> >>> The problem is if I fix it the HTTP way, it means my WSGI app has to do
> >>> things that are the webserver's job (like gzip compression).
> >>> For example, I tried sending a 1,000 byte document with correct
> >>> Content-Length from my app, via nginx with gzip compression on. If I
> kill
> >>> the connection at the 500th byte, nginx happily ignores the fact it has
> a
> >>> 500 byte document with Content-Length of 1,000 bytes, removes the
> >>> Content-Length header, does gzip and adds chunked transfer encoding.
> >>
> >> This is indeed an nginx issue. Btw did you tried latest nginx?
> >> If it not works in latest nginx then possible workaround:
> >> * write a patch for nginx
> >> * disable gzip and pass content-length provided by wsgi app directly
> >> (maybe with uwsgi_pass_header) to browser the browser will show
> >> that a page don`t loaded.. dont know if it will cache. but still this
> >> is 200 ok. if you want 500 you have to teach
> >> nginx recognize that this is incomplete response.
> >> Also wsgi application may not provide already gzipped or transfer
> >> encoded content
> >> this is described in
> >> http://www.python.org/dev/peps/pep-0333/#other-http-features
> >> and
> http://www.python.org/dev/peps/pep-0333/#handling-the-content-length-header
> >>
> >>> 2. Answering "fix your app so uwsgi will not die" is not a serious
> response
> >>> in my opinion. Maybe my app had a bug, maybe uWSGI had a bug, maybe any
> of
> >>> the libraries they use had a bug, maybe my kernel had a bug, maybe my
> system
> >>> administrator made a mistake. This is a possibility that I don't think
> you
> >>> can seriously ignore, and not necessarily something I can fix. I prefer
> a
> >>> 502 in production instead of an incorrect 200 OK anyday.
> >>> This is a serious and complicated issue, I'd be happy to meet in IRC
> and
> >>> further explain this to those interested.
> >>
> >> Sorry Yaniv, maybe i misunderstood you point a bit.
> >> But still: If you app have a bug: try/catch it. If a bug in kernel:
> >> nothing will help you really.
> >> The system administrators do really make errors :)
> >> Although i agree with you: sad that nginx doesn't support checking of
> >> body response against content-length header.
> >> Also i think this issue can be solved with a proper support of ETag by
> >> nginx - so that ETag of incomplete response and complete will differ,
> >> but this is more like workaround.
> >
> > i don't think the problem *strictly* rests with nginx, but rather with
> > the well-known semantics of the HTTP/1.0 protocol:
> >
> > http://www.w3.org/Protocols/HTTP/1.0/spec.html#BodyLength
> > "... If a Content-Length header field is present, its value in bytes
> > represents the length of the Entity-Body. Otherwise, the body length
> > is determined by the closing of the connection by the server. ...
> > Closing the connection cannot be used to indicate the end of a request
> > body, since it leaves no possibility for the server to send back a
> > response. Therefore, HTTP/1.0 requests containing an entity body must
> > include a valid Content-Length header field. If a request contains an
> > entity body and Content-Length is not specified, and the server does
> > not recognize or cannot calculate the length from other fields, then
> > the server should send a 400 (bad request) response."
> >
> > ... so in all correctness, nginx should fail with 400, but nobody
> > honors this because the front-end client-connection is really
> > HTTP/1.1, ie. keepalive/chunked (i don't think a missing
> > content-length is treated as 400 even if client is 1.0).
>
> This is not about 1.0 vs 1.1 Yaniv did send Content-length back to nginx
> and it didn't helped him because nginx does not checks length of body
> vs content-length header.
> And this wont change with http 1.1 support in nginx.
>
> > immediate issue is that nginx doesn't natively support HTTP/1.1 to
> > backends ... though i imagine this could technically be implemented by
> > uwsgi by special casing an nginx connection -- a workaround to relying
> > on a bottomless 1.0 connection. uwsgi would need to [re]implement
> > chunked encoding between the nginx module and uwsgi, out-of-band, and
> > reassemble/remove the frames before actually handing each chunk to
> > nginx, yes?
>
> Nginx already supports http1.1 in latest version (with keepalive
> support) to http fcgi and scgi backends.
> The bad thing about adding content-length header is buffering: nginx
> would have to buffer whole
> response body check the length and decide whatever it is 400 500 or 200 ok.
> You are suggesting that uwsgi should implement chunking and nginx
> should reassemble chunks:
> again if nginx will reassemble the chunks then it has to buffer
> response.(if nginx will not buffer response and stream chunks directly
> to client - and then in the middle of reseponse will find that length
> do not match - there is no way to tell cleint this is 500 or 400, 200
> ok already sent). And response body can be really big or a workload
> can be big. Also there is no reason to chunking because nginx would
> still buffer whole response.
> So this is applicable only to small responses under low load and with
> accetable buffering (not a comet like and not willing to du chunked
> response in order to load client page faster because of early load of
> js/css by browser)
>
>
> > ... sounds annoying :-) is there a way you (the app developer) could
> > append a finalizing "EOT" marker to your response? perhaps you could
> > strip the marker on 200/OK, and have nginx redirect/kill if it never
> > shows before socket close. i know nginx can do some interesting
> > matching via configuration file alone, but it might complicate/bloat
> > your setup, if it's possible at all.
> >
> > --
> >
> > C Anthony
> > _______________________________________________
> > uWSGI mailing list
> > [email protected]
> > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
> >
>
>
>
> --
> --------------------------------------------
> Турнаев Евгений Викторович
> +7 906 875 09 43
> --------------------------------------------
> _______________________________________________
> uWSGI mailing list
> [email protected]
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
