Cool. Thanks! Initial testing looks like this fixed it.
--Will On Thu, Nov 21, 2013 at 3:34 AM, Maxim Dounin <[email protected]> wrote: > Hello! > > On Wed, Nov 20, 2013 at 06:03:11PM -0800, Will Pugh wrote: > > > Hi folks, > > > > We are using Nginx for SSL termination, and then it proxies to an ATS or > > Haproxy server depending on our environment. > > > > We're running into a problem where every now and then, Nginx closes a > > connection due to a timeout. When investigating, it looks like the > > connections that are being timed-out are not being forwarded to the > backend > > service. The scenario when we were able to best reproduce this is one > > where one of our Java client was running about 100 REST requests that > were > > fairly similar. I've attached files that contain both the tcpdump from > the > > client side as well as the debug log on the nginx side. > > > > I tried comparing a successful and unsuccessful request next to each > > other. From the client side, it looks like the messages back and forth > > look very consistent. On the nginx side, the first difference seems to > > happen when reading in the Http Request. The requests that fail, all > seem > > to do a partial read: > > [...] > > I think I see what happens here: > > - Due to a partial read the c->read->ready flag is reset. > > - While processing request headers rest of the body becomes available in a > socket buffer. > > - The ngx_http_do_read_client_request_body() function calls c->recv(), > ngx_ssl_recv() in case of SSL, and rest of the data is read from the > kernel to OpenSSL buffers. The c->recv() is called with a limited > buffer space though (in the debug log provided, only 16 bytes - as this > happens during reading http chunk header), and only part of the data > becomes available to nginx. > > - As c->read->ready isn't set, ngx_http_do_read_client_request_body() arms > a read event and returns to the event loop. > > - The socket buffer is empty, so no further events are reported by > the kernel till timeout. > > Please try the following patch: > > --- a/src/event/ngx_event_openssl.c > +++ b/src/event/ngx_event_openssl.c > @@ -1025,6 +1025,7 @@ ngx_ssl_recv(ngx_connection_t *c, u_char > size -= n; > > if (size == 0) { > + c->read->ready = 1; > return bytes; > } > > @@ -1034,6 +1035,10 @@ ngx_ssl_recv(ngx_connection_t *c, u_char > } > > if (bytes) { > + if (c->ssl->last != NGX_AGAIN) { > + c->read->ready = 1; > + } > + > return bytes; > } > > -- > Maxim Dounin > http://nginx.org/en/donation.html > > _______________________________________________ > nginx mailing list > [email protected] > http://mailman.nginx.org/mailman/listinfo/nginx >
_______________________________________________ nginx mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx
