Hello! On Fri, Mar 24, 2017 at 01:31:35PM +0100, Richard Stanway wrote:
> Hello, > I recently moved our site to a new server running Linux 4.9, Debian > 8.7 64 bit with nginx 1.11.11 from the nginx repository. Our config is > straightforward - epoll, a few proxy backends and a few fastcgi > backends, a handful of vhosts, some with HTTP2, geoip module loaded. > No AIO, no threads, no timer_resolution. > > After some time, nginx worker processes are getting stuck at 100% CPU > use in what seems to be ngx_http_finalize_connection. New requests > hitting the worker are completely stalled. Eventually all nginx > workers will become stuck and the sites become unreachable. > > I'm running older versions of nginx on the same versions of Debian and > Linux at other sites without a problem, but the server giving me > problems also receives a much larger amount of traffic than the > others. Due to the traffic, the debug log gets incredibly large which > makes it difficult to isolate the error. I've posted a 1 second > excerpt of the core debug log at http://pastebin.com/hqzGzjTV during > the time that some of the workers were at 100%, however I'm not sure > this contains enough information. I'll look into enabling HTTP level > logging if necessary. > > Has anyone experienced anything similar to this or have any ideas > where to start looking to debug this? > > Thanks. > > nginx version: nginx/1.11.11 > built by gcc 4.9.2 (Debian 4.9.2-10) [...] > #0 0x000055d533ab87e8 in ngx_pfree (pool=0x55d536202fe0, > p=0x55d5361636c0) at src/core/ngx_palloc.c:282 > #1 0x000055d533af54d9 in ngx_http_set_keepalive (r=<optimized out>) > at src/http/ngx_http_request.c:3000 > #2 ngx_http_finalize_connection (r=<optimized out>) at > src/http/ngx_http_request.c:2556 > #3 0x000055d533af0d8b in ngx_http_core_content_phase > (r=0x55d536136f10, ph=0x55d537cbf210) at > src/http/ngx_http_core_module.c:1391 I think I see the problem. Please try the following patch: diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -2904,6 +2904,7 @@ ngx_http_set_keepalive(ngx_http_request_ } cl->buf = b; + cl->next = NULL; hc->busy = cl; hc->nbusy = 1; -- Maxim Dounin http://nginx.org/ _______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
