Hello! On Fri, Sep 26, 2014 at 03:45:40PM +0800, Zhang Qiang wrote:
> Hi community, > > We are using Nginx as static http server to cache/send static smal file. > The average file size is about 10k ~ 200k -, and nginx server can process > ~10000 request per second. It runs well but some times most of nginx worker > go into Stat D. and there's no way to kill them but restart system. > > Here's the call stack for kernel/user space: > > With AIO enabled > -----Kernel Space Call Stack----- > 0xffffffff8111f700 : sync_page+0x0/0x50 [kernel] > 0xffffffff8111f75e : sync_page_killable+0xe/0x40 [kernel] > 0xffffffff81529e7a : __wait_on_bit_lock+0x5a/0xc0 [kernel] > 0xffffffff8111f667 : __lock_page_killable+0x67/0x70 [kernel] > 0xffffffff81121394 : generic_file_aio_read+0x4b4/0x700 [kernel] > 0xffffffff811d58d4 : aio_rw_vect_retry+0x84/0x200 [kernel] > 0xffffffff811d7294 : aio_run_iocb+0x64/0x170 [kernel] > 0xffffffff811d86c1 : do_io_submit+0x291/0x920 [kernel] > 0xffffffff811d8d60 : sys_io_submit+0x10/0x20 [kernel] > 0xffffffff8100b288 : tracesys+0xd9/0xde [kernel] > -----User Space Call Stack----- > 0x3c362e50c9 : syscall+0x19/0x40 [/lib64/libc-2.12.so] > 0x4d2232 : ngx_linux_sendfile_chain+0xc2a/0xc2c > [/opt/soft/nginx/sbin/nginx] > 0x4d24ea : ngx_file_aio_read+0x2b6/0x528 [/opt/soft/nginx/sbin/nginx] > 0x515247 : ngx_http_file_cache_open+0xbef/0x1437 > [/opt/soft/nginx/sbin/nginx] > 0x514df6 : ngx_http_file_cache_open+0x79e/0x1437 > [/opt/soft/nginx/sbin/nginx] > 0x514abb : ngx_http_file_cache_open+0x463/0x1437 > [/opt/soft/nginx/sbin/nginx] Just a side note: it looks like there are very wrong symbols shown in the stack, likely due to optimizations used. [...] > Is anyone see these issue (pending on Stat D) before? how can I resolve it? > it seems that not the kernel issue. What makes you think so? It looks like the kernel issue from here - "unkillable" proceses just can't happen due to userland code unless there is a problem in the kernel. -- Maxim Dounin http://nginx.org/ _______________________________________________ nginx mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx
