>From what you provided, I do not see anything unexpected... Enabling threads usage with thread_pool and aio threads enables asynchronous file reading/writing. The connections_writing stat indicates... the number of connections to which the server is writing a response.
Since you are working asynchronously, if the request takes much less time to process than the response to be sent, it is normal this numbers increases, until reaching a balance point. You saw this number increasing when you reloaded the configuration -> shows the new configuration has been applied immediately You saw this number drop on restart -> restarting kills the current processes and the attached connections Nothing to see on CPU & RAM -> they are probably not the bottleneck, and one nginx strength is to have a minimal fingerprint there, compared to other Web server alternatives. Have a look at I/O wait to check if the worker processes might be having a hard time getting content they seek from the disk. Less often, it might also be a congestion on backend communication if content is being read from some. To me, it is just a matter of raton time to process request/time to serve answer. Multithreading means a worker might be answering several connections at the same time. If I am right about nginx' design, events might be processed in any order, but each worker still only does one thing at a time and onky switch task when the current one is finished/waiting on something else. Multithreading changes the whole behavior. Hope to help, --- *B. R.* On Fri, Dec 18, 2015 at 6:55 PM, Marcelo MD <[email protected]> wrote: > Hi, > > Recently we added a 'thread_pool' directive to our main configuration. A > few hours later we saw a huge increase in the connections_writing stat as > reported by stub_status module. This number reached +- 3800 and is stuck > there since. The server in question is operating normally, but this is very > strange. > > Any hints on what this could be? > > > Some info: > > - Here is a graph of the stats reported, for a server with thread_pool and > another without: http://imgur.com/a/lF2EL > > - I don`t have older data anymore, but the jump from <100 to +- 3800 > connections_writing happened in two sharp jumps. The first one following a > reload; > > - The machines' hardware and software are identical except for the > thread_pool directive in their nginx.conf. They live in two different data > centers; > > - Both machines are performing normally. Nothing unusual in CPU or RAM > usage. Nginx performance is about the same. > > - Reloading Nginx with 'nginx -s reload' does nothing. Restarting the > process brings connections_writing down. > > > Debug stuff: > > mallmann# uname -a > Linux xxx 3.8.13-98.5.2.el6uek.x86_64 #2 SMP Tue Nov 3 18:32:04 PST 2015 > x86_64 x86_64 x86_64 GNU/Linux > > mallmann# nginx -V > nginx version: nginx/1.8.0 > built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) > built with OpenSSL 1.0.1e-fips 11 Feb 2013 > TLS SNI support enabled > configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx > --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log > --http-log-path=/var/log/nginx/access.log > --http-client-body-temp-path=/var/lib/nginx/tmp/client_body > --http-proxy-temp-path=/var/lib/nginx/tmp/proxy > --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi > --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi > --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid > --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-ipv6 > --with-http_ssl_module --with-http_realip_module > --with-http_addition_module --with-http_xslt_module > --with-http_image_filter_module --with-http_geoip_module > --with-http_sub_module --with-http_flv_module --with-http_mp4_module > --with-http_gunzip_module --with-http_gzip_static_module > --with-http_random_index_module --with-http_secure_link_module > --with-http_degradation_module --with-http_stub_status_module > --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre > --with-google_perftools_module > --add-module=/builddir/build/BUILD/nginx-1.8.0/headers-more-nginx-module-0.25 > --add-module=/builddir/build/BUILD/nginx-1.8.0/ngx_http_bytes_filter_module > --add-module=/builddir/build/BUILD/nginx-1.8.0/echo-nginx-module-0.55 > --with-threads --with-debug --with-cc-opt='-O2 -g -pipe -Wall > -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector > --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E' > > > Affected server: > > mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr > 4172 REG > 2140 IPv4 > 100 unix > 30 CHR > 20 DIR > 20 0000 > 3 sock > 1 TYPE > mallmann# curl http://127.0.0.1/status > Active connections: 5924 > server accepts handled requests > 5864099 5864099 15527178 > Reading: 0 Writing: 3883 Waiting: 2040 > > > Normal server: > > mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr > 4454 REG > 1967 IPv4 > 100 unix > 30 CHR > 20 DIR > 20 0000 > 1 unknown > 1 TYPE > 1 sock > mallmann# curl http://127.0.0.1/status > Active connections: 2096 > server accepts handled requests > 1136132 1136132 3464904 > Reading: 0 Writing: 107 Waiting: 1989 > > -- > Marcelo Mallmann Dias > > _______________________________________________ > nginx mailing list > [email protected] > http://mailman.nginx.org/mailman/listinfo/nginx >
_______________________________________________ nginx mailing list [email protected] http://mailman.nginx.org/mailman/listinfo/nginx
