Unfortunately the issue persists, albeit at a lesser scale. Spooler jobs were successfully created this morning and the workers don't appear to be dying off quite as often. The config is the same as I had posted before minus the max-requests and optimize flags. Here are two excerpts from the logs:
[uWSGI DEBUG] HTTP_ACCEPT: application/vnd.rim.html,text/html,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/vnd.sun.j2me.app-descriptor,image/vnd.rim.png,image/jpeg,application/x-vnd.rim.pme.b,application/vnd.rim.ucs,image/gif;anim=1,application/vnd.rim.css;v=1,text/css;media=screen,application/vnd.wap.wmlc;q=0.9,application/vnd.wap.wmlscriptc;q=0.7,text/vnd.wap.wml;q=0.7,*/*;q=0.5 [uWSGI DEBUG] HTTP_PROFILE: http://www.blackberry.net/go/mobile/profiles/uaprof/8330m/4.5.0.rdf [uWSGI DEBUG] HTTP_VIA: BISB_3.5.1.84, 1.1 pmds116.p4.bisb5.blackberry:3128 (squid/2.7.STABLE7) [uWSGI DEBUG] HTTP_CACHE_CONTROL: max-age=259200 [uWSGI DEBUG] HTTP_CONNECTION: keep-alive [uWSGI DEBUG] called 0xb2f478ec 0xb2f4742c 1 {address space usage: 115351552 bytes/110MB} {rss usage: 23814144 bytes/22MB} [pid: 8244|app: 0|req: 870/662] 74.82.64.16 () {46 vars in 1311 bytes} [Fri Oct 14 11:22:15 2011] GET / => generated 0 bytes in 2 msecs (HTTP/1.0 301) 4 headers in 248 bytes (1 switches on core 0) Fri Oct 14 11:22:15 2011 - master sent signal 19 to worker 4 DAMN ! worker 4 (pid: 8305) died :( trying respawn ... Respawned uWSGI worker 4 (new pid: 8309) [uWSGI DEBUG] called 0xb2f18374 0xb2fc802c 4641 Fri Oct 14 11:22:21 2011 - master sent signal 45 to worker 10 DAMN ! worker 10 (pid: 8295) died :( trying respawn ... Respawned uWSGI worker 10 (new pid: 8310) [uWSGI DEBUG] called 0xb2f18374 0xb2fc802c 4641 ... [uWSGI DEBUG] called 0xb2f478ec 0xb2f4742c 1 {address space usage: 149348352 bytes/142MB} {rss usage: 24137728 bytes/23MB} [pid: 8280|app: 0|req: 719/692] 154.5.185.254 () {40 vars in 1057 bytes} [Fri Oct 14 11:24:44 2011] GET /register/dJhBARJLRDhkJbGw/31e-b71d55fcd539a7ebdb2b/ => generated 0 bytes in 1 msecs (HTTP/1.1 301) 4 headers in 299 bytes (1 switches on core 0) DAMN ! worker 10 (pid: 8333) died :( trying respawn ... Respawned uWSGI worker 10 (new pid: 8341) [uWSGI DEBUG] called 0xb2f18374 0xb2fc802c 4641 [uWSGI DEBUG] uwsgi payload size: 972 (0x3CC) modifier1: 0 modifier2: 0 [uWSGI DEBUG] PATH_INFO=/register/dJhBARJLRDhkJbGw/31e-b71d55fcd539a7ebdb2b/ [uWSGI DEBUG] SERVER_NAME=www.example.com searching for in 0xb2f478ec [uWSGI DEBUG] QUERY_STRING: [uWSGI DEBUG] REQUEST_METHOD: GET [uWSGI DEBUG] CONTENT_TYPE: [uWSGI DEBUG] CONTENT_LENGTH: [uWSGI DEBUG] REQUEST_URI: /register/dJhBARJLRDhkJbGw/31e-b71d55fcd539a7ebdb2b/ [uWSGI DEBUG] PATH_INFO: /register/dJhBARJLRDhkJbGw/31e-b71d55fcd539a7ebdb2b/ [uWSGI DEBUG] DOCUMENT_ROOT: /var/www/www.example.com [uWSGI DEBUG] SERVER_PROTOCOL: HTTP/1.1 [uWSGI DEBUG] REMOTE_ADDR: 154.5.185.254 [uWSGI DEBUG] REMOTE_PORT: 51910 [uWSGI DEBUG] SERVER_PORT: 443 [uWSGI DEBUG] SERVER_NAME: www.example.com [uWSGI DEBUG] UWSGI_SCHEME: https [uWSGI DEBUG] HTTP_ACCEPT: */* [uWSGI DEBUG] HTTP_ACCEPT_LANGUAGE: en [uWSGI DEBUG] HTTP_ACCEPT_ENCODING: gzip, deflate [uWSGI DEBUG] HTTP_COOKIE: __utmc=241079170; sessionid=3f76d18acea925f202791ea70d43fead; __utmb=241079170.2.10.1318616649; __utmz=241079170.1318616649.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); csrftoken=8d1ae47378b993aa2fb251c497800867; __utma=241079170.1269303664.1318616649.1318616649.1318616649.1 [uWSGI DEBUG] HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/4.1.3 Safari/533.19.4 [uWSGI DEBUG] HTTP_CONNECTION: keep-alive [uWSGI DEBUG] HTTP_HOST: www.example.com [uWSGI DEBUG] called 0xb2f478ec 0xb2f4742c 1 {address space usage: 115351552 bytes/110MB} {rss usage: 23818240 bytes/22MB} [pid: 8244|app: 0|req: 875/693] 154.5.185.254 () {40 vars in 972 bytes} [Fri Oct 14 11:24:44 2011] GET /register/dJhBARJLRDhkJbGw/31e-b71d55fcd539a7ebdb2b/ => generated 1250 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 258 bytes (1 switches on core 0) [uWSGI DEBUG] uwsgi payload size: 1066 (0x42A) modifier1: 0 modifier2: 0 [uWSGI DEBUG] PATH_INFO=/biz/196/223/self_visit/ [uWSGI DEBUG] SERVER_NAME=www.example.com searching for in 0xb2f478ec [uWSGI DEBUG] QUERY_STRING: [uWSGI DEBUG] REQUEST_METHOD: GET [uWSGI DEBUG] CONTENT_TYPE: [uWSGI DEBUG] CONTENT_LENGTH: [uWSGI DEBUG] REQUEST_URI: /biz/196/223/self_visit/ [uWSGI DEBUG] PATH_INFO: /biz/196/223/self_visit/ [uWSGI DEBUG] DOCUMENT_ROOT: /var/www/www.example.com [uWSGI DEBUG] SERVER_PROTOCOL: HTTP/1.1 [uWSGI DEBUG] REMOTE_ADDR: 198.228.213.97 [uWSGI DEBUG] REMOTE_PORT: 49905 [uWSGI DEBUG] SERVER_PORT: 80 [uWSGI DEBUG] SERVER_NAME: www.example.com [uWSGI DEBUG] UWSGI_SCHEME: http [uWSGI DEBUG] HTTP_HOST: www.example.com [uWSGI DEBUG] HTTP_USER_AGENT: Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Mobile/8C148 [uWSGI DEBUG] HTTP_ACCEPT: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 [uWSGI DEBUG] HTTP_REFERER: http://www.example.com/biz/196/223/self_visit/ [uWSGI DEBUG] HTTP_CACHE_CONTROL: max-age=0 [uWSGI DEBUG] HTTP_ACCEPT_LANGUAGE: en-us [uWSGI DEBUG] HTTP_ACCEPT_ENCODING: gzip, deflate [uWSGI DEBUG] HTTP_COOKIE: csrftoken=65533535a38a0fd90719a6b85537978f; sessionid=9d0a6e858a6edf4b3d07e3b2e0863cd1; __utma=241079170.59454198.1311924321.1316443452.1316448832.26; __utmz=241079170.1311924321.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __qca=P0-871702505-1311924321780 [uWSGI DEBUG] HTTP_CONNECTION: keep-alive [uWSGI DEBUG] called 0xb2f478ec 0xb2f4742c 1 {address space usage: 132333568 bytes/126MB} {rss usage: 23863296 bytes/22MB} [pid: 8155|app: 0|req: 730/691] 198.228.213.97 () {44 vars in 1066 bytes} [Fri Oct 14 11:24:46 2011] GET /biz/196/223/self_visit/ => generated 2982 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Fri Oct 14 11:24:46 2011 - master sent signal 56 to worker 10 DAMN ! worker 10 (pid: 8341) died :( trying respawn ... Respawned uWSGI worker 10 (new pid: 8342) [uWSGI DEBUG] called 0xb2f18374 0xb2fc802c 4641 Ryan- On Fri, Oct 14, 2011 at 1:32 AM, Ryan Showalter <[email protected]> wrote: > On Fri, Oct 14, 2011 at 1:15 AM, Roberto De Ioris <[email protected]> wrote: >> >> Il giorno 14/ott/2011, alle ore 10:11, Ryan Showalter ha scritto: >> >>> Hey everyone, >>> >>> I've been running into an issue lately where after my server has been >>> running for a few hours, the uWSGI master process will start killing >>> off workers for seemingly no reason. When this happens, spool jobs >>> also fail to be created, but no failures seem to occur in the logs. >>> What's weird is that this error seems to have only begun manifesting >>> itself in the last couple of weeks. I'm not sure if this is related >>> to increased traffic or what. My memory usage is not outrageous, I >>> still have plenty of physical memory and swap available. Also, CPU >>> usage is very minimal. Restarting uwsgi will fix the issue for a few >>> hours before it begins to creep up again, at first just one or two >>> workers will be killed off every few minutes, and eventually it gets >>> to the point where at least 1 worker is killed at every request. >>> >>> Below is my configuration, along with an excerpt from my log file >>> which shows several workers being killed and respawned within less >>> than a minute. I'm running uWSGI 0.9.9.2 with debug enabled (this >>> issue also happens with debug disabled). >>> >>> Any help is much appreciated! >>> >>> [program:uwsgi-www] >>> command=/opt/webapps/www.example.com/bin/uwsgi >>> --master >>> --processes 16 >>> --memory-report >>> --harakiri 7200 >>> --vacuum >>> --max-requests 500 >> >> Can you try removing --max-requests directive ? >> You should check in the logs if the continuous reloads starts after the max >> requests count has been reached >> >> >>> --optimize 2 >> >> remove even optimizations (just to be sure) > > Thanks Roberto, > > I'll let you know how it goes tomorrow after it's had a chance to > build up the error again. > >> >> >> -- >> Roberto De Ioris >> http://unbit.it >> JID: [email protected] >> >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > > Ryan- > _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
