Hi Robert, Finally, I've solved the issue. Unfortunately, neither of the options you mentioned were set in my nginx config. Here's what I have set for the uWSGI settings:
uwsgi_pass unix:///tmp/example.sock; uwsgi_param HTTP_X_REAL_IP $http_x_forwarded_for; uwsgi_param HTTP_X_FORWARDED_FOR $proxy_add_x_forwarded_for; include uwsgi_params; I tried explicitly disabling uwsgi_intercept_errors and tried a few different options for uwsgi_next_upstream (incluidng 'off') but nothing changed. There are no redirects or rewrites in the Nginx config and I removed all timeout settings. However, I did a test and accessed the app via the standalone method (outside Facebook, direct to the Nginx/uWSGI server) and as I said before, it returns correctly. However, this request is a GET request, whereas the Facebook request is a POST request. I went back and had a look at this page: http://forum.nginx.org/read.php?2,196581 again and I'm pretty sure the issue detailed in this post is the cause of my problem. I tried adding the --pep333-input directive to the uWSGI Upstart script for my app, and lo and behold, it worked! The annoying thing is that I tried that previously when I first came across that Nginx forum post about the POST < 4096 bytes issue and it didn't work, but I must have not restart the server properly or something like that. In any case, the problem is solved now but I thought I'd detail my troubleshooting in case it helps anyone down the road. Thanks again for your fast responses to my questions! Kind regards, Guy On Mon, Jan 30, 2012 at 3:00 AM, <[email protected]> wrote: > Send uWSGI mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of uWSGI digest..." > > > Today's Topics: > > 1. Re: Odd Django app 503 behaviour (Roberto De Ioris) > 2. Re: requests showing up in uwsgi log, but not in nginx log > (Roberto De Ioris) > 3. Re: static file response time vs memory usage (Roberto De Ioris) > 4. Re: "self-extracting" uwsgi bundle (Roberto De Ioris) > 5. Re: static file response time vs memory usage (?ukasz Mierzwa) > 6. Re: Mac OS and Spooler (Roberto De Ioris) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 29 Jan 2012 19:54:55 +0100 (CET) > From: "Roberto De Ioris" <[email protected]> > To: "uWSGI developers and users list" <[email protected]> > Subject: Re: [uWSGI] Odd Django app 503 behaviour > Message-ID: > <[email protected]> > Content-Type: text/plain;charset=iso-8859-1 > > >> Ok, one more post: I've done a bit more troubleshooting and I've found >> that when I put uWSGI in http mode and stop Nginx from running, I get >> the correct response while accessing the app through facebook. So, it >> looks like the problem is Nginx or uWSGI closing the connection before >> the response is read. However, the question that still remains is why >> is it doing this? Any ideas? > > check if there is some "uwsgi_intercept_*" options in nginx enabled or > some "uwsgi_next" triggering a fallback in nginx itself. > > Take in account that uWSGI makes no difference between normal requests or > nginx internal redirect. Maybe the 503 is genrated by a nginx internal > redirect. > > > -- > Roberto De Ioris > http://unbit.it > > > ------------------------------ > > Message: 2 > Date: Sun, 29 Jan 2012 20:04:39 +0100 (CET) > From: "Roberto De Ioris" <[email protected]> > To: "uWSGI developers and users list" <[email protected]> > Subject: Re: [uWSGI] requests showing up in uwsgi log, but not in > nginx log > Message-ID: > <[email protected]> > Content-Type: text/plain;charset=iso-8859-1 > > >> Hi everybody, >> >> I'm having following setup: >> >> nginx <-unix_socket-> uwsgi -> django >> >> What I noticed is, that some requests of the uwsgi.log file do NOT show >> up in nginx' access_log file. >> >> >> Example: >> in uwsgi.log I can see: >> [pid: 1205|app: 0|req: 537/1453] xx.xx.xx.33 () {40 vars in 1794 bytes} >> [Thu Jan 26 18:42:31 2012] POST /my_url/run?var=123 => generated 129 >> bytes in 94 msecs (HTTP/1.0 200) 3 headers in 174 bytes (1 switches on >> core 0) >> >> However I don't see any corresponding request in nginx log. >> >> This is driving me rather crazy while trying to debug some issues. >> >> What could be the reason? > > > Check if they are not nginx internal requests, generated by a main one. > > I have seen you have posted the same question to the nginx list, feel free > to redirect here the official answer (if any) > > >> >> >> >> >> My uwsgi config file: >> -------------------------- >> <uwsgi> >> <uid>usr</uid> >> <gid>grp</gid> >> <socket>/var/sock/nginx.sock</socket> >> <workers>4</workers> >> <master /> >> <pidfile>/var/run/uwsgi/uwsgi.pid</pidfile> >> <daemonize>/var/log/uwsgi/uwsgi.log</daemonize> >> <pp>/www_basedir</pp> >> <module>my.module</module> >> </uwsgi> >> >> My nginx.conf >> #-------------- >> worker_processes 2; >> events { >> worker_connections 1024; >> } >> http { >> include mime.types; >> default_type application/octet-stream; >> >> sendfile on; >> keepalive_timeout 65; >> >> include sites-enabled/*; >> >> uwsgi_temp_path /etc/uwsgi/; >> >> } >> >> # sites_enabled/ssl; >> # --------------------------------------- >> The probably relevant nginx config lines: >> upstream django { >> ip_hash; >> server unix:/var/sock/nginx.sock; >> } >> >> >> server { >> listen 443; >> server_name myserver; >> access_log /var/log/nginx/access.log; >> error_log /var/log/nginx/error.log; >> >> ssl on; >> # cert info . . . . >> ssl_verify_client optional; >> ssl_session_timeout 5m; >> # cipher info >> >> location ^~ /static { >> root /my/static; >> index index.html index.htm; >> } >> >> location ^~ /registration { >> uwsgi_pass django; >> uwsgi_param UWSGI_SCHEME https; >> include uwsgi_params; >> } >> >> } >> >> >> >> >> >> >> >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > > > -- > Roberto De Ioris > http://unbit.it > > > ------------------------------ > > Message: 3 > Date: Sun, 29 Jan 2012 20:06:24 +0100 (CET) > From: "Roberto De Ioris" <[email protected]> > To: "uWSGI developers and users list" <[email protected]> > Subject: Re: [uWSGI] static file response time vs memory usage > Message-ID: > <[email protected]> > Content-Type: text/plain;charset=iso-8859-1 > > >> Hi, >> >> after adding avg_rt and rss_size fields to carbon graphs I've noticed >> funny >> thing: avg_rt looks is related to memory usage (or looks like it is), when >> rss_size grows avg_rt also grows, look at the graphs below. >> >> average response time: >> http://imageshack.us/photo/my-images/11/avgrt.png/ >> >> memory usage: >> http://imageshack.us/photo/my-images/831/rsssize.png/ >> >> Maybe: >> a) stats are wrong, response times are constants, only the time needed for >> metrics gathering grows (?) or other issue related to measuring avg_rt >> b) when memory grows uWSGI has more pages to crawl when serving request >> >> App used is just serving static files with --static-index, I use it to >> test my >> whole setup in real life conditions. I've disabled KSM in vassal config >> and it >> had no effect. The avg_rt values are small and so it is likly that this is >> measurment error but the behaviour is very stable and happens always after >> restarting vassal. Version used is 1.0.2.1 with carbon patch from #76. >> I do not report this as an issue that needs resolving, it's only >> observation. > > > Have you checked if it happens after a full reload ? > > > -- > Roberto De Ioris > http://unbit.it > > > ------------------------------ > > Message: 4 > Date: Sun, 29 Jan 2012 20:17:11 +0100 (CET) > From: "Roberto De Ioris" <[email protected]> > To: "uWSGI developers and users list" <[email protected]> > Subject: Re: [uWSGI] "self-extracting" uwsgi bundle > Message-ID: > <[email protected]> > Content-Type: text/plain;charset=iso-8859-1 > > >> On Sun, Sep 18, 2011 at 12:40 AM, Roberto De Ioris <[email protected]> >> wrote: >>> >>> >> ok, i knew i couldn't return to python from php, i mainly wanted to >> know which uWSGI children would remain in a python image for reception >> of signals/etc ... seems spooler/mules/master are the main ones. > > exactly > >> >> i'll have to think about this a bit more ... since PHP SAPI support >> was added recently i may no longer need worker-exec. the wiki >> specifically said "controlling a pool of fast-cgi PHP processes", and >> i wanted to know *how* :-) > > > re-compiling php each time is becaming a real pita, i am investigating if > libphp (named embed-sapi in php sources) could be used instead. (this > should make life easier for packagers too) > >> >> this [part 8] does work, but has anything changed here since >> 2011-18-09? i'd still like to make use of the php-fpm fastcgi server, >> even if i also use some via uWSGI SAPI, and i'd rather not proxy thru >> a python handler ... >> >> would --protocol fastcgi do the trick? any way to speak to a fastcgi >> process directly? > > > i have stopped pushing by about 4 days as i am focusing on a new way to > manage options (instead of the current > 3-hook-based|give-unique-id-to-each-one|decouple-your-code nightmare) and > the unification of the http and fastrouter plugins. Once they are ported > to the new api, we can directly add fastcgi (and obviously http) support > directly to the fastrouter. > > Regarding uwsgi.fcgi() i really do not know how it is working now as i > have no test-suite for it and i have no more fastcgi-based apps. > > >> >>> >>> a 'blacklist = all' will create a "frozen" binary. >> >> this is something i'd really like ... the ability to *disable* runtime >> argument processing *completely* (or partially as you suggest) ... >> uWSGI becomes a transparent container for my app, any arguments passed >> to uWSGI are simply passed to my master process normally via >> `sys.argv`. >> >> ... basically i'd love a way to: >> >> a) embed options at compile time (already possible via embedded_config, >> IIRC) >> b) disable *all* forms of runtime argument processing >> c) populate sys.argv with CLI args, if any, passed at runtime >> >> ... or equivalent. this makes uWSGI behave like the normal python >> interpreter (albeit 100x awesom-er); does all that makes sense? wrap >> my application in a hard super-shell, and pretend to be a little more >> than an embedded interpreter. > > as soon as the new option-subsystem will be ready, > masking/marking/blocking (and 'adding' obviously) options should became > really easy. > > > > -- > Roberto De Ioris > http://unbit.it > > > ------------------------------ > > Message: 5 > Date: Sun, 29 Jan 2012 20:21:19 +0100 > From: ?ukasz Mierzwa <[email protected]> > To: [email protected] > Cc: Roberto De Ioris <[email protected]> > Subject: Re: [uWSGI] static file response time vs memory usage > Message-ID: <1683429.IkvI6adQsQ@szmacer> > Content-Type: text/plain; charset="utf-8" > > Dnia niedziela, 29 stycznia 2012 20:06:24 Roberto De Ioris pisze: >> > Hi, >> > >> > after adding avg_rt and rss_size fields to carbon graphs I've noticed >> > funny >> > thing: avg_rt looks is related to memory usage (or looks like it is), when >> > rss_size grows avg_rt also grows, look at the graphs below. >> > >> > average response time: >> > http://imageshack.us/photo/my-images/11/avgrt.png/ >> > >> > memory usage: >> > http://imageshack.us/photo/my-images/831/rsssize.png/ >> > >> > Maybe: >> > a) stats are wrong, response times are constants, only the time needed for >> > metrics gathering grows (?) or other issue related to measuring avg_rt >> > b) when memory grows uWSGI has more pages to crawl when serving request >> > >> > App used is just serving static files with --static-index, I use it to >> > test my >> > whole setup in real life conditions. I've disabled KSM in vassal config >> > and it >> > had no effect. The avg_rt values are small and so it is likly that this is >> > measurment error but the behaviour is very stable and happens always after >> > restarting vassal. Version used is 1.0.2.1 with carbon patch from #76. >> > I do not report this as an issue that needs resolving, it's only >> > observation. >> >> Have you checked if it happens after a full reload ? > > It was caused by large number of workers and low request rate. I have 16 > workers and up to 20 requests/second and I've noticed that most requests are > handled by 2-3 workers, so the rest had just only few requests served. So > after starting workers I had 13x0ms + 3x0.10ms and the total average was very > low. After some time when other workres had also some requests served, they > got more realistic average response times in statistics output. After changing > the number of processes to 1 graphs looks normal, worker shows realistic > avg_rt times immediately after start. It's just something one needs to keeps > in mind when looking at carbon graphs. > > ?ukasz Mierzwa > > > ------------------------------ > > Message: 6 > Date: Sun, 29 Jan 2012 20:30:45 +0100 (CET) > From: "Roberto De Ioris" <[email protected]> > To: "uWSGI developers and users list" <[email protected]> > Subject: Re: [uWSGI] Mac OS and Spooler > Message-ID: > <[email protected]> > Content-Type: text/plain;charset=iso-8859-1 > > >> Ok, now that I am actually at my machine I get the following lines >> repeated many times in my log (with a different pid each time -- this >> is before forking in the uwsgi_example_spooler function): >> *** start log snip **** >> I am the spooler -- my pid is39320 >> This is very bad!!! >> Return from execve(): -1 >> Err (2): No such file or directoryunable to find the spooler function, >> have you loaded it into the spooler process ? >> kevent(): Bad file descriptor [event.c line 446] >> [spooler] managing request >> uwsgi_spoolfile_on_x-134-84-172-44.hsci.umn.edu_39280_1_0_1327518821_426577 >> ... >> *** end log snip **** >> >> After some limit is reached (probably max num processes), fork will >> fail and the following is printed to the log >> *** start log snip *** >> I am the spooler -- my pid is39498 >> Blargh -- fork() failed!] >> [spooler] done with task >> uwsgi_spoolfile_on_x-134-84-172-44.hsci.umn.edu_39280_1_0_1327518821_426577 >> after 0 seconds >> *** end log snip *** >> >> After fork fails, I then get spammed with a seemingly infinite number >> of lines of: >> "kevent(): Bad file descriptor [event.c line 446]" >> >> Oh, oops... near the top of the log is this >> *** start log snip *** >> [spooler] written 437 bytes to file >> /Users/jvanvoorst/code/drugsite-dev/myspool/uwsgi_spoolfile_on_x-134-84-172-44.hsci.umn.edu_39280_1_0_1327518821_426577 >> kevent(): Interrupted system call [event.c line 446] >> i am the DrugSite Search plugin after request function >> [spooler] managing request >> uwsgi_spoolfile_on_x-134-84-172-44.hsci.umn.edu_39280_1_0_1327518821_426577 >> ... >> *** end log snip *** >> >> First, I have little experience with fork and execve. What is the >> proper way to exit if execve fails (is calling exit()) sufficient)? >> >> Second, given the trouble others have had with kqueue/poll/select on >> Mac OS, it may not be useful to even try to fix such an issue as we >> don't know what apple will break in the next releases of Mac OS. Or >> maybe the problem is my code sucks in this case .... >> > > Even if i have developed a good part of uWSGI on a Mac, spooler is one of > those things always present in my production systems, so i directly use > them as a "final test base" before releases. > > Honestly i have been beaten by OSX kqueue so much time that i am bit tired > (fortunately other BSDs have a rock solid implementation) of adding #ifdef > all over the place. Maybe it will be an easy fix, i will check it in the > next few days. > > -- > Roberto De Ioris > http://unbit.it > > > ------------------------------ > > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > > End of uWSGI Digest, Vol 28, Issue 39 > ************************************* -- Guy Knights Systems Administrator Eastside Games www.eastsidegamestudio.com [email protected] _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
