HI Roger, Would it help me to setup two separate programs? One on 8001 and one on 8002 or something? And then reference both in nginx? Or is it irrelevant?
I can't figure out what to do, but I have to restart my prog every 15 minutes or I start getting problems. Best, Jacob On Thu, Apr 16, 2009 at 5:12 AM, Roger Hoover <[email protected]> wrote: > Hi Jacob, > > If the processes have exited, you should see that in the supervisord.log > (BTW, you can write a simple event listener to send you email alerts if > processes die). If requests are timing out, it may not be that they have > exited but just that they are taking too long to respond, possibly running > super long DB queries. You should be able to run the ps command to verify > if they process have died or are just not responding. > > In any case, this behavior is entirely controlled by the FastCGI > implementation you are running and not related to supervisord. All that > supervisord does it fork the process with file descriptor 0 pointing to a > socket to use for FastCGI. It then calls exec() and the process is > completely taken over by the FastCGI implementation. > > Roger > > On Wed, Apr 15, 2009 at 11:07 AM, Jacob Singh <[email protected]> wrote: >> >> Hi Roger, >> >> I'm actually finding a separate issue which is that my procs >> mysteriously start dying if I leave them up for too long... It's quite >> weird, no errors, nothing in the log to suggest they have exited, >> however after 20-30 minutes of load, I start getting these: >> >> 2009/04/15 09:34:15 [error] 18840#0: *127695 upstream timed out (110: >> Connection timed out) while reading response header from upstream, >> client: 174.129.xxx.xxx, server: xxxxxxxx, request: "GET /xxxxxxxxxx >> HTTP/1.0", upstream: "fastcgi://127.0.0.1:8080", host: "xxx.com" >> >> Do you have any idea why this is? Everything works fine, supervisord >> doesn't report any issues, nor is there anything in the stderr or >> stdout logs, and all of the sudden, I'm getting thousands of 504 >> errors (get about 5 req/sec). >> >> This one is really killing me. I hate having a cron job run: >> supervisorctl restart all every 15 minutes, and that also adds some >> 504s to the mix :( >> >> Best, >> Jacob >> >> Thanks! >> Jacob >> >> >> >> >> On Wed, Apr 15, 2009 at 10:51 PM, Roger Hoover <[email protected]> >> wrote: >> > Hi Jacob, >> > >> > If you use supervisorctl to restart just the gate processes (and not a >> > full >> > shutdown of supervisrod), requests should not get dropped during the >> > restart. The reason is that the fastcgi socket will still be open. >> > There >> > will be no processes accepting connections on it temporarily but the >> > connection requests will queue up in the OS until restart is successful >> > and >> > the new gate processes are accepting connections again. >> > >> > The only potential issue I see is if the restart takes too long and >> > either >> > the inbound connection queue buffer limit is reached or nginx >> > mod_fastcgi >> > times out the request for taking too long. >> > >> > If stopping and starting all the gate processes in one command takes to >> > long, you can restart them one by one with supervisorctl. >> > >> > Hope that helps, >> > >> > Roger >> > >> > On Wed, Apr 15, 2009 at 5:50 AM, Jacob Singh <[email protected]> >> > wrote: >> >> >> >> Hi Roger, >> >> >> >> I'm having some issues since going into production. >> >> >> >> Pardon my ignorance, but... >> >> >> >> I've got the following config: >> >> >> >> [fcgi-program:gate] >> >> socket=tcp://127.0.0.1:8080 ; We reference this later in nginx >> >> command = /usr/local/mything/bin/gate2.py >> >> environment=PYTHON_EGG_CACHE=/tmp/.pythoneggs,LOG_LEVEL=WARNING >> >> #warning >> >> numprocs=10 >> >> process_name=gate_%(process_num)s >> >> >> >> I then use: >> >> >> >> fcgi_pass 127.0.0.1:8080 in nginx to pass the requests in. It works >> >> fine, but then, after a while, I started getting timeout errors, so I >> >> added a cronjob to restart them every hour. >> >> >> >> This works okay, but I really don't want to restart them all together, >> >> because then requests are dropped. >> >> >> >> Best, >> >> Jacob >> >> >> >> >> >> On Wed, Apr 8, 2009 at 1:11 AM, Roger Hoover <[email protected]> >> >> wrote: >> >> > Sweet. Glad it worked for you. A release of supervisor should be >> >> > coming >> >> > soon. >> >> > >> >> > >> >> > >> >> > http://www.mail-archive.com/[email protected]/msg00144.html >> >> > >> >> > On Mon, Apr 6, 2009 at 10:46 PM, Jacob Singh <[email protected]> >> >> > wrote: >> >> >> >> >> >> Nevermind, I got it figured out. >> >> >> >> >> >> Thanks! This is awseome. I hope it gets into a release because my >> >> >> company is wary of using something that hasn't had a release for a >> >> >> year. >> >> >> >> >> >> On Tue, Apr 7, 2009 at 11:02 AM, Jacob Singh <[email protected]> >> >> >> wrote: >> >> >> > Hmm... >> >> >> > >> >> >> > okay, I've got it making the request through nginx, however, the >> >> >> > environ variable is empty in my wsgi script. The same script >> >> >> > works >> >> >> > fine when I create my own named sockets and add them to an nginx >> >> >> > upstream... >> >> >> > >> >> >> > Not sure how to proceed on that. >> >> >> > >> >> >> > Best, >> >> >> > Jacob >> >> >> > >> >> >> > On Mon, Apr 6, 2009 at 9:46 PM, Roger Hoover >> >> >> > <[email protected]> >> >> >> > wrote: >> >> >> >> Hi Jacob, >> >> >> >> >> >> >> >> Your configuration has the FastCGI process listening on >> >> >> >> 127.0.0.1:1212 >> >> >> >> so >> >> >> >> that socket is expecting the client to speak FCGI. If you use >> >> >> >> curl >> >> >> >> to >> >> >> >> send >> >> >> >> an HTTP request, it won't understand the request. You need to >> >> >> >> configure a >> >> >> >> web server such as nginx that will proxy HTTP requests over >> >> >> >> FastCGI. >> >> >> >> Nginx >> >> >> >> will need to run listen on another socket (say 5000) and proxy >> >> >> >> requests >> >> >> >> to >> >> >> >> your FastCGI processes listening on 127.0.0.1:1212. >> >> >> >> >> >> >> >> Hope that helps, >> >> >> >> >> >> >> >> Roger >> >> >> >> >> >> >> >> On Sun, Apr 5, 2009 at 10:28 PM, Jacob Singh >> >> >> >> <[email protected]> >> >> >> >> wrote: >> >> >> >>> >> >> >> >>> Hi folks! >> >> >> >>> >> >> >> >>> I just found out about this project from: >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> http://just-another.net/2009/01/18/byteflowdjangosupervisordnginx-win/#comments >> >> >> >>> >> >> >> >>> I've been trying to accomplish the same goal, but not using >> >> >> >>> django. >> >> >> >>> >> >> >> >>> It all *kinda* works, but when I try to curl my fcgi program, I >> >> >> >>> get >> >> >> >>> nada, and it just hangs forever with no logs... don't know where >> >> >> >>> to >> >> >> >>> start. I'm using trunk. >> >> >> >>> >> >> >> >>> >> >> >> >>> Server info: >> >> >> >>> Python 2.4.3 (#1, Mar 14 2007, 18:51:08) >> >> >> >>> [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2 >> >> >> >>> >> >> >> >>> >> >> >> >>> Here's my config (relevant bits): >> >> >> >>> -------------------------------------------------- >> >> >> >>> [supervisord] >> >> >> >>> logfile=/tmp/supervisord.log ; (main log file;default >> >> >> >>> $CWD/supervisord.log) >> >> >> >>> logfile_maxbytes=50MB ; (max main logfile bytes b4 >> >> >> >>> rotation;default >> >> >> >>> 50MB) >> >> >> >>> logfile_backups=10 ; (num of main logfile rotation >> >> >> >>> backups;default 10) >> >> >> >>> loglevel=debug ; (log level;default info; others: >> >> >> >>> debug,warn,trace) >> >> >> >>> pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default >> >> >> >>> supervisord.pid) >> >> >> >>> nodaemon=true ; (start in foreground if true;default >> >> >> >>> false) >> >> >> >>> minfds=1024 ; (min. avail startup file >> >> >> >>> descriptors;default >> >> >> >>> 1024) >> >> >> >>> minprocs=200 ; (min. avail process >> >> >> >>> descriptors;default >> >> >> >>> 200) >> >> >> >>> ;umask=022 ; (process file creation >> >> >> >>> umask;default >> >> >> >>> 022) >> >> >> >>> user=nobody ; (default is current user, required >> >> >> >>> if >> >> >> >>> root) >> >> >> >>> >> >> >> >>> ; Production setup >> >> >> >>> [fcgi-program:gate] >> >> >> >>> socket=tcp://127.0.0.1:1212 ; We reference this later in nginx >> >> >> >>> #command = /usr/local/solrflare/bin/gate.py ; Calls the above >> >> >> >>> code >> >> >> >>> command = /tmp/new.py >> >> >> >>> environment=PYTHON_EGG_CACHE=/tmp ; Setup needed environment >> >> >> >>> >> >> >> >>> >> >> >> >>> And here is new.py: >> >> >> >>> ---------------------------------------------- >> >> >> >>> >> >> >> >>> #!/usr/bin/python >> >> >> >>> from flup.server.fcgi import WSGIServer >> >> >> >>> import time, os, sys >> >> >> >>> >> >> >> >>> open('/tmp/new.log','a').write('something') >> >> >> >>> def app(environ, start_response): >> >> >> >>> open('/tmp/new.log','a').write('else') >> >> >> >>> status = "200 OK" >> >> >> >>> response_headers = [('Content-type', 'text/plain')] >> >> >> >>> start_response(status, response_headers) >> >> >> >>> return ['LOALALA\n'] >> >> >> >>> WSGIServer(app).run() >> >> >> >>> >> >> >> >>> >> >> >> >>> My Log: >> >> >> >>> ------------------------------------------------- >> >> >> >>> [r...@balancer:/tmp] supervisord >> >> >> >>> 2009-04-06 01:19:01,308 CRIT Set uid to user 99 >> >> >> >>> 2009-04-06 01:19:01,500 INFO RPC interface 'supervisor' >> >> >> >>> initialized >> >> >> >>> 2009-04-06 01:19:01,501 INFO RPC interface 'supervisor' >> >> >> >>> initialized >> >> >> >>> 2009-04-06 01:19:01,501 INFO supervisord started with pid 5886 >> >> >> >>> 2009-04-06 01:19:02,499 DEBG fd 8 closed, stopped monitoring >> >> >> >>> <PInputDispatcher at -1216741876 for <Subprocess at -1216915476 >> >> >> >>> with >> >> >> >>> name gate in state STARTING> (stdin)> >> >> >> >>> 2009-04-06 01:19:02,510 INFO spawned: 'gate' with pid 5888 >> >> >> >>> 2009-04-06 01:19:03,508 INFO success: gate entered RUNNING >> >> >> >>> state, >> >> >> >>> process has stayed up for > than 1 seconds (startsecs) >> >> >> >>> >> >> >> >>> >> >> >> >>> curl localhost:1212 >> >> >> >>> Just sits there forever... >> >> >> >>> >> >> >> >>> >> >> >> >>> Help!? >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Jacob >> >> >> >>> >> >> >> >>> >> >> >> >>> -- >> >> >> >>> >> >> >> >>> +1 510 277-0891 (o) >> >> >> >>> +91 9999 33 7458 (m) >> >> >> >>> >> >> >> >>> web: http://pajamadesign.com >> >> >> >>> >> >> >> >>> Skype: pajamadesign >> >> >> >>> Yahoo: jacobsingh >> >> >> >>> AIM: jacobsingh >> >> >> >>> gTalk: [email protected] >> >> >> >>> _______________________________________________ >> >> >> >>> Supervisor-users mailing list >> >> >> >>> [email protected] >> >> >> >>> http://lists.supervisord.org/mailman/listinfo/supervisor-users >> >> >> >> >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > >> >> >> > +1 510 277-0891 (o) >> >> >> > +91 9999 33 7458 (m) >> >> >> > >> >> >> > web: http://pajamadesign.com >> >> >> > >> >> >> > Skype: pajamadesign >> >> >> > Yahoo: jacobsingh >> >> >> > AIM: jacobsingh >> >> >> > gTalk: [email protected] >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> +1 510 277-0891 (o) >> >> >> +91 9999 33 7458 (m) >> >> >> >> >> >> web: http://pajamadesign.com >> >> >> >> >> >> Skype: pajamadesign >> >> >> Yahoo: jacobsingh >> >> >> AIM: jacobsingh >> >> >> gTalk: [email protected] >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> >> >> +1 510 277-0891 (o) >> >> +91 9999 33 7458 (m) >> >> >> >> web: http://pajamadesign.com >> >> >> >> Skype: pajamadesign >> >> Yahoo: jacobsingh >> >> AIM: jacobsingh >> >> gTalk: [email protected] >> > >> > >> >> >> >> -- >> >> +1 510 277-0891 (o) >> +91 9999 33 7458 (m) >> >> web: http://pajamadesign.com >> >> Skype: pajamadesign >> Yahoo: jacobsingh >> AIM: jacobsingh >> gTalk: [email protected] > > -- +1 510 277-0891 (o) +91 9999 33 7458 (m) web: http://pajamadesign.com Skype: pajamadesign Yahoo: jacobsingh AIM: jacobsingh gTalk: [email protected] _______________________________________________ Supervisor-users mailing list [email protected] http://lists.supervisord.org/mailman/listinfo/supervisor-users
