I'm know if that would help. There seems to be some issue with either flup
or your application. I don't know if you're using fcgi_fork but if so,
maybe try switching to fcgi.
if __name__ == '__main__':
from flup.server.fcgi import WSGIServer
WSGIServer(myapp).run()
Roger
On Sun, Apr 19, 2009 at 5:29 AM, Jacob Singh <[email protected]> wrote:
> HI Roger,
>
> Would it help me to setup two separate programs? One on 8001 and one
> on 8002 or something? And then reference both in nginx? Or is it
> irrelevant?
>
> I can't figure out what to do, but I have to restart my prog every 15
> minutes or I start getting problems.
>
> Best,
> Jacob
>
> On Thu, Apr 16, 2009 at 5:12 AM, Roger Hoover <[email protected]>
> wrote:
> > Hi Jacob,
> >
> > If the processes have exited, you should see that in the supervisord.log
> > (BTW, you can write a simple event listener to send you email alerts if
> > processes die). If requests are timing out, it may not be that they have
> > exited but just that they are taking too long to respond, possibly
> running
> > super long DB queries. You should be able to run the ps command to
> verify
> > if they process have died or are just not responding.
> >
> > In any case, this behavior is entirely controlled by the FastCGI
> > implementation you are running and not related to supervisord. All that
> > supervisord does it fork the process with file descriptor 0 pointing to a
> > socket to use for FastCGI. It then calls exec() and the process is
> > completely taken over by the FastCGI implementation.
> >
> > Roger
> >
> > On Wed, Apr 15, 2009 at 11:07 AM, Jacob Singh <[email protected]>
> wrote:
> >>
> >> Hi Roger,
> >>
> >> I'm actually finding a separate issue which is that my procs
> >> mysteriously start dying if I leave them up for too long... It's quite
> >> weird, no errors, nothing in the log to suggest they have exited,
> >> however after 20-30 minutes of load, I start getting these:
> >>
> >> 2009/04/15 09:34:15 [error] 18840#0: *127695 upstream timed out (110:
> >> Connection timed out) while reading response header from upstream,
> >> client: 174.129.xxx.xxx, server: xxxxxxxx, request: "GET /xxxxxxxxxx
> >> HTTP/1.0", upstream: "fastcgi://127.0.0.1:8080", host: "xxx.com"
> >>
> >> Do you have any idea why this is? Everything works fine, supervisord
> >> doesn't report any issues, nor is there anything in the stderr or
> >> stdout logs, and all of the sudden, I'm getting thousands of 504
> >> errors (get about 5 req/sec).
> >>
> >> This one is really killing me. I hate having a cron job run:
> >> supervisorctl restart all every 15 minutes, and that also adds some
> >> 504s to the mix :(
> >>
> >> Best,
> >> Jacob
> >>
> >> Thanks!
> >> Jacob
> >>
> >>
> >>
> >>
> >> On Wed, Apr 15, 2009 at 10:51 PM, Roger Hoover <[email protected]>
> >> wrote:
> >> > Hi Jacob,
> >> >
> >> > If you use supervisorctl to restart just the gate processes (and not a
> >> > full
> >> > shutdown of supervisrod), requests should not get dropped during the
> >> > restart. The reason is that the fastcgi socket will still be open.
> >> > There
> >> > will be no processes accepting connections on it temporarily but the
> >> > connection requests will queue up in the OS until restart is
> successful
> >> > and
> >> > the new gate processes are accepting connections again.
> >> >
> >> > The only potential issue I see is if the restart takes too long and
> >> > either
> >> > the inbound connection queue buffer limit is reached or nginx
> >> > mod_fastcgi
> >> > times out the request for taking too long.
> >> >
> >> > If stopping and starting all the gate processes in one command takes
> to
> >> > long, you can restart them one by one with supervisorctl.
> >> >
> >> > Hope that helps,
> >> >
> >> > Roger
> >> >
> >> > On Wed, Apr 15, 2009 at 5:50 AM, Jacob Singh <[email protected]>
> >> > wrote:
> >> >>
> >> >> Hi Roger,
> >> >>
> >> >> I'm having some issues since going into production.
> >> >>
> >> >> Pardon my ignorance, but...
> >> >>
> >> >> I've got the following config:
> >> >>
> >> >> [fcgi-program:gate]
> >> >> socket=tcp://127.0.0.1:8080 ; We reference this later in nginx
> >> >> command = /usr/local/mything/bin/gate2.py
> >> >> environment=PYTHON_EGG_CACHE=/tmp/.pythoneggs,LOG_LEVEL=WARNING
> >> >> #warning
> >> >> numprocs=10
> >> >> process_name=gate_%(process_num)s
> >> >>
> >> >> I then use:
> >> >>
> >> >> fcgi_pass 127.0.0.1:8080 in nginx to pass the requests in. It works
> >> >> fine, but then, after a while, I started getting timeout errors, so I
> >> >> added a cronjob to restart them every hour.
> >> >>
> >> >> This works okay, but I really don't want to restart them all
> together,
> >> >> because then requests are dropped.
> >> >>
> >> >> Best,
> >> >> Jacob
> >> >>
> >> >>
> >> >> On Wed, Apr 8, 2009 at 1:11 AM, Roger Hoover <[email protected]
> >
> >> >> wrote:
> >> >> > Sweet. Glad it worked for you. A release of supervisor should be
> >> >> > coming
> >> >> > soon.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> http://www.mail-archive.com/[email protected]/msg00144.html
> >> >> >
> >> >> > On Mon, Apr 6, 2009 at 10:46 PM, Jacob Singh <[email protected]
> >
> >> >> > wrote:
> >> >> >>
> >> >> >> Nevermind, I got it figured out.
> >> >> >>
> >> >> >> Thanks! This is awseome. I hope it gets into a release because my
> >> >> >> company is wary of using something that hasn't had a release for a
> >> >> >> year.
> >> >> >>
> >> >> >> On Tue, Apr 7, 2009 at 11:02 AM, Jacob Singh <
> [email protected]>
> >> >> >> wrote:
> >> >> >> > Hmm...
> >> >> >> >
> >> >> >> > okay, I've got it making the request through nginx, however, the
> >> >> >> > environ variable is empty in my wsgi script. The same script
> >> >> >> > works
> >> >> >> > fine when I create my own named sockets and add them to an nginx
> >> >> >> > upstream...
> >> >> >> >
> >> >> >> > Not sure how to proceed on that.
> >> >> >> >
> >> >> >> > Best,
> >> >> >> > Jacob
> >> >> >> >
> >> >> >> > On Mon, Apr 6, 2009 at 9:46 PM, Roger Hoover
> >> >> >> > <[email protected]>
> >> >> >> > wrote:
> >> >> >> >> Hi Jacob,
> >> >> >> >>
> >> >> >> >> Your configuration has the FastCGI process listening on
> >> >> >> >> 127.0.0.1:1212
> >> >> >> >> so
> >> >> >> >> that socket is expecting the client to speak FCGI. If you use
> >> >> >> >> curl
> >> >> >> >> to
> >> >> >> >> send
> >> >> >> >> an HTTP request, it won't understand the request. You need to
> >> >> >> >> configure a
> >> >> >> >> web server such as nginx that will proxy HTTP requests over
> >> >> >> >> FastCGI.
> >> >> >> >> Nginx
> >> >> >> >> will need to run listen on another socket (say 5000) and proxy
> >> >> >> >> requests
> >> >> >> >> to
> >> >> >> >> your FastCGI processes listening on 127.0.0.1:1212.
> >> >> >> >>
> >> >> >> >> Hope that helps,
> >> >> >> >>
> >> >> >> >> Roger
> >> >> >> >>
> >> >> >> >> On Sun, Apr 5, 2009 at 10:28 PM, Jacob Singh
> >> >> >> >> <[email protected]>
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> Hi folks!
> >> >> >> >>>
> >> >> >> >>> I just found out about this project from:
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> http://just-another.net/2009/01/18/byteflowdjangosupervisordnginx-win/#comments
> >> >> >> >>>
> >> >> >> >>> I've been trying to accomplish the same goal, but not using
> >> >> >> >>> django.
> >> >> >> >>>
> >> >> >> >>> It all *kinda* works, but when I try to curl my fcgi program,
> I
> >> >> >> >>> get
> >> >> >> >>> nada, and it just hangs forever with no logs... don't know
> where
> >> >> >> >>> to
> >> >> >> >>> start. I'm using trunk.
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> Server info:
> >> >> >> >>> Python 2.4.3 (#1, Mar 14 2007, 18:51:08)
> >> >> >> >>> [GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> Here's my config (relevant bits):
> >> >> >> >>> --------------------------------------------------
> >> >> >> >>> [supervisord]
> >> >> >> >>> logfile=/tmp/supervisord.log ; (main log file;default
> >> >> >> >>> $CWD/supervisord.log)
> >> >> >> >>> logfile_maxbytes=50MB ; (max main logfile bytes b4
> >> >> >> >>> rotation;default
> >> >> >> >>> 50MB)
> >> >> >> >>> logfile_backups=10 ; (num of main logfile rotation
> >> >> >> >>> backups;default 10)
> >> >> >> >>> loglevel=debug ; (log level;default info;
> others:
> >> >> >> >>> debug,warn,trace)
> >> >> >> >>> pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default
> >> >> >> >>> supervisord.pid)
> >> >> >> >>> nodaemon=true ; (start in foreground if
> true;default
> >> >> >> >>> false)
> >> >> >> >>> minfds=1024 ; (min. avail startup file
> >> >> >> >>> descriptors;default
> >> >> >> >>> 1024)
> >> >> >> >>> minprocs=200 ; (min. avail process
> >> >> >> >>> descriptors;default
> >> >> >> >>> 200)
> >> >> >> >>> ;umask=022 ; (process file creation
> >> >> >> >>> umask;default
> >> >> >> >>> 022)
> >> >> >> >>> user=nobody ; (default is current user,
> required
> >> >> >> >>> if
> >> >> >> >>> root)
> >> >> >> >>>
> >> >> >> >>> ; Production setup
> >> >> >> >>> [fcgi-program:gate]
> >> >> >> >>> socket=tcp://127.0.0.1:1212 ; We reference this later in
> nginx
> >> >> >> >>> #command = /usr/local/solrflare/bin/gate.py ; Calls the above
> >> >> >> >>> code
> >> >> >> >>> command = /tmp/new.py
> >> >> >> >>> environment=PYTHON_EGG_CACHE=/tmp ; Setup needed environment
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> And here is new.py:
> >> >> >> >>> ----------------------------------------------
> >> >> >> >>>
> >> >> >> >>> #!/usr/bin/python
> >> >> >> >>> from flup.server.fcgi import WSGIServer
> >> >> >> >>> import time, os, sys
> >> >> >> >>>
> >> >> >> >>> open('/tmp/new.log','a').write('something')
> >> >> >> >>> def app(environ, start_response):
> >> >> >> >>> open('/tmp/new.log','a').write('else')
> >> >> >> >>> status = "200 OK"
> >> >> >> >>> response_headers = [('Content-type', 'text/plain')]
> >> >> >> >>> start_response(status, response_headers)
> >> >> >> >>> return ['LOALALA\n']
> >> >> >> >>> WSGIServer(app).run()
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> My Log:
> >> >> >> >>> -------------------------------------------------
> >> >> >> >>> [r...@balancer:/tmp] supervisord
> >> >> >> >>> 2009-04-06 01:19:01,308 CRIT Set uid to user 99
> >> >> >> >>> 2009-04-06 01:19:01,500 INFO RPC interface 'supervisor'
> >> >> >> >>> initialized
> >> >> >> >>> 2009-04-06 01:19:01,501 INFO RPC interface 'supervisor'
> >> >> >> >>> initialized
> >> >> >> >>> 2009-04-06 01:19:01,501 INFO supervisord started with pid 5886
> >> >> >> >>> 2009-04-06 01:19:02,499 DEBG fd 8 closed, stopped monitoring
> >> >> >> >>> <PInputDispatcher at -1216741876 for <Subprocess at
> -1216915476
> >> >> >> >>> with
> >> >> >> >>> name gate in state STARTING> (stdin)>
> >> >> >> >>> 2009-04-06 01:19:02,510 INFO spawned: 'gate' with pid 5888
> >> >> >> >>> 2009-04-06 01:19:03,508 INFO success: gate entered RUNNING
> >> >> >> >>> state,
> >> >> >> >>> process has stayed up for > than 1 seconds (startsecs)
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> curl localhost:1212
> >> >> >> >>> Just sits there forever...
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> Help!?
> >> >> >> >>>
> >> >> >> >>> Thanks,
> >> >> >> >>> Jacob
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>>
> >> >> >> >>> +1 510 277-0891 (o)
> >> >> >> >>> +91 9999 33 7458 (m)
> >> >> >> >>>
> >> >> >> >>> web: http://pajamadesign.com
> >> >> >> >>>
> >> >> >> >>> Skype: pajamadesign
> >> >> >> >>> Yahoo: jacobsingh
> >> >> >> >>> AIM: jacobsingh
> >> >> >> >>> gTalk: [email protected]
> >> >> >> >>> _______________________________________________
> >> >> >> >>> Supervisor-users mailing list
> >> >> >> >>> [email protected]
> >> >> >> >>>
> http://lists.supervisord.org/mailman/listinfo/supervisor-users
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > +1 510 277-0891 (o)
> >> >> >> > +91 9999 33 7458 (m)
> >> >> >> >
> >> >> >> > web: http://pajamadesign.com
> >> >> >> >
> >> >> >> > Skype: pajamadesign
> >> >> >> > Yahoo: jacobsingh
> >> >> >> > AIM: jacobsingh
> >> >> >> > gTalk: [email protected]
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >>
> >> >> >> +1 510 277-0891 (o)
> >> >> >> +91 9999 33 7458 (m)
> >> >> >>
> >> >> >> web: http://pajamadesign.com
> >> >> >>
> >> >> >> Skype: pajamadesign
> >> >> >> Yahoo: jacobsingh
> >> >> >> AIM: jacobsingh
> >> >> >> gTalk: [email protected]
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> +1 510 277-0891 (o)
> >> >> +91 9999 33 7458 (m)
> >> >>
> >> >> web: http://pajamadesign.com
> >> >>
> >> >> Skype: pajamadesign
> >> >> Yahoo: jacobsingh
> >> >> AIM: jacobsingh
> >> >> gTalk: [email protected]
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> +1 510 277-0891 (o)
> >> +91 9999 33 7458 (m)
> >>
> >> web: http://pajamadesign.com
> >>
> >> Skype: pajamadesign
> >> Yahoo: jacobsingh
> >> AIM: jacobsingh
> >> gTalk: [email protected]
> >
> >
>
>
>
> --
>
> +1 510 277-0891 (o)
> +91 9999 33 7458 (m)
>
> web: http://pajamadesign.com
>
> Skype: pajamadesign
> Yahoo: jacobsingh
> AIM: jacobsingh
> gTalk: [email protected]
>
_______________________________________________
Supervisor-users mailing list
[email protected]
http://lists.supervisord.org/mailman/listinfo/supervisor-users
