Re: Killing reloaders when parent receives TERM signals

[Malcolm Tredinnick]

On Thu, 2009-03-26 at 21:58 +, Chris Lamb wrote:
> Malcolm Tredinnick wrote:
> 
> > The development server is intended for single user, simple development
> > stuff. It runs as a foreground process and ^C stops it very nicely.
> [..]
> > The goal is, intentionally, to keep the development server simple since
> > there are numerous options for more functional setups when required.
> 
> You will have to excuse my ham-fisted segue, but it would be nice to see
> more testing of:
> 
>   http://code.djangoproject.com/ticket/9722
> 
> I would certainly agree that the development server should be simple, but
> eating all my laptop's battery is somewhat antithetical to development.

It'll probably go in one day. I saw the ticket when you posted it and
thought it was worth considering, but it's not drop-dead urgent. It's
for the 1.2 timeframe now.

Regards,
Malcolm



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Killing reloaders when parent receives TERM signals

[Chris Lamb]

Malcolm Tredinnick wrote:

> The development server is intended for single user, simple development
> stuff. It runs as a foreground process and ^C stops it very nicely.
[..]
> The goal is, intentionally, to keep the development server simple since
> there are numerous options for more functional setups when required.

You will have to excuse my ham-fisted segue, but it would be nice to see
more testing of:

  http://code.djangoproject.com/ticket/9722

I would certainly agree that the development server should be simple, but
eating all my laptop's battery is somewhat antithetical to development.


Regards,

-- 
Chris Lamb, UK ch...@chris-lamb.co.uk
  GPG: 0x634F9A20


signature.asc
Description: PGP signature

Re: Killing reloaders when parent receives TERM signals

[Dave Benjamin]

On Mar 24, 5:04 pm, Graham Dumpleton 
wrote:
> Use Apache/mod_wsgi and the documented method of running a WSGI
> application in a specific daemon process in a autoreloading
> development mode. You could even have both the production and
> development instances under same Apache under different hosts, with
> only development instance having autoreload enabled. Would be much
> simpler as you don't have to muck around with the supervisor software.
> Also basically otherwise the same set up as your production
> environment and so better test as far as making sure everything works
> properly. See:
>
>  http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
>
>  http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django...
>  http://blog.dscpl.com.au/2009/02/source-code-reloading-with-modwsgi-o...

I like mucking around with supervisor software. But this is clearly a
superior approach. Thanks for the links!

Dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Killing reloaders when parent receives TERM signals

[Graham Dumpleton]

On Mar 25, 9:27 am, Dave Benjamin  wrote:
> No, I'm trying to run it as a dev server. I use mod_python in
> production. However, I use the dev server in a shared environment so
> that a few other programmers and testers in the office can view the
> site.
>
> In any case, sending a TERM signal to the parent process should cause
> the child to die. There is no reasonable justification for the current
> behavior, IMHO.

Use Apache/mod_wsgi and the documented method of running a WSGI
application in a specific daemon process in a autoreloading
development mode. You could even have both the production and
development instances under same Apache under different hosts, with
only development instance having autoreload enabled. Would be much
simpler as you don't have to muck around with the supervisor software.
Also basically otherwise the same set up as your production
environment and so better test as far as making sure everything works
properly. See:

  http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode

  http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
  http://blog.dscpl.com.au/2009/02/source-code-reloading-with-modwsgi-on.html

Graham

> On Mar 24, 2:45 pm, graham_king  wrote:
>
> >  It looks like you're trying to run the dev server as a real web
> > server, which is a very bad idea (it's only serves one request at a
> > time, for starters). As it says here:
>
> >http://docs.djangoproject.com/en/dev/ref/django-admin/#runserver-opti...
>
> > "DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone
> > through security audits or performance tests. (And that's how it's
> > gonna stay. We're in the business of making Web frameworks, not Web
> > servers, so improving this server to be able to handle a production
> > environment is outside the scope of Django.)"
>
> > Is there a reason you can't use Apache, or a different server with a
> > WSGI or FCGI setup? If you're not running it in production, why do you
> > need daemontools?
>
> >  Best regards,
> >  Graham
>
> > On Mar 24, 11:13 am, Dave Benjamin  wrote:
>
> > > Hello,
>
> > > I run the Django development server (manage.py runserver) under
> > > Daemontools on Linux. Daemontools is a server management framework
> > > that runs a foreground process as a daemon and automates the tasks of
> > > starting, restarting, signal handling, and logging. I use a run script
> > > that looks like the following:
>
> > > #!/bin/sh
> > > exec 2>&1
> > > exec setuidgid www-data envdir ./env myapp/manage.py runserver --
> > > adminmedia=/usr/local/myapp/media/ 0.0.0.0:8000
>
> > > This has been working well for me for awhile except for one problem.
> > > The usual way of causing a Daemontools service to exit, either to
> > > terminate the service or to do a full restart, is to type the
> > > following:
>
> > > svc -t /service/myapp
>
> > > This sends a TERM signal to the parent process. Due to the way that
> > > Django's autoreloader module (django.utils.autoreload) is written,
> > > there are actually two processes, a monitoring parent process and a
> > > child process exits whenever a reload is necessary. Sending a TERM
> > > signal to the parent process leaves the child process running, and
> > > this causes an orphan process that never exits and hogs the socket.
>
> > > If the --noreload argument is passed to manage.py runserver, this
> > > problem goes away because there is only one process running.
> > > Unfortunately, this disables the autoreload functionality.
>
> > > I have written a modification to my manage.py script that
> > > monkeypatches the autoreload module so that it keeps a set of child
> > > PIDs and sends them TERM signals when the parent process receives one.
> > > I changed the call to spawnve so that it sends P_NOWAIT instead of
> > > P_WAIT, causing the call to return a PID instead of an exit code. I
> > > then stash this code in the set and do the os.waitpid() call myself.
> > > The code is a modified version of the spawn* wrappers in os.py, and
> > > handles the exit code processing and removal of exited/signalled PIDs
> > > from the set.
>
> > > I haven't tested this change on anything besides Linux, so at the very
> > > least I'd imagine it causes some problems under Windows. I'm posting
> > > it here in hope that it will be useful to someone else, and possibly
> > > with some tweaking be integrated into the autoreload module.
>
> > > The following is what my manage.py script looks like now:
>
> > > #!/usr/bin/python
> > > from django.core.management import execute_manager
> > > try:
> > >     import settings # Assumed to be in the same directory.
> > > except ImportError:
> > >     import sys
> > >     sys.stderr.write("Error: Can't find the file 'settings.py' in the
> > > directory containing %r. It appears you've customized things.\nYou'll
> > > have to run django-admin.py, passing it your settings module.\n(If the
> > > file 

Re: Killing reloaders when parent receives TERM signals

[Dave Benjamin]

On Mar 24, 4:13 pm, Malcolm Tredinnick 
wrote:
> On Tue, 2009-03-24 at 15:27 -0700, Dave Benjamin wrote:
> > In any case, sending a TERM signal to the parent process should cause
> > the child to die. There is no reasonable justification for the current
> > behavior, IMHO.
>
> There's a lot of justification for the current behaviour: it's a *lot*
> simpler than the change you are proposing we make and maintain forever.

Well, it's simple enough for me to hack around it, so I withdraw my
proposal. =)

> The development server is intended for single user, simple development
> stuff. It runs as a foreground process and ^C stops it very nicely.

I admit I am a bit of a daemontools addict. I use it even for single
user, simple development. The ability to easily start and stop
processes, and the built-in, auto-rotating logs that record output
even when apps crash hard are totally worth the effort for me.

> If you want anything more than that, use a real webserver. It's easy to
> set up mod_wsgi so that it reloads (see [1]). Or use --noreload and
> restart if/when you make changes to something (if other people are
> viewing it at the same time, random restarts aren't generally going to
> be a good idea).
>
> The goal is, intentionally, to keep the development server simple since
> there are numerous options for more functional setups when required.

I didn't realize that it was possible to do auto-reloading with WSGI.
I was under the assumption that the choice was between mod_python,
which reloads unpredictably (unless you use the MaxRequestsPerChild
trick), and the standalone server, which reloads reliably but
sometimes crashes. I really want the best of both worlds: a reliable,
auto-reloading, shared development environment. I now understand that
there are many more options than those I've considered.

Thanks,
Dave

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Killing reloaders when parent receives TERM signals

[Malcolm Tredinnick]

On Tue, 2009-03-24 at 15:27 -0700, Dave Benjamin wrote:
> No, I'm trying to run it as a dev server. I use mod_python in
> production. However, I use the dev server in a shared environment so
> that a few other programmers and testers in the office can view the
> site.
> 
> In any case, sending a TERM signal to the parent process should cause
> the child to die. There is no reasonable justification for the current
> behavior, IMHO.

There's a lot of justification for the current behaviour: it's a *lot*
simpler than the change you are proposing we make and maintain forever.

The development server is intended for single user, simple development
stuff. It runs as a foreground process and ^C stops it very nicely.

If you want anything more than that, use a real webserver. It's easy to
set up mod_wsgi so that it reloads (see [1]). Or use --noreload and
restart if/when you make changes to something (if other people are
viewing it at the same time, random restarts aren't generally going to
be a good idea).

The goal is, intentionally, to keep the development server simple since
there are numerous options for more functional setups when required.

[1]
http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html

Regards,
Malcolm



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Killing reloaders when parent receives TERM signals

[Dave Benjamin]

No, I'm trying to run it as a dev server. I use mod_python in
production. However, I use the dev server in a shared environment so
that a few other programmers and testers in the office can view the
site.

In any case, sending a TERM signal to the parent process should cause
the child to die. There is no reasonable justification for the current
behavior, IMHO.

On Mar 24, 2:45 pm, graham_king  wrote:
>  It looks like you're trying to run the dev server as a real web
> server, which is a very bad idea (it's only serves one request at a
> time, for starters). As it says here:
>
> http://docs.djangoproject.com/en/dev/ref/django-admin/#runserver-opti...
>
> "DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone
> through security audits or performance tests. (And that's how it's
> gonna stay. We're in the business of making Web frameworks, not Web
> servers, so improving this server to be able to handle a production
> environment is outside the scope of Django.)"
>
> Is there a reason you can't use Apache, or a different server with a
> WSGI or FCGI setup? If you're not running it in production, why do you
> need daemontools?
>
>  Best regards,
>  Graham
>
> On Mar 24, 11:13 am, Dave Benjamin  wrote:
>
> > Hello,
>
> > I run the Django development server (manage.py runserver) under
> > Daemontools on Linux. Daemontools is a server management framework
> > that runs a foreground process as a daemon and automates the tasks of
> > starting, restarting, signal handling, and logging. I use a run script
> > that looks like the following:
>
> > #!/bin/sh
> > exec 2>&1
> > exec setuidgid www-data envdir ./env myapp/manage.py runserver --
> > adminmedia=/usr/local/myapp/media/ 0.0.0.0:8000
>
> > This has been working well for me for awhile except for one problem.
> > The usual way of causing a Daemontools service to exit, either to
> > terminate the service or to do a full restart, is to type the
> > following:
>
> > svc -t /service/myapp
>
> > This sends a TERM signal to the parent process. Due to the way that
> > Django's autoreloader module (django.utils.autoreload) is written,
> > there are actually two processes, a monitoring parent process and a
> > child process exits whenever a reload is necessary. Sending a TERM
> > signal to the parent process leaves the child process running, and
> > this causes an orphan process that never exits and hogs the socket.
>
> > If the --noreload argument is passed to manage.py runserver, this
> > problem goes away because there is only one process running.
> > Unfortunately, this disables the autoreload functionality.
>
> > I have written a modification to my manage.py script that
> > monkeypatches the autoreload module so that it keeps a set of child
> > PIDs and sends them TERM signals when the parent process receives one.
> > I changed the call to spawnve so that it sends P_NOWAIT instead of
> > P_WAIT, causing the call to return a PID instead of an exit code. I
> > then stash this code in the set and do the os.waitpid() call myself.
> > The code is a modified version of the spawn* wrappers in os.py, and
> > handles the exit code processing and removal of exited/signalled PIDs
> > from the set.
>
> > I haven't tested this change on anything besides Linux, so at the very
> > least I'd imagine it causes some problems under Windows. I'm posting
> > it here in hope that it will be useful to someone else, and possibly
> > with some tweaking be integrated into the autoreload module.
>
> > The following is what my manage.py script looks like now:
>
> > #!/usr/bin/python
> > from django.core.management import execute_manager
> > try:
> >     import settings # Assumed to be in the same directory.
> > except ImportError:
> >     import sys
> >     sys.stderr.write("Error: Can't find the file 'settings.py' in the
> > directory containing %r. It appears you've customized things.\nYou'll
> > have to run django-admin.py, passing it your settings module.\n(If the
> > file settings.py does indeed exist, it's causing an ImportError
> > somehow.)\n" % __file__)
> >     sys.exit(1)
>
> > import os, sys, signal
> > from django.utils import autoreload
>
> > def restart_with_reloader():
> >     reloader_pids = set()
>
> >     def kill_reloaders(signum, frame):
> >         try:
> >             while True:
> >                 pid = reloader_pids.pop()
> >                 os.kill(pid, signal.SIGTERM)
> >         except KeyError:
> >             pass
>
> >     signal.signal(signal.SIGTERM, kill_reloaders)
>
> >     while True:
> >         args = [sys.executable] + sys.argv
> >         if sys.platform == "win32":
> >             args = ['"%s"' % arg for arg in args]
> >         new_environ = os.environ.copy()
> >         new_environ["RUN_MAIN"] = 'true'
> >         pid = os.spawnve(os.P_NOWAIT, sys.executable, args,
> > new_environ)
> >         reloader_pids.add(pid)
> >         while True:
> >             try:
> >                 wpid, sts = 

Re: Killing reloaders when parent receives TERM signals

[graham_king]

 It looks like you're trying to run the dev server as a real web
server, which is a very bad idea (it's only serves one request at a
time, for starters). As it says here:

http://docs.djangoproject.com/en/dev/ref/django-admin/#runserver-optional-port-number-or-ipaddr-port

"DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone
through security audits or performance tests. (And that's how it's
gonna stay. We're in the business of making Web frameworks, not Web
servers, so improving this server to be able to handle a production
environment is outside the scope of Django.)"

Is there a reason you can't use Apache, or a different server with a
WSGI or FCGI setup? If you're not running it in production, why do you
need daemontools?

 Best regards,
 Graham

On Mar 24, 11:13 am, Dave Benjamin  wrote:
> Hello,
>
> I run the Django development server (manage.py runserver) under
> Daemontools on Linux. Daemontools is a server management framework
> that runs a foreground process as a daemon and automates the tasks of
> starting, restarting, signal handling, and logging. I use a run script
> that looks like the following:
>
> #!/bin/sh
> exec 2>&1
> exec setuidgid www-data envdir ./env myapp/manage.py runserver --
> adminmedia=/usr/local/myapp/media/ 0.0.0.0:8000
>
> This has been working well for me for awhile except for one problem.
> The usual way of causing a Daemontools service to exit, either to
> terminate the service or to do a full restart, is to type the
> following:
>
> svc -t /service/myapp
>
> This sends a TERM signal to the parent process. Due to the way that
> Django's autoreloader module (django.utils.autoreload) is written,
> there are actually two processes, a monitoring parent process and a
> child process exits whenever a reload is necessary. Sending a TERM
> signal to the parent process leaves the child process running, and
> this causes an orphan process that never exits and hogs the socket.
>
> If the --noreload argument is passed to manage.py runserver, this
> problem goes away because there is only one process running.
> Unfortunately, this disables the autoreload functionality.
>
> I have written a modification to my manage.py script that
> monkeypatches the autoreload module so that it keeps a set of child
> PIDs and sends them TERM signals when the parent process receives one.
> I changed the call to spawnve so that it sends P_NOWAIT instead of
> P_WAIT, causing the call to return a PID instead of an exit code. I
> then stash this code in the set and do the os.waitpid() call myself.
> The code is a modified version of the spawn* wrappers in os.py, and
> handles the exit code processing and removal of exited/signalled PIDs
> from the set.
>
> I haven't tested this change on anything besides Linux, so at the very
> least I'd imagine it causes some problems under Windows. I'm posting
> it here in hope that it will be useful to someone else, and possibly
> with some tweaking be integrated into the autoreload module.
>
> The following is what my manage.py script looks like now:
>
> #!/usr/bin/python
> from django.core.management import execute_manager
> try:
>     import settings # Assumed to be in the same directory.
> except ImportError:
>     import sys
>     sys.stderr.write("Error: Can't find the file 'settings.py' in the
> directory containing %r. It appears you've customized things.\nYou'll
> have to run django-admin.py, passing it your settings module.\n(If the
> file settings.py does indeed exist, it's causing an ImportError
> somehow.)\n" % __file__)
>     sys.exit(1)
>
> import os, sys, signal
> from django.utils import autoreload
>
> def restart_with_reloader():
>     reloader_pids = set()
>
>     def kill_reloaders(signum, frame):
>         try:
>             while True:
>                 pid = reloader_pids.pop()
>                 os.kill(pid, signal.SIGTERM)
>         except KeyError:
>             pass
>
>     signal.signal(signal.SIGTERM, kill_reloaders)
>
>     while True:
>         args = [sys.executable] + sys.argv
>         if sys.platform == "win32":
>             args = ['"%s"' % arg for arg in args]
>         new_environ = os.environ.copy()
>         new_environ["RUN_MAIN"] = 'true'
>         pid = os.spawnve(os.P_NOWAIT, sys.executable, args,
> new_environ)
>         reloader_pids.add(pid)
>         while True:
>             try:
>                 wpid, sts = os.waitpid(pid, 0)
>             except OSError:
>                 continue
>             if os.WIFSTOPPED(sts):
>                 continue
>             elif os.WIFSIGNALED(sts):
>                 exit_code = -os.WTERMSIG(sts)
>                 if wpid in reloader_pids: reloader_pids.remove(wpid)
>                 break
>             elif os.WIFEXITED(sts):
>                 exit_code = os.WEXITSTATUS(sts)
>                 if wpid in reloader_pids: reloader_pids.remove(wpid)
>                 break
>             else:
>                 raise error, "Not stopped, signaled or