Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-22 Thread Chris McDonough
On Wed, 2012-02-22 at 09:06 +1100, Graham Dumpleton wrote:
 If you want to be able to control a thread like that from an atexit
 callback, you need to create the thread as daemonised. Ie.
 setDaemon(True) call on thread.
 
 By default a thread will actually inherit the daemon flag from the
 parent. For a command line Python where thread created from main
 thread it will not be daemonised and thus why the thread will be
 waited upon on shutdown prior to atexit being called.
 
 If you ran the same code in mod_wsgi, my memory is that the thread
 will actually inherit as being daemonised because request handler in
 mod_wsgi, from which import is trigger, are notionally daemonised.
 
 Thus the code should work in mod_wsgi. Even so, to be portable, if
 wanting to manipulate thread from atexit, make it daemonised.
 
 Example of background threads in mod_wsgi at:
 
 http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes
 
 shows use of setDaemon().
 
 Graham

I've read all the messages in this thread and the traffic on the bug
entry at http://bugs.python.org/issue14073 but I'm still not sure what
to tell people who want to invoke code at shutdown.

Do we tell them to use atexit?  If so, are we saying that atexit is
sufficient for all user-defined shutdown code that needs to run save for
code that needs to stop threads?

Is it sufficient to define shutdown as when the process associated
with the application exits?  It still seems to not necessarily be
directly correlated.

- C


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 8:56 AM, Simon Sapin simon.sa...@exyr.org wrote:

 Le 21/02/2012 08:47, Tarek Ziadé a écrit :

  Yes, I also think shutting down the server is completely orthogonal to
 requests.


 If the shutdown callback is next to the application and not registered in
 a request, why not also have the symmetric server start up callback that
 would not wait for a request? This would avoid workarounds like
 Flask.before_first_request.

 Both of these callbacks could be called once per process (aka. space where
 requests share memory.)


Fair point,


 Instead of having to provide two or three objects separately to a server,
 how about making the callbacks attributes of the application callable?


can you show us an example ?



 Regards,
 --
 Simon Sapin




-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Sylvain Hellegouarch


 Yes, I also think shutting down the server is completely orthogonal to
 requests.



CherryPy is using an approach where, indeed, both are orthogonal. Tools
[1], which are similar to middlewares, are dealing with requests whereas
the CherryPy engine [2] is the basis upon which the whole application and
HTTP servers are running. The engine provides, amongst others, a clean way
to stop and/or exit the whole stack independently from running requests.
However, due to the way Robert Brewer designed it, it also means this can,
if wanted, be performed from a request handler at any time. Note that the
engine architecture, a bus, allows obviously for functions to subscribe at
the time the engine shuts down in order to perform further operations that
you would require.

Robert has also started the work of extracting the engine from CherryPy
itself for other frameworks to rely on [3].

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

[1] http://docs.cherrypy.org/stable/concepts/tools.html
[2] http://docs.cherrypy.org/stable/concepts/engine.html
[3] https://bitbucket.org/cherrypy/magicbus
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Benoit Chesneau
On Mon, Feb 20, 2012 at 5:03 PM, Tarek Ziadé ziade.ta...@gmail.com wrote:
 Hello

 I need to be able to call a function when the web application shuts down
 (SIGTERM/SIGINT) -- the use case is to stop a background thread.

 I am currently using signals because it seems to be the most clean way to do
 this. atexit is much trickier since you don't know when it's going to get
 called and you might try to call objects that were garbage collected unless
 you hack something to keep references alive.

 But signals are also tricky beasts since you may compete with other code
 that are listening to them. For instance mod_wsgi don't like apps that have
 signal handlers.

 Anyways, the bottom line is that the cleanest way to do this -- as per Chris
 McDonough idea, would be to introduce in the WSGI protocol a shutdown
 function the servers would be obligated to call before exiting.

 I am not sure yet about its arguments, maybe a signum + frame or simply an
 exit code...

 But how do you like the idea ?  That would solve for me the problem of
 having to deal differently here depending on if I am called with mod_wsgi or
 gunicorn or xxx

Hi Tarek,

In gunicorn we have the concept of hook which exactly does what you want
(on_exit hook). You can do this via the configuration file. And you can
do this for the other steps (reload, pre/post fork, pre/post request
...).

Anyway, I think there are 2 issues to solve with this shutdown thing
(and more) in a generic way:

1. how to make your application aware of the server capabilities (can it
or not handle shutdown)
2. how to pass such functions to the server.

There are also some others details like determining what is the context
needed by the application at this step.

Today the server is acting as a gateway so it just pass a request to an
application and return the response from the application. The
application by itself only lives in this context. Maybe, like some said,
if we want to give more info about that, we could imagine the application
object or function as a bus reacting on different contexts:

def application(environ, start_application):
pass

where the environ can be different depending on the context an action, we
could have:

http request{wsgi.context: http, ...}
shutdown{wsgi.context; server, action: shutdown, ...}

smth like it. But how to know when the server can't pass the action
shutdown? Also what would be the environ at shutdown?

One other method, like the one described by Sylvain Hellegouarch, would
be adding some properties/attributes to the application object. So the
server would eventually check if they exists and eventually call them.
But again how to know when the server can't pass the action shutdown?
Though, I don't like to handle more than the application callable today,
I can see some problems when you have to handle server reload and such
things.

It seems important for me that the application knows about the server
capabilities if it want to handle such things like the `shutdown`. And
this step is solved by gunicorn  others by specifically handling a
configuration. But maybe we could just considers that a server always
handle the reload, shutdown, also pre/post request evenst and have
optionnals message passed to the application? (how do we handled process
context changes - if we re in a thread for ex - ?).

We could also imagine a web application package that provides hooks
handling depending on the server target... I quite like this idea.

Not sure what is the right way. What do you think?

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Graham Dumpleton
On 21 February 2012 18:53, Tarek Ziadé ziade.ta...@gmail.com wrote:


 On Tue, Feb 21, 2012 at 2:39 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:

 ...

 Overall the best chance of being able to do anything is relying on atexit.

 You are though at the mercy of the WSGI hosting mechanism shutting
 down the process and so the interpreter, in an orderly manner such
 that atexit callbacks get called.

 In Apache/mod_wsgi you get this guarantee, even in sub interpreters
 where atexit callbacks wouldn't normally be called when they are
 destroyed.

 For uWSGI, atexit callbacks will not be called at the moment, by
 Robert is making changes to it so you get a guarantee there as well.
 It is possible he is only doing this though for case where main
 interpreter is being used, as doing it for sub interpreters is a bit
 fiddly.


 But I don't think you can guarantee that everything is still up in memory by
 the time atexit gets called,
 so you can't really call cleanup code there.

The only thing which is done prior to atexit callbacks being called is
waiting on threads which weren't marked as daemonised.

void
Py_Finalize(void)
{
PyInterpreterState *interp;
PyThreadState *tstate;

if (!initialized)
return;

wait_for_thread_shutdown();

/* The interpreter is still entirely intact at this point, and the
 * exit funcs may be relying on that.  In particular, if some thread
 * or exit func is still waiting to do an import, the import machinery
 * expects Py_IsInitialized() to return true.  So don't say the
 * interpreter is uninitialized until after the exit funcs have run.
 * Note that Threading.py uses an exit func to do a join on all the
 * threads created thru it, so this also protects pending imports in
 * the threads created via Threading.
 */
call_sys_exitfunc();

...

 Any pure Python WSGI servers shouldn't have issues so long as they
 aren't force exiting the whole process and bypassing normal
 interpreter destruction.


 what do you mean by bypassing its destruction ?

Non catchable signal from within process or from a distinct monitoring process.

One of this things I pointed out is being missed.

That is, a WSGI adapter may be running on top of another layer of
abstraction, such as FASTCGI for example, where the lower layer isn't
going to have any callback mechanism of its own to even notify the
WSGI layer to trigger registered cleanup callbacks.

This is why the only mechanism one can universally rely on is the
Python interpreters own atexit mechanism.

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Simon Sapin

Le 21/02/2012 09:23, Tarek Ziadé a écrit :

Instead of having to provide two or three objects separately to a
server, how about making the callbacks attributes of the application
callable?


can you show us an example ?


Proposal:

Function-based:

def startup():
return open_resource(something)

def shutdown(resource):
resource.close()

def application(environ, start_response):
# ...
return response_body

application.startup = startup
application.shutdown = shutdown

Class-based:

class App(object):
def startup(self):
return open_resource(something)

def shutdown(self, resource):
resource.close()

def __call__(self, environ, start_response):
# ...
return response_body

application = App()

The return value of startup() can be any python object and is opaque to 
the server. It is passed as-is to shutdown()


startup() could take more parameters. Maybe the application (though can 
we already have it as self for class-based or in a closure for 
function-based)


Regards,
--
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Graham Dumpleton
On 21 February 2012 20:26, Simon Sapin simon.sa...@exyr.org wrote:
 Le 21/02/2012 09:23, Tarek Ziadé a écrit :

    Instead of having to provide two or three objects separately to a
    server, how about making the callbacks attributes of the application
    callable?


 can you show us an example ?


 Proposal:

 Function-based:

    def startup():
        return open_resource(something)

    def shutdown(resource):
        resource.close()

    def application(environ, start_response):
        # ...
        return response_body

    application.startup = startup
    application.shutdown = shutdown

 Class-based:

    class App(object):
        def startup(self):
            return open_resource(something)

        def shutdown(self, resource):
            resource.close()

        def __call__(self, environ, start_response):
            # ...
            return response_body

    application = App()

 The return value of startup() can be any python object and is opaque to the
 server. It is passed as-is to shutdown()

 startup() could take more parameters. Maybe the application (though can we
 already have it as self for class-based or in a closure for function-based)

You do realise you are just reinventing context managers?

With this 'application' do requests.

But then it was sort of suggested that was a bit too radical idea when
I have mentioned viewing it that way before. :-(

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Sylvain Hellegouarch
On Tue, Feb 21, 2012 at 10:31 AM, Graham Dumpleton 
graham.dumple...@gmail.com wrote:

 On 21 February 2012 20:26, Simon Sapin simon.sa...@exyr.org wrote:
  Le 21/02/2012 09:23, Tarek Ziadé a écrit :
 
 Instead of having to provide two or three objects separately to a
 server, how about making the callbacks attributes of the application
 callable?
 
 
  can you show us an example ?
 
 
  Proposal:
 
  Function-based:
 
 def startup():
 return open_resource(something)
 
 def shutdown(resource):
 resource.close()
 
 def application(environ, start_response):
 # ...
 return response_body
 
 application.startup = startup
 application.shutdown = shutdown
 
  Class-based:
 
 class App(object):
 def startup(self):
 return open_resource(something)
 
 def shutdown(self, resource):
 resource.close()
 
 def __call__(self, environ, start_response):
 # ...
 return response_body
 
 application = App()
 
  The return value of startup() can be any python object and is opaque to
 the
  server. It is passed as-is to shutdown()
 
  startup() could take more parameters. Maybe the application (though can
 we
  already have it as self for class-based or in a closure for
 function-based)

 You do realise you are just reinventing context managers?

 With this 'application' do requests.

 But then it was sort of suggested that was a bit too radical idea when
 I have mentioned viewing it that way before. :-(


One might wonder if having access to process management should be part of
WSGI in the first place.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Benoit Chesneau
On Tue, Feb 21, 2012 at 10:35 AM, Sylvain Hellegouarch s...@defuze.org wrote:


 On Tue, Feb 21, 2012 at 10:31 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:

 On 21 February 2012 20:26, Simon Sapin simon.sa...@exyr.org wrote:
  Le 21/02/2012 09:23, Tarek Ziadé a écrit :
 
     Instead of having to provide two or three objects separately to a
     server, how about making the callbacks attributes of the application
     callable?
 
 
  can you show us an example ?
 
 
  Proposal:
 
  Function-based:
 
     def startup():
         return open_resource(something)
 
     def shutdown(resource):
         resource.close()
 
     def application(environ, start_response):
         # ...
         return response_body
 
     application.startup = startup
     application.shutdown = shutdown
 
  Class-based:
 
     class App(object):
         def startup(self):
             return open_resource(something)
 
         def shutdown(self, resource):
             resource.close()
 
         def __call__(self, environ, start_response):
             # ...
             return response_body
 
     application = App()
 
  The return value of startup() can be any python object and is opaque to
  the
  server. It is passed as-is to shutdown()
 
  startup() could take more parameters. Maybe the application (though can
  we
  already have it as self for class-based or in a closure for
  function-based)

 You do realise you are just reinventing context managers?

 With this 'application' do requests.

 But then it was sort of suggested that was a bit too radical idea when
 I have mentioned viewing it that way before. :-(


 One might wonder if having access to process management should be part of
 WSGI in the first place.


that's the thing. This is no more a gateway. Like I said in my
previous post, maybe having another spec describing a web app package
would do the trick?

- benoît
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Simon Sapin

Le 21/02/2012 08:56, Simon Sapin a écrit :

Both of these callbacks could be called once per process (aka. space
where requests share memory.)


Sorry, that should be once per interpreter.

--
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Simon Sapin

Le 21/02/2012 10:31, Graham Dumpleton a écrit :
 You do realise you are just reinventing context managers?

 With this 'application' do requests.

Indeed. I didn’t want to go too far from the initial shutdown function 
proposal, but actual context managers would be better.



Le 21/02/2012 10:38, Benoit Chesneau a écrit :

One might wonder if having access to process management should be part of
  WSGI in the first place.


that's the thing. This is no more a gateway. Like I said in my
previous post, maybe having another spec describing a web app package
would do the trick?


So a super-set of WSGI that is not just a gateway but also does process 
management?



@contextlib.contextmanager
def super_application(server_infos):
with contextlib.closing(open_resource()) as resource:
wsgi_callable = make_app(server_infos, resource)
yield wsgi_callable

--
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton 
graham.dumple...@gmail.com wrote:

 ...
  But I don't think you can guarantee that everything is still up in
 memory by
  the time atexit gets called,
  so you can't really call cleanup code there.

 The only thing which is done prior to atexit callbacks being called is
 waiting on threads which weren't marked as daemonised.


which can lead to completely lock the shutdown if a lib or the program has
a thread with a loop that waits for a condition.

which it is not the case with signals, since you get a chance to properly
stop everything beforehand.



 what do you mean by bypassing its destruction ?

 Non catchable signal from within process or from a distinct monitoring
process.
 One of this things I pointed out is being missed.
 That is, a WSGI adapter may be running on top of another layer of
 abstraction, such as FASTCGI for example, where the lower layer isn't
 going to have any callback mechanism of its own to even notify the
 WSGI layer to trigger registered cleanup callbacks.
 This is why the only mechanism one can universally rely on is the
 Python interpreters own atexit mechanism.

I see.. but what I don't understand is the following: when the whole stack
is shut down, the python process is being killed by *someone*.

And that someone, as far as I understand, is also able to send requests to
the WSGI application.

So what makes it impossible to send a shutdown signal prior to killing the
process ?

Sorry if I miss the obvious, I am probably over-simplifying things here :d

Cheers
Tarek
-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 11:07 AM, Simon Sapin simon.sa...@exyr.org wrote:

 ...



 So a super-set of WSGI that is not just a gateway but also does process
 management?


 @contextlib.contextmanager
 def super_application(server_**infos):
with contextlib.closing(open_**resource()) as resource:
wsgi_callable = make_app(server_infos, resource)
yield wsgi_callable


I like this form a lot, but I think this is an implementation detail --
since we've not answered to the main question yet.

Here's my attempt to formulate how I understand the problem at this point

current assumptions/limitations:

-  the application can be shutdown without handling a request / which makes
it orthogonal to the requests handling
-  the underlying code may use threads, making it unreliable to use atexit()
-  using signals may be problematic if some other code use it too -- for
example the wsgi server itself

- the cleanest way seems to ask the web server itself to ping the wsgi app.

problems:

- how can we declare a shutdown entry point in the application, the web
server can use.
- how can this work with extra indirections (FASTCGI, etc)

leads:
- define startup/shutdown functions, declare them to the web server
- use the existing environ to send a 'shutdown request'
-

Cheers
Tarek

-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Graham Dumpleton
On 21 February 2012 21:41, Tarek Ziadé ziade.ta...@gmail.com wrote:


 On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:

 ...

  But I don't think you can guarantee that everything is still up in
  memory by
  the time atexit gets called,
  so you can't really call cleanup code there.

 The only thing which is done prior to atexit callbacks being called is
 waiting on threads which weren't marked as daemonised.

 which can lead to completely lock the shutdown if a lib or the program has a
 thread with a loop that waits for a condition.

In mod_wsgi at least there are fail safes such that background C
threads will force kill the process if such a lockup occurs on
shutdown.

 which it is not the case with signals, since you get a chance to properly
 stop everything beforehand.

Yes and no. For a signal handler to even be able to be triggered,
there must be Python code executing in the main thread that originally
created the main interpreter.

In an embedded system such as mod_wsgi, the main thread is never used
to handle requests and actually runs in C code blocked waiting for an
internal notification that process is being shutdown.

 what do you mean by bypassing its destruction ?

 Non catchable signal from within process or from a distinct monitoring
 process.
 One of this things I pointed out is being missed.
 That is, a WSGI adapter may be running on top of another layer of
 abstraction, such as FASTCGI for example, where the lower layer isn't
 going to have any callback mechanism of its own to even notify the
 WSGI layer to trigger registered cleanup callbacks.
 This is why the only mechanism one can universally rely on is the
 Python interpreters own atexit mechanism.

 I see.. but what I don't understand is the following: when the whole stack
 is shut down, the python process is being killed by *someone*.

 And that someone, as far as I understand, is also able to send requests to
 the WSGI application.

 So what makes it impossible to send a shutdown signal prior to killing the
 process ?

Is not impossible and in mod_wsgi at least a signal is used to
initiate shutdown, this coming either from itself in some cases, or
from Apache parent process in others. The signal handler then uses a
socketpair pipe to wake up the blocked main thread to begin shutdown
steps. Either way it is handled at C code level because can't rely on
Python level signal handlers to actually run.

To further complicate things, in a process with multiple sub
interpreters where would the Python signal handler even run. There is
no main thread running waiting to exit. It also can't just cause the
main Python interpreter to be exited. Simply exiting a main thread
even if it did exist wouldn't allow you to cleanup sub interpreters.

In short, embedded systems are going to be quite different to what you
are used to with pure WSGI servers. It is because it is doing all this
to ensure that reliable shutdown can occur that mod_wsgi ignores all
Python signal handler registrations by default.

That all said, technically mod_wsgi could on reception of its signal,
and if there was a registry of known WSGI applications, it could tell
them the process is being shutdown. Presence of sub interpreters makes
that a lot of fun, but would be doable.

Right now without such a registry of applications with enter/exit
methods as being discussed in this thread, the only way in mod_wsgi is
to rely on atexit.

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Graham Dumpleton
On 21 February 2012 21:07, Simon Sapin simon.sa...@exyr.org wrote:
 Le 21/02/2012 10:31, Graham Dumpleton a écrit :

 You do realise you are just reinventing context managers?

 With this 'application' do requests.

 Indeed. I didn’t want to go too far from the initial shutdown function
 proposal, but actual context managers would be better.

FWIW, I have been playing with context managers in other ways to solve
per request resource cleanups issues as well. I will cover some of
what I have been doing with that in my State of WSGI 2 talk at PyCon
web summit.

I sort of wish this whole discussion could perhaps wait until the web
summit where after my talk I can perhaps discuss with interested
parties all the stuff I have been playing with around improving WSGI
rather than taking shots at little bits now when there is a lot more
to consider than just this.

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Antoine Pitrou
Tarek Ziadé ziade.tarek@... writes:
 
 
 On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
graham.dumple...@gmail.com wrote:
 ...
  But I don't think you can guarantee that everything is still up in memory by
  the time atexit gets called,
  so you can't really call cleanup code there.
 The only thing which is done prior to atexit callbacks being called is
 waiting on threads which weren't marked as daemonised.
 
 
 which can lead to completely lock the shutdown if a lib or the program has a
 thread with a loop that waits for a condition.which it is not the case with
 signals, since you get a chance to properly stop everything beforehand.

That's a buggy lib or program. This has nothing to do with WSGI really. The
snippet Graham showed is run at any interpreter shutdown, even when you simply
run python in your shell.

Regards

Antoine.


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 1:43 PM, Antoine Pitrou solip...@pitrou.net wrote:

 Tarek Ziadé ziade.tarek@... writes:
 
 
  On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:
  ...
   But I don't think you can guarantee that everything is still up in
 memory by
   the time atexit gets called,
   so you can't really call cleanup code there.
  The only thing which is done prior to atexit callbacks being called is
  waiting on threads which weren't marked as daemonised.
 
 
  which can lead to completely lock the shutdown if a lib or the program
 has a
  thread with a loop that waits for a condition.which it is not the case
 with
  signals, since you get a chance to properly stop everything beforehand.

 That's a buggy lib or program. This has nothing to do with WSGI really.


No, that has to do with : please let me clean my program before you try to
kill it because I can't use signals :)



 The
 snippet Graham showed is run at any interpreter shutdown, even when you
 simply
 run python in your shell.


here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505

Run it with plain python, and try to ctrl-C it. You won't reach atexit and
will get locked.

(here: python 2.7 / mac os)

If you use signals instead of atexit, you'll have it working.

And this pattern (a thread in the background) is pretty common -- unless I
am missing something here


Cheers
Tarek



 Regards

 Antoine.


 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com




-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 2:46 PM, Tarek Ziadé ziade.ta...@gmail.com wrote:


 here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505


There are two typos but the effect remains the same since you are locked
before you reach those lines:

- atexit call worker.stop() instead  of worker.join()
- in worker.stop(), it calls worker.join() instead of worker.join(self)

-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Antoine Pitrou
Tarek Ziadé ziade.tarek@... writes:
 
 here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505
 
 There are two typos but the effect remains the same since you are locked
 before you reach those lines:
 - atexit call worker.stop() instead 
 of worker.join()
 - in worker.stop(), it calls worker.join() instead of
 worker.join(self)

I find your example convincing. I think that's worth fixing in Python (e.g. by 
offering an atexit() method on Thread objects). Perhaps you can open a bug?

Regards

Antoine.


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 3:02 PM, Antoine Pitrou solip...@pitrou.net wrote:

 ...
 I find your example convincing. I think that's worth fixing in Python
 (e.g. by
 offering an atexit() method on Thread objects). Perhaps you can open a bug?



Sure yeah --  Notice that I think it's still a good idea to provide the
shutdown function in WSGI, because it gives the full control to the web
server on the ordering of events.

Cheers
Tarek




 Regards

 Antoine.


 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com




-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Simon Sapin



Simon Sapin simon.sa...@exyr.org a écrit :

Both of these callbacks could be called once per process (aka. space 
where requests share memory.)

Sorry, that should be once per interpreter.
-- 
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-21 Thread Graham Dumpleton
If you want to be able to control a thread like that from an atexit
callback, you need to create the thread as daemonised. Ie.
setDaemon(True) call on thread.

By default a thread will actually inherit the daemon flag from the
parent. For a command line Python where thread created from main
thread it will not be daemonised and thus why the thread will be
waited upon on shutdown prior to atexit being called.

If you ran the same code in mod_wsgi, my memory is that the thread
will actually inherit as being daemonised because request handler in
mod_wsgi, from which import is trigger, are notionally daemonised.

Thus the code should work in mod_wsgi. Even so, to be portable, if
wanting to manipulate thread from atexit, make it daemonised.

Example of background threads in mod_wsgi at:

http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes

shows use of setDaemon().

Graham

On 22 February 2012 00:46, Tarek Ziadé ziade.ta...@gmail.com wrote:


 On Tue, Feb 21, 2012 at 1:43 PM, Antoine Pitrou solip...@pitrou.net wrote:

 Tarek Ziadé ziade.tarek@... writes:
 
 
  On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
 graham.dumple...@gmail.com wrote:
  ...
   But I don't think you can guarantee that everything is still up in
   memory by
   the time atexit gets called,
   so you can't really call cleanup code there.
  The only thing which is done prior to atexit callbacks being called is
  waiting on threads which weren't marked as daemonised.
 
 
  which can lead to completely lock the shutdown if a lib or the program
  has a
  thread with a loop that waits for a condition.which it is not the case
  with
  signals, since you get a chance to properly stop everything beforehand.

 That's a buggy lib or program. This has nothing to do with WSGI really.


 No, that has to do with : please let me clean my program before you try to
 kill it because I can't use signals :)



 The
 snippet Graham showed is run at any interpreter shutdown, even when you
 simply
 run python in your shell.


 here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505

 Run it with plain python, and try to ctrl-C it. You won't reach atexit and
 will get locked.

 (here: python 2.7 / mac os)

 If you use signals instead of atexit, you'll have it working.

 And this pattern (a thread in the background) is pretty common -- unless I
 am missing something here


 Cheers
 Tarek



 Regards

 Antoine.


 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com




 --
 Tarek Ziadé | http://ziade.org

 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Eric Larson
CherryPy provides a bus that allows you to add events to the web server 
process. It is specified pretty clearly and CherryPy recently made it available 
as a standalone package, Magicbus 
(https://bitbucket.org/cherrypy/magicbus/overview). Specifically it allows you 
to send events on different signals the main server process might get. You can 
also use it for a general bus within the app server, but at its most basic 
level, the goal was to make the stop/start/restart events easy to hook into.

I've found it to be really helpful for managing processes and wrote a simple 
supervisor-ish app called Dad using it (http://bitbucket.org/elarson/dad).

HTH

Eric  

--  
Eric Larson


On Monday, February 20, 2012 at 10:03 AM, Tarek Ziadé wrote:

 Hello
  
 I need to be able to call a function when the web application shuts down 
 (SIGTERM/SIGINT) -- the use case is to stop a background thread.
  
 I am currently using signals because it seems to be the most clean way to do 
 this. atexit is much trickier since you don't know when it's going to get 
 called and you might try to call objects that were garbage collected unless 
 you hack something to keep references alive.
  
 But signals are also tricky beasts since you may compete with other code that 
 are listening to them. For instance mod_wsgi don't like apps that have signal 
 handlers.
  
 Anyways, the bottom line is that the cleanest way to do this -- as per Chris 
 McDonough idea, would be to introduce in the WSGI protocol a shutdown 
 function the servers would be obligated to call before exiting.
  
 I am not sure yet about its arguments, maybe a signum + frame or simply an 
 exit code...
  
 But how do you like the idea ? That would solve for me the problem of having 
 to deal differently here depending on if I am called with mod_wsgi or 
 gunicorn or xxx
  
  
 Cheers
 Tarek
  
 --  
 Tarek Ziadé | http://ziade.org
 ___
 Web-SIG mailing list
 Web-SIG@python.org (mailto:Web-SIG@python.org)
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe: http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org



___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Tarek Ziadé
On Mon, Feb 20, 2012 at 8:09 PM, Eric Larson e...@ionrock.org wrote:

 CherryPy provides a bus that allows you to add events to the web server
 process. It is specified pretty clearly and CherryPy recently made it
 available as a standalone package, Magicbus (
 https://bitbucket.org/cherrypy/magicbus/overview). Specifically it allows
 you to send events on different signals the main server process might get.
 You can also use it for a general bus within the app server, but at its
 most basic level, the goal was to make the stop/start/restart events easy
 to hook into.

 I've found it to be really helpful for managing processes and wrote a
 simple supervisor-ish app called Dad using it (
 http://bitbucket.org/elarson/dad).


Thanks for the pointer -- that looks pretty neat

I would be more interested though, in defining an extension to the WSGI
standard

A rough example of what I am talking about:

If I take the wsgiref doc, here's an example of a minimal wsgi application
(myapp.py):

from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
status = '200 OK' # HTTP Status
headers = [('Content-type', 'text/plain')]
start_response(status, headers)
return [Hello World]
def main():
 return make_server('', 8000, hello_world_app)


That module can be run by any web server out there that understands WSGI.
For instance, with gunicorn I can do:

$ gunicorn myapp:main

What I am talking about is a second entry point for the shutdown - example:

from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
status = '200 OK' # HTTP Status
headers = [('Content-type', 'text/plain')]
start_response(status, headers)
return [Hello World]
def main():
   return make_server('', 8000, hello_world_app)

def shutdown():   # or maybe something else as an argument I don't know
   do_some_cleanup()


 And point the shutdown callable to a web server:

$ gunicorn myapp:main myapp:shutdown

If this is defined in the WSGI standard it means any wsgi web server could
call shutdown() , not only gunicorn

Cheers
Tarek



 HTH

 Eric

 --
 Eric Larson


 On Monday, February 20, 2012 at 10:03 AM, Tarek Ziadé wrote:

  Hello
 
  I need to be able to call a function when the web application shuts down
 (SIGTERM/SIGINT) -- the use case is to stop a background thread.
 
  I am currently using signals because it seems to be the most clean way
 to do this. atexit is much trickier since you don't know when it's going to
 get called and you might try to call objects that were garbage collected
 unless you hack something to keep references alive.
 
  But signals are also tricky beasts since you may compete with other code
 that are listening to them. For instance mod_wsgi don't like apps that have
 signal handlers.
 
  Anyways, the bottom line is that the cleanest way to do this -- as per
 Chris McDonough idea, would be to introduce in the WSGI protocol a
 shutdown function the servers would be obligated to call before exiting.
 
  I am not sure yet about its arguments, maybe a signum + frame or simply
 an exit code...
 
  But how do you like the idea ? That would solve for me the problem of
 having to deal differently here depending on if I am called with mod_wsgi
 or gunicorn or xxx
 
 
  Cheers
  Tarek
 
  --
  Tarek Ziadé | http://ziade.org
  ___
  Web-SIG mailing list
  Web-SIG@python.org (mailto:Web-SIG@python.org)
  Web SIG: http://www.python.org/sigs/web-sig
  Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org






-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Tarek Ziadé
oops my examples were broken, should be:

def hello_world_app(environ, start_response):
status = '200 OK' # HTTP Status
headers = [('Content-type', 'text/plain')]
start_response(status, headers)
return [Hello World]
def shutdown():   # or maybe something else as an argument I don't know
   do_some_cleanup()


and:

$ gunicorn myapp:hello_world_app myapp:shutdown



Cheers
Tarek
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread PJ Eby
The standard way to do this would be to define an optional server
extension API supplied in the environ; for example, a
'x-wsgiorg.register_shutdown' function.  The wsgi.org wiki used to be the
place to propose these sorts of things for standardization, but it appears
to no longer be a wiki, so the mailing list is probably a good place to
discuss such a proposal.

On Mon, Feb 20, 2012 at 2:30 PM, Tarek Ziadé ziade.ta...@gmail.com wrote:

 oops my examples were broken, should be:

 def hello_world_app(environ, start_response):
 status = '200 OK' # HTTP Status
 headers = [('Content-type', 'text/plain')]
 start_response(status, headers)
 return [Hello World]
 def shutdown():   # or maybe something else as an argument I don't know
do_some_cleanup()


 and:

 $ gunicorn myapp:hello_world_app myapp:shutdown



 Cheers
 Tarek

 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Chris McDonough
On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
 The standard way to do this would be to define an optional server
 extension API supplied in the environ; for example, a
 'x-wsgiorg.register_shutdown' function.

Unlikely, AFACIT, as shutdown may happen when no request is active.
Even if this somehow happened to not be the case, asking the application
to put it in the environ is not useful, as the environ can't really be
relied on to retain values up the call stack.

- C


   The wsgi.org wiki used to be the place to propose these sorts of
 things for standardization, but it appears to no longer be a wiki, so
 the mailing list is probably a good place to discuss such a proposal.
 
 On Mon, Feb 20, 2012 at 2:30 PM, Tarek Ziadé ziade.ta...@gmail.com
 wrote:
 oops my examples were broken, should be:
 
 def hello_world_app(environ, start_response): status = '200
 OK' # HTTP Status headers = [('Content-type', 'text/plain')]
 start_response(status, headers) return [Hello World] 
 
 def shutdown():   # or maybe something else as an argument I
 don't know
 do_some_cleanup()
 
 
 
 and:
 
 $ gunicorn myapp:hello_world_app myapp:shutdown
 
 
 
 Cheers
 Tarek
 
 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/pje%
 40telecommunity.com
 
 
 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe: http://mail.python.org/mailman/options/web-sig/chrism%40plope.com


___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Simon Sapin

Le 21/02/2012 01:18, Chris McDonough a écrit :

On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:

  The standard way to do this would be to define an optional server
  extension API supplied in the environ; for example, a
  'x-wsgiorg.register_shutdown' function.

Unlikely, AFACIT, as shutdown may happen when no request is active.
Even if this somehow happened to not be the case, asking the application
to put it in the environ is not useful, as the environ can't really be
relied on to retain values up the call stack.


Hi,

I like environ['x-wsgiorg.register_shutdown']. It would work without 
changes to WSGI itself.


I think that the idea is not to put your shutdown function in the 
environment and hope it stays there up the stack, but to register it 
by calling register_shutdown:


@environ.get('x-wsgiorg.register_shutdown', lambda f: f)
def do_cleanup():
pass

Also, a shutdown function would be used to clean up something that was 
set up in a request. So if the server shuts down without having ever 
served a request, there probably is nothing to clean up.


Regards,
--
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Graham Dumpleton
On 21 February 2012 12:03, Simon Sapin simon.sa...@exyr.org wrote:
 Le 21/02/2012 01:18, Chris McDonough a écrit :

 On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:

   The standard way to do this would be to define an optional server
   extension API supplied in the environ; for example, a
   'x-wsgiorg.register_shutdown' function.

 Unlikely, AFACIT, as shutdown may happen when no request is active.
 Even if this somehow happened to not be the case, asking the application
 to put it in the environ is not useful, as the environ can't really be
 relied on to retain values up the call stack.


 Hi,

 I like environ['x-wsgiorg.register_shutdown']. It would work without changes
 to WSGI itself.

 I think that the idea is not to put your shutdown function in the
 environment and hope it stays there up the stack, but to register it by
 calling register_shutdown:

 @environ.get('x-wsgiorg.register_shutdown', lambda f: f)
 def do_cleanup():
    pass

 Also, a shutdown function would be used to clean up something that was set
 up in a request. So if the server shuts down without having ever served a
 request, there probably is nothing to clean up.

Using environ is not going to work it is supplied on a per request basis.

You would typically want an application scope cleanup handler to only
be registered once.

In this scheme you are relying on it being registered from within a
request scope.

To ensure that it is only registered once, the caller would need to
use a flag protected by a thread mutex to know whether should call a
second time, which is cumbersome.

If you don't do that you could end up registering a separate callback
for every single request that occurs and memory usage alone would blow
out just from recording them all.

Alternatively, you would have to require the underlying WSGI
server/adapter to weed out duplicates, but even if you do that, you
still waste the time of the per request scope registering it all the
time.

Even if you have a registration mechanism, especially with a WSGI
adapter riding on top of something else, how is the WSGI adapter going
to get notified to call them.

All you have therefore done is shift the problem of how it is
triggered somewhere else.

Overall the best chance of being able to do anything is relying on atexit.

You are though at the mercy of the WSGI hosting mechanism shutting
down the process and so the interpreter, in an orderly manner such
that atexit callbacks get called.

In Apache/mod_wsgi you get this guarantee, even in sub interpreters
where atexit callbacks wouldn't normally be called when they are
destroyed.

For uWSGI, atexit callbacks will not be called at the moment, by
Robert is making changes to it so you get a guarantee there as well.
It is possible he is only doing this though for case where main
interpreter is being used, as doing it for sub interpreters is a bit
fiddly.

Any pure Python WSGI servers shouldn't have issues so long as they
aren't force exiting the whole process and bypassing normal
interpreter destruction.

Graham
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread PJ Eby
2012/2/20 Chris McDonough chr...@plope.com

 On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
  The standard way to do this would be to define an optional server
  extension API supplied in the environ; for example, a
  'x-wsgiorg.register_shutdown' function.

 Unlikely, AFACIT, as shutdown may happen when no request is active.
 Even if this somehow happened to not be the case, asking the application
 to put it in the environ is not useful, as the environ can't really be
 relied on to retain values up the call stack.


Optional server extension APIs are things that the server puts in the
environ, not things the app puts there.  That's why it's
'register_shutdown', e.g.
environ['x-wsgiorg.register_shutdown'](shutdown_function).



 - C


The wsgi.org wiki used to be the place to propose these sorts of
  things for standardization, but it appears to no longer be a wiki, so
  the mailing list is probably a good place to discuss such a proposal.
 
  On Mon, Feb 20, 2012 at 2:30 PM, Tarek Ziadé ziade.ta...@gmail.com
  wrote:
  oops my examples were broken, should be:
 
  def hello_world_app(environ, start_response): status = '200
  OK' # HTTP Status headers = [('Content-type', 'text/plain')]
  start_response(status, headers) return [Hello World]
 
  def shutdown():   # or maybe something else as an argument I
  don't know
  do_some_cleanup()
 
 
 
  and:
 
  $ gunicorn myapp:hello_world_app myapp:shutdown
 
 
 
  Cheers
  Tarek
 
  ___
  Web-SIG mailing list
  Web-SIG@python.org
  Web SIG: http://www.python.org/sigs/web-sig
  Unsubscribe:
  http://mail.python.org/mailman/options/web-sig/pje%
  40telecommunity.com
 
 
  ___
  Web-SIG mailing list
  Web-SIG@python.org
  Web SIG: http://www.python.org/sigs/web-sig
  Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/chrism%40plope.com



___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Chris McDonough
On Mon, 2012-02-20 at 20:54 -0500, PJ Eby wrote:
 2012/2/20 Chris McDonough chr...@plope.com
 On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
  The standard way to do this would be to define an optional
 server
  extension API supplied in the environ; for example, a
  'x-wsgiorg.register_shutdown' function.
 
 
 Unlikely, AFACIT, as shutdown may happen when no request is
 active.
 Even if this somehow happened to not be the case, asking the
 application
 to put it in the environ is not useful, as the environ can't
 really be
 relied on to retain values up the call stack.
 
 
 Optional server extension APIs are things that the server puts in
 the environ, not things the app puts there.  That's why it's
 'register_shutdown', e.g.
 environ['x-wsgiorg.register_shutdown'](shutdown_function).  

I get it now, but it's still not the right thing I don't think.  Servers
shut down without issuing any requests at all.

- C



___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Tarek Ziadé
2012/2/21 Chris McDonough chr...@plope.com

 On Mon, 2012-02-20 at 20:54 -0500, PJ Eby wrote:
  2012/2/20 Chris McDonough chr...@plope.com
  On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
   The standard way to do this would be to define an optional
  server
   extension API supplied in the environ; for example, a
   'x-wsgiorg.register_shutdown' function.
 
 
  Unlikely, AFACIT, as shutdown may happen when no request is
  active.
  Even if this somehow happened to not be the case, asking the
  application
  to put it in the environ is not useful, as the environ can't
  really be
  relied on to retain values up the call stack.
 
 
  Optional server extension APIs are things that the server puts in
  the environ, not things the app puts there.  That's why it's
  'register_shutdown', e.g.
  environ['x-wsgiorg.register_shutdown'](shutdown_function).

 I get it now, but it's still not the right thing I don't think.  Servers
 shut down without issuing any requests at all.


Yes, I also think shutting down the server is completely orthogonal to
requests.

Maybe another option would be to call the application with the usual
callable, but an ending request that's a signal for the application about
being shut down.

When the app receives that very specific request, it would do the cleaning
job. It sounds hackish but would work without changing the standard





 - C






-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Tarek Ziadé
On Tue, Feb 21, 2012 at 2:39 AM, Graham Dumpleton 
graham.dumple...@gmail.com wrote:

 ...
 Overall the best chance of being able to do anything is relying on atexit.

 You are though at the mercy of the WSGI hosting mechanism shutting
 down the process and so the interpreter, in an orderly manner such
 that atexit callbacks get called.

 In Apache/mod_wsgi you get this guarantee, even in sub interpreters
 where atexit callbacks wouldn't normally be called when they are
 destroyed.

 For uWSGI, atexit callbacks will not be called at the moment, by
 Robert is making changes to it so you get a guarantee there as well.
 It is possible he is only doing this though for case where main
 interpreter is being used, as doing it for sub interpreters is a bit
 fiddly.


But I don't think you can guarantee that everything is still up in memory
by the time atexit gets called,
so you can't really call cleanup code there.


Any pure Python WSGI servers shouldn't have issues so long as they
 aren't force exiting the whole process and bypassing normal
 interpreter destruction.


what do you mean by bypassing its destruction ?

Cheers
Tarek




 Graham
 ___
 Web-SIG mailing list
 Web-SIG@python.org
 Web SIG: http://www.python.org/sigs/web-sig
 Unsubscribe:
 http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com




-- 
Tarek Ziadé | http://ziade.org
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com


Re: [Web-SIG] A 'shutdown' function in WSGI

2012-02-20 Thread Simon Sapin

Le 21/02/2012 08:47, Tarek Ziadé a écrit :

Yes, I also think shutting down the server is completely orthogonal to
requests.


If the shutdown callback is next to the application and not registered 
in a request, why not also have the symmetric server start up callback 
that would not wait for a request? This would avoid workarounds like 
Flask.before_first_request.


Both of these callbacks could be called once per process (aka. space 
where requests share memory.)


Instead of having to provide two or three objects separately to a 
server, how about making the callbacks attributes of the application 
callable?


Regards,
--
Simon Sapin
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com