BTW, forgot to mention that should try forcing WSGI application to run in main interpreter. See:
http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API If Twisted as C extension module components and they use simplified GIL API, will only work in main interpreter. I need to look more at how multiprocessing module works, but if it is only doing a fork and runs in same thread, then it may not even be usable in Apache/mod_wsgi context as various odd things are done with SystemExit exception. Ie., caught and ignored. So, not sure how you would get process to shutdown properly. This is on top of all the weird sub interpreter shit it does. Graham 2009/2/19 Tom Davis <[email protected]>: > > That's a good idea, and one I hadn't considered for some reason. > Twisted has a very simple xmlrpc server/client lib and it would > certainly be easier to do that than trying to figure this mess out ;) > > Thanks for your help, Graham! > > On Feb 17, 11:31 pm, Graham Dumpleton <[email protected]> > wrote: >> 2009/2/18 Tom Davis <[email protected]>: >> >> >> >> >> >> >> It is a bit antisocial of library type component modules to think they >> >> control the process execution environment and register signals. :-( >> >> >> Was it only for signal 15? Is that SIGTERM on your operating system? >> >> >> Are you using embedded mode or daemon mode? There are currently issues >> >> with creating sub processes from daemon mode processes where sub >> >> processes need to handle signals. >> >> > True, it is a bit anti-social, but Twisted is a controlling beast ;) >> >> > Other signals I noticed were "2" and "20". I use embedded mode, >> > though I tried out daemon mode just for kicks and it did not garner >> > different results. >> >> >> Any particular reason you are using Twisted just to do HTTP requests >> >> rather than simpler Python HTTP client modules. >> >> > The functionality this method provides is a sort of "live update" via >> > numerous third-party web APIs. To make these requests synchronously >> > would take far more time since the requests need to be made, parsed, >> > then inserted into a database. Meanwhile, users are waiting for the >> > results of an "AJAX" call to view the requested data. >> >> > I could make these calls asynchronously outside of the main thread, >> > perhaps on another machine not tied to the oddities of mod_wsgi and >> > apache, but then I wouldn't know *when* they completed. I could call >> > them using subprocess.call as well and wait for the results, but this >> > would involve importing all the necessary modules each request which >> > adds precious time to the transaction as well. These are the reasons I >> > have chosen to do it asynchronously using Twisted running in a >> > separate process. >> >> Have you considered running Twisted process by itself with XML-RPC >> service enabled so could call into it using xmlrpclib from web >> application processes under mod_wsgi. The XML-RPC calling interface on >> client side is really quite simple. Not sure about the Twisted side >> though. The XML-RPC interface also gives you a separate admin >> interface as well if you need to control it from management scripts. >> >> Graham >> >> > Thank you for the very quick replies and your continued efforts! >> >> > On Feb 17, 11:03 pm, Graham Dumpleton <[email protected]> >> > wrote: >> >> 2009/2/18 Tom Davis <[email protected]>: >> >> >> > Graham, >> >> >> > I did as you recommended and got the following output in error_log: >> >> >> > subprocess pid 30467 >> >> > hello grumpy >> >> >> > However, following that were various warnings about signal >> >> > registration being ignored, as it attempted to do the rest of the >> >> > function (I tacked your code onto the beginning): >> >> >> > [Tue Feb 17 21:48:32 2009] [warn] mod_wsgi (pid=30467): Callback >> >> > registration for signal 15 ignored. >> >> >> It is a bit antisocial of library type component modules to think they >> >> control the process execution environment and register signals. :-( >> >> >> Was it only for signal 15? Is that SIGTERM on your operating system? >> >> >> Are you using embedded mode or daemon mode? There are currently issues >> >> with creating sub processes from daemon mode processes where sub >> >> processes need to handle signals. >> >> >> > So, it would appear that the child is being spawned/run, but only in >> >> > the simplest sense. The mod_wsgi rules seem to make it ignore >> >> > signals. If I add back WSGIRestrictSignal Off, the warnings go away, >> >> > but the function still doesn't *do* anything. The purpose of the >> >> > target function is to make some http requests (via Twisted) then parse >> >> > those requests, which involves running the Twisted reactor (the entire >> >> > reason I am spawning a new process; the reactor cannot run within a >> >> > mod_wsgi thread). It seems this is causing a problem for some reason. >> >> > Obviously the target function is being run, as evidenced by the output >> >> > in error_log, but Twisted is having trouble with signaling within it >> >> > (at least, that is my only guess). As I said, this only seems to be >> >> > affected when running w/ mod_wsgi; any other time the function works >> >> > fine. >> >> >> Any particular reason you are using Twisted just to do HTTP requests >> >> rather than simpler Python HTTP client modules. >> >> >> Anyway, let me have a think about it for a while. >> >> >> Graham >> >> >> > On Feb 17, 10:03 pm, Graham Dumpleton <[email protected]> >> >> > wrote: >> >> >> 2009/2/18 Tom Davis <[email protected]>: >> >> >> >> > I am trying to use multiprocessing within a modwsgi app and am having >> >> >> > an issue. At first I was getting errors regarding permissions to use >> >> >> > stdout, etc. I fixed these by setting WSGIRestrictStdout, >> >> >> > WSGIRestrictStdin, and WSGIRestrictSignal to Off in my conf file. Now >> >> >> > I get no errors, but the process doesn't actually run. My usage is >> >> >> > quite simple: on a page request, I try something along the lines of: >> >> >> >> > p = multiprocessing.Process(target=fn, args...) >> >> >> > p.start() >> >> >> > p.join() >> >> >> >> > The target doesn't output anything, but if run properly it should add >> >> >> > some records to the database. I tested that it works normally by >> >> >> > running the app under the django development server, which doesn't >> >> >> > seem to have an issue with multiprocessing. Has anyone encountered >> >> >> > this before? >> >> >> >> What does the target function do? >> >> >> >> Try following WSGI script file. Note that for this script you will >> >> >> need to look in main Apache error log for output, even if WSGI script >> >> >> defined in VirtualHost context and that VirtualHost has its own error >> >> >> log. This is because am reseting stdin/stdout/stderr back to original >> >> >> values. Take out the WSGIRestrictStdin, WSGIRestrictStdout and >> >> >> WSGIRestrictSignal directives you already added. >> >> >> >> from multiprocessing import Process >> >> >> >> import sys, os >> >> >> >> sys.stdin = sys.__stdin__ >> >> >> sys.stdout = sys.__stdout__ >> >> >> sys.stderr = sys.__stderr__ >> >> >> >> def f(name): >> >> >> sys.stderr.write('subprocess pid %d\n' % os.getpid()) >> >> >> sys.stderr.write('hello %s\n' % name) >> >> >> sys.stderr.flush() >> >> >> >> def application(environ, start_response): >> >> >> status = '200 OK' >> >> >> output = 'Hello World!' >> >> >> >> sys.stderr.write('parent pid %d\n' % os.getpid()) >> >> >> sys.stderr.flush() >> >> >> >> p = Process(target=f, args=('grumpy',)) >> >> >> p.start() >> >> >> p.join() >> >> >> >> response_headers = [('Content-type', 'text/plain'), >> >> >> ('Content-Length', str(len(output)))] >> >> >> start_response(status, response_headers) >> >> >> >> return [output] > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
