That's a good idea, and one I hadn't considered for some reason. Twisted has a very simple xmlrpc server/client lib and it would certainly be easier to do that than trying to figure this mess out ;)
Thanks for your help, Graham! On Feb 17, 11:31 pm, Graham Dumpleton <[email protected]> wrote: > 2009/2/18 Tom Davis <[email protected]>: > > > > > > >> It is a bit antisocial of library type component modules to think they > >> control the process execution environment and register signals. :-( > > >> Was it only for signal 15? Is that SIGTERM on your operating system? > > >> Are you using embedded mode or daemon mode? There are currently issues > >> with creating sub processes from daemon mode processes where sub > >> processes need to handle signals. > > > True, it is a bit anti-social, but Twisted is a controlling beast ;) > > > Other signals I noticed were "2" and "20". I use embedded mode, > > though I tried out daemon mode just for kicks and it did not garner > > different results. > > >> Any particular reason you are using Twisted just to do HTTP requests > >> rather than simpler Python HTTP client modules. > > > The functionality this method provides is a sort of "live update" via > > numerous third-party web APIs. To make these requests synchronously > > would take far more time since the requests need to be made, parsed, > > then inserted into a database. Meanwhile, users are waiting for the > > results of an "AJAX" call to view the requested data. > > > I could make these calls asynchronously outside of the main thread, > > perhaps on another machine not tied to the oddities of mod_wsgi and > > apache, but then I wouldn't know *when* they completed. I could call > > them using subprocess.call as well and wait for the results, but this > > would involve importing all the necessary modules each request which > > adds precious time to the transaction as well. These are the reasons I > > have chosen to do it asynchronously using Twisted running in a > > separate process. > > Have you considered running Twisted process by itself with XML-RPC > service enabled so could call into it using xmlrpclib from web > application processes under mod_wsgi. The XML-RPC calling interface on > client side is really quite simple. Not sure about the Twisted side > though. The XML-RPC interface also gives you a separate admin > interface as well if you need to control it from management scripts. > > Graham > > > Thank you for the very quick replies and your continued efforts! > > > On Feb 17, 11:03 pm, Graham Dumpleton <[email protected]> > > wrote: > >> 2009/2/18 Tom Davis <[email protected]>: > > >> > Graham, > > >> > I did as you recommended and got the following output in error_log: > > >> > subprocess pid 30467 > >> > hello grumpy > > >> > However, following that were various warnings about signal > >> > registration being ignored, as it attempted to do the rest of the > >> > function (I tacked your code onto the beginning): > > >> > [Tue Feb 17 21:48:32 2009] [warn] mod_wsgi (pid=30467): Callback > >> > registration for signal 15 ignored. > > >> It is a bit antisocial of library type component modules to think they > >> control the process execution environment and register signals. :-( > > >> Was it only for signal 15? Is that SIGTERM on your operating system? > > >> Are you using embedded mode or daemon mode? There are currently issues > >> with creating sub processes from daemon mode processes where sub > >> processes need to handle signals. > > >> > So, it would appear that the child is being spawned/run, but only in > >> > the simplest sense. The mod_wsgi rules seem to make it ignore > >> > signals. If I add back WSGIRestrictSignal Off, the warnings go away, > >> > but the function still doesn't *do* anything. The purpose of the > >> > target function is to make some http requests (via Twisted) then parse > >> > those requests, which involves running the Twisted reactor (the entire > >> > reason I am spawning a new process; the reactor cannot run within a > >> > mod_wsgi thread). It seems this is causing a problem for some reason. > >> > Obviously the target function is being run, as evidenced by the output > >> > in error_log, but Twisted is having trouble with signaling within it > >> > (at least, that is my only guess). As I said, this only seems to be > >> > affected when running w/ mod_wsgi; any other time the function works > >> > fine. > > >> Any particular reason you are using Twisted just to do HTTP requests > >> rather than simpler Python HTTP client modules. > > >> Anyway, let me have a think about it for a while. > > >> Graham > > >> > On Feb 17, 10:03 pm, Graham Dumpleton <[email protected]> > >> > wrote: > >> >> 2009/2/18 Tom Davis <[email protected]>: > > >> >> > I am trying to use multiprocessing within a modwsgi app and am having > >> >> > an issue. At first I was getting errors regarding permissions to use > >> >> > stdout, etc. I fixed these by setting WSGIRestrictStdout, > >> >> > WSGIRestrictStdin, and WSGIRestrictSignal to Off in my conf file. Now > >> >> > I get no errors, but the process doesn't actually run. My usage is > >> >> > quite simple: on a page request, I try something along the lines of: > > >> >> > p = multiprocessing.Process(target=fn, args...) > >> >> > p.start() > >> >> > p.join() > > >> >> > The target doesn't output anything, but if run properly it should add > >> >> > some records to the database. I tested that it works normally by > >> >> > running the app under the django development server, which doesn't > >> >> > seem to have an issue with multiprocessing. Has anyone encountered > >> >> > this before? > > >> >> What does the target function do? > > >> >> Try following WSGI script file. Note that for this script you will > >> >> need to look in main Apache error log for output, even if WSGI script > >> >> defined in VirtualHost context and that VirtualHost has its own error > >> >> log. This is because am reseting stdin/stdout/stderr back to original > >> >> values. Take out the WSGIRestrictStdin, WSGIRestrictStdout and > >> >> WSGIRestrictSignal directives you already added. > > >> >> from multiprocessing import Process > > >> >> import sys, os > > >> >> sys.stdin = sys.__stdin__ > >> >> sys.stdout = sys.__stdout__ > >> >> sys.stderr = sys.__stderr__ > > >> >> def f(name): > >> >> sys.stderr.write('subprocess pid %d\n' % os.getpid()) > >> >> sys.stderr.write('hello %s\n' % name) > >> >> sys.stderr.flush() > > >> >> def application(environ, start_response): > >> >> status = '200 OK' > >> >> output = 'Hello World!' > > >> >> sys.stderr.write('parent pid %d\n' % os.getpid()) > >> >> sys.stderr.flush() > > >> >> p = Process(target=f, args=('grumpy',)) > >> >> p.start() > >> >> p.join() > > >> >> response_headers = [('Content-type', 'text/plain'), > >> >> ('Content-Length', str(len(output)))] > >> >> start_response(status, response_headers) > > >> >> return [output] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
