Thank you very much for you response. I will explore it, however, it is hard fo me to imagine, how to insert my long-running functions into manager class.
On Thursday, February 26, 2015 at 12:15:28 PM UTC+2, Graham Dumpleton wrote: > > You will need to install latest mod_wsgi from git repo as I made some > tweaks to make this easier. > > So on your own personal system (not Webfaction) so you can experiment, do: > > pip install -U > https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip > > Now create three files. > > The first called 'task-queue-client.py'. > > This contains the following and is a simple WSGI hello world program. > > import os > > from multiprocessing.managers import BaseManager > > class MyManager(BaseManager): > pass > > MyManager.register('Maths') > > sockpath = os.path.join(os.path.dirname(__file__), > 'task-queue-manager.sock') > > def application(environ, start_response): > status = '200 OK' > output = b'Hello World!' > > m = MyManager(address=sockpath, authkey='abracadabra') > m.connect() > > maths = m.Maths() > print maths.add(1, 2) > > response_headers = [('Content-type', 'text/plain'), > ('Content-Length', str(len(output)))] > start_response(status, response_headers) > > return [output] > > The second called 'task-queue-manager.py'. > > import os > > from tasks import MyManager > > sockpath = os.path.join(os.path.dirname(__file__), > 'task-queue-manager.sock') > > try: > os.unlink(sockpath) > except OSError: > pass > > m = MyManager(address=sockpath, authkey='abracadabra') > s = m.get_server() > s.serve_forever() > > And finally 'tasks.py'. This can't be in 'task-queue-manager.py' as can't > have functions you want to pickle in the service script because of how > mod_wsgi names modules. > > import os > import multiprocessing > import multiprocessing.managers > import signal > import time > import os > > class TimeoutException(Exception): > pass > > class RunableProcessing(multiprocessing.Process): > def __init__(self, func, *args, **kwargs): > self.queue = multiprocessing.Queue(maxsize=1) > args = (func,) + args > super(RunableProcessing, self).__init__(target=self.run_func, > args=args, kwargs=kwargs) > > def run_func(self, func, *args, **kwargs): > try: > result = func(*args, **kwargs) > self.queue.put((True, result)) > except Exception as e: > self.queue.put((False, e)) > > def done(self): > return self.queue.full() > > def result(self): > return self.queue.get() > > def timeout(seconds, force_kill=True): > def wrapper(function): > def inner(*args, **kwargs): > now = time.time() > proc = RunableProcessing(function, *args, **kwargs) > proc.start() > proc.join(seconds) > if proc.is_alive(): > print 'still alive' > if force_kill: > print 'kill it' > proc.terminate() > proc.join(3) > print 'is it dead', proc.is_alive() > if proc.is_alive(): > try: > print 'force kill' > os.kill(proc.pid, signal.SIGKILL) > except Exception: > pass > proc.join(1) > print 'alive', proc.is_alive() > runtime = int(time.time() - now) > raise TimeoutException('timed out after {0} > seconds'.format(runtime)) > assert proc.done() > success, result = proc.result() > if success: > return result > else: > raise result > return inner > return wrapper > > class MathsClass(object): > @timeout(5.0) > def add(self, x, y): > print os.getpid(), 'add', x, y > return x + y > @timeout(5.0) > def mul(self, x, y): > print os.getpid(), 'mul', x, y > return x * y > > class MyManager(multiprocessing.managers.BaseManager): > pass > > MyManager.register('Maths', MathsClass) > > With those all in the same directory run with that latest mod_wsgi repo > version: > > mod_wsgi-express start-server task-queue-client.py --service-script > tasks task-queue-manager.py > > When you start the server what will happen is that in addition to creating > a process to run your WSGI application, mod_wsgi-express will create an > extra daemon process labelled as 'tasks'. This will not handle any web > requests. Instead into that process will be loaded > the task-queue-manager.py script. > > So Apache/mod_wsgi is simply acting as a process supervisor for that > process and will keep it running. > > The task-queue-manager.py script is going to use multiprocessing module to > run a server that accepts client requests made from some other process > using the multiprocessing module mechanisms for remote calls. > > When that call is made from the client code, the manager service using > your existing timeout decorator, will actually run the target function in a > forked subprocess rather than the manager process. When the result is > available it gets pulled back from the forked worker process into the > manager process and then back across to the client. > > So the key thing here is that the manager process which is running the > worker as forked subprocesses, is an independent process to your web > application. Don't include your web application into the manager. Keep the > manager to the bare minimum of code it just needs to run your algorithm, or > even delay importing that code until needed in the forked sub process when > function runs. That way it will be light on memory. > > So look very carefully through that and study > what multiprocessing.managers.BaseManager is all about. > > When you totally understand it, look at working out how to integrate that > into your web application. > > Remember that when trying to use on WebFaction, first update mod_wsgi to > that repo version. I am not sure when the next official version will be > just yet, bogged down in understanding some SSL stuff. > > Also add: > > --service-script tasks path/to/python/script/file > > to express-config you are using to manage generation of Apache config, > regenerate config and restart server. > > When I get a chance again, I will play some more and see if can't make > that RunableProcessing simpler. Not sure whether it really needs to be that > complicated. > > Graham > > On 24/02/2015, at 6:24 AM, Paul Royik <[email protected] <javascript:>> > wrote: > > Hello. > I explored that, but still don't understand how to properly use it. > If you have time, please, help me. > > On Sunday, February 22, 2015 at 1:03:57 PM UTC+2, Paul Royik wrote: >> >> OK. >> I will explore that. >> Thanks. >> >> On Sunday, February 22, 2015 at 12:55:20 PM UTC+2, Graham Dumpleton wrote: >> >> I can only offer you a direction of where to look right now. I have some >> things I have to deal with over the new few days and so you may not get a >> quick response. >> >> On signals and using SIGALRM in particular, no that cannot be used. >> >> As far as implementing a simple task execution system, start researching >> the examples in: >> >> https://docs.python.org/2/library/multiprocessing.html#customized-managers >> >> https://docs.python.org/2/library/multiprocessing.html#using-a-remote-manager >> >> I believe that functionality from the multiprocessing module could be >> used, but I simply don't have the time to be able to explain it right now. >> >> Graham >> >> On 22/02/2015, at 9:00 AM, Paul Royik <[email protected]> wrote: >> >> OK. Let's summarize it. >> I have an external function and want to interrupt it in the cleanest and >> most natural way. >> I'm newbie to all this threading and multiprocessing. >> You gave me multiprocessing code and then warned that it might work not >> very clean. >> >> Can you tell me possible way and give all pros and cons, so I can make >> some conclusion? >> You said something about separate task system? I have no idea how to >> implement it. All python solutions seemed to be around multithreading, >> signal, multiprocessing. >> By the way, will signal solution work with new mod_wsgi? What are pros >> and cons? >> >> import signal >> def signal_handler(signum, frame): >> raise Exception("Timed out!") >> >> signal.signal(signal.SIGALRM, signal_handler) >> signal.alarm(10) # Ten secondstry: >> long_function_call()except Exception, msg: >> print "Timed out!" >> >> >> Thank you for your time. >> >> On Saturday, February 21, 2015 at 10:44:22 PM UTC+2, Paul Royik wrote: >> >> *Which is part of the reason why I said it isn't a good idea to be >> forking worker subprocesses from your web application to do work. When you >> do that, the starting memory usage will be whatever the process was that >> was handling the web request to begin with. Your algorithm will then add on >> top of that.* >> >> *So if you web application has very fat memory usage, that carries >> through.* >> >> *This is why I was saying that you are better off with a separate task >> execution system. Its processes would have very small amount of memory >> usage to start with as it doesn't have all your web application memory as >> well. The separate task execution system can also control how many parallel >> tasks run and so limit overall memory usage.* >> >> *So, this seems not very good solution. I have only 512 MB. How to >> implement separate task system?* >> >> *Concerning your last question. I use FTP manager like Filezilla and that >> time I reuploaded all project (even that wsgi.py wasn't changed).* >> >> >> On Saturday, February 21, 2015 at 10:00:53 PM UTC+2, Graham Dumpleton >> wrote: >> >> On 22/02/2015, at 5:33 AM, Paul Royik <[email protected]> wrote: >> >> From time to time I've got following error: >> [Sat Feb 21 12:24:27.181049 2015] [wsgi:info] [pid 5482:tid >> 139829356984064] [remote >> >> ... > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
