I can only offer you a direction of where to look right now. I have some things I have to deal with over the new few days and so you may not get a quick response.
On signals and using SIGALRM in particular, no that cannot be used. As far as implementing a simple task execution system, start researching the examples in: https://docs.python.org/2/library/multiprocessing.html#customized-managers https://docs.python.org/2/library/multiprocessing.html#using-a-remote-manager I believe that functionality from the multiprocessing module could be used, but I simply don't have the time to be able to explain it right now. Graham On 22/02/2015, at 9:00 AM, Paul Royik <[email protected]> wrote: > OK. Let's summarize it. > I have an external function and want to interrupt it in the cleanest and most > natural way. > I'm newbie to all this threading and multiprocessing. > You gave me multiprocessing code and then warned that it might work not very > clean. > > Can you tell me possible way and give all pros and cons, so I can make some > conclusion? > You said something about separate task system? I have no idea how to > implement it. All python solutions seemed to be around multithreading, > signal, multiprocessing. > By the way, will signal solution work with new mod_wsgi? What are pros and > cons? > > import signal > > def signal_handler(signum, frame): > raise Exception("Timed out!") > > signal.signal(signal.SIGALRM, signal_handler) > signal.alarm(10) # Ten seconds > try: > long_function_call() > except Exception, msg: > print "Timed out!" > > Thank you for your time. > > On Saturday, February 21, 2015 at 10:44:22 PM UTC+2, Paul Royik wrote: > Which is part of the reason why I said it isn't a good idea to be forking > worker subprocesses from your web application to do work. When you do that, > the starting memory usage will be whatever the process was that was handling > the web request to begin with. Your algorithm will then add on top of that. > > So if you web application has very fat memory usage, that carries through. > > This is why I was saying that you are better off with a separate task > execution system. Its processes would have very small amount of memory usage > to start with as it doesn't have all your web application memory as well. The > separate task execution system can also control how many parallel tasks run > and so limit overall memory usage. > > So, this seems not very good solution. I have only 512 MB. How to implement > separate task system? > > Concerning your last question. I use FTP manager like Filezilla and that time > I reuploaded all project (even that wsgi.py wasn't changed). > > > On Saturday, February 21, 2015 at 10:00:53 PM UTC+2, Graham Dumpleton wrote: > On 22/02/2015, at 5:33 AM, Paul Royik <[email protected]> wrote: > > From time to time I've got following error: > [Sat Feb 21 12:24:27.181049 2015] [wsgi:info] [pid 5482:tid 139829356984064] > [remote 127.0.0.1:12680] mod_wsgi (pid=5482): Force restart of process > 'localhost:20241'. > > Is it normal? > > That means that the WSGI script file for your application was modified. This > is detected and the mod_wsgi daemon process restarted automatically. > > This is another reason why you shouldn't be doing stuff under web application > processes. > > When this occurs, if you had an algorithm worker process running at the time, > the killing off of the daemon process running your web application may not > result in the worker being killed and so it could keep running with nothing > then killing it. > > Will have to check that though, as depends on whether when daemon process is > killed that children also killed automatically. > > How do you upload code changes to your system? Are you just using ssh to copy > your whole code base every time and so the WSGI script file modification time > will be updated even if not changed? > > Graham > > On Saturday, February 21, 2015 at 3:14:16 PM UTC+2, Paul Royik wrote: > Thank you very much. > It works now. > But I noticed one thing. When calculator works, memory jumps from 180 to 250 > (and the back to 180 as expected). When I used thread method, I didn't see > such big jump. Can you tell me why in this case the jump is so big? I guess, > that if 10 people will do it simulaneously, then I will run out of memory. > > Anyway, this is the cleanest and best solution. > > > On Saturday, February 21, 2015 at 2:26:40 PM UTC+2, Graham Dumpleton wrote: > The short answer is that code will not work with the version of mod_wsgi you > are using. It will not kill the forked process properly and would actually > kill the WSGI application daemon process instead, leaving the forked process > still running and thus why it consumed memory. That it was killing the WSGI > application daemon process would have been evident in the error logs which is > why I asked about logs. > > Some extra tricks were required with a special directive needing to be given > to mod_wsgi for the version you are using to allow signal handlers to be > overridden and the forked process would need to reset the signal handlers > before running the job. > > Since it was a pain to have to use the fiddles I worked out a way in mod_wsgi > to avoid needing to do all that. You will need to upgrade to the latest > mod_wsgi version though, which is 4.4.9. > > That isn't all that is required though as that code doesn't properly wait on > the process when terminating it. This will result in a zombie process hanging > around, which isn't a good idea. In worst case, since terminate() only tries > SIGTERM, it may not go away. > > Now you can't simply go run 'pip install -U mod_wsgi' because of the issue we > had with that breaking the mod_wsgi-httpd install. That issue has also now > been fixed though. What you therefore want to do is reinstall both > mod_wsgi-httpd and mod_wsgi. > > So if I got it right, you should be able to do. > > pip2.7 -v -v -v --log $HOME/pip.log install -U --user --build $HOME/tmp/pip > mod_wsgi-httpd > pip2.7 install -U --user mod_wsgi > > I am hoping that doesn't effect your running site. To be safe you may want to > shut it down when doing the update. > > Once upgraded, use the following code. Remove print statements when happy. I > would get rid of the force_kill flag as you probably always want to be force > killing it and escalating that to SIGKILL to definitely get rid of it. > > import multiprocessing > import signal > import time > import os > > class TimeoutException(Exception): > pass > > class RunableProcessing(multiprocessing.Process): > def __init__(self, func, *args, **kwargs): > self.queue = multiprocessing.Queue(maxsize=1) > args = (func,) + args > multiprocessing.Process.__init__(self, target=self.run_func, > args=args, kwargs=kwargs) > > def run_func(self, func, *args, **kwargs): > try: > result = func(*args, **kwargs) > self.queue.put((True, result)) > except Exception as e: > self.queue.put((False, e)) > > def done(self): > return self.queue.full() > > def result(self): > return self.queue.get() > > def timeout(seconds, force_kill=True): > def wrapper(function): > def inner(*args, **kwargs): > now = time.time() > proc = RunableProcessing(function, *args, **kwargs) > proc.start() > proc.join(seconds) > if proc.is_alive(): > print 'still alive' > if force_kill: > print 'kill it' > proc.terminate() > proc.join(3) > print 'is it dead', proc.is_alive() > if proc.is_alive(): > try: > > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
