OK. Let's summarize it.
I have an external function and want to interrupt it in the cleanest and
most natural way.
I'm newbie to all this threading and multiprocessing.
You gave me multiprocessing code and then warned that it might work not
very clean.
Can you tell me possible way and give all pros and cons, so I can make some
conclusion?
You said something about separate task system? I have no idea how to
implement it. All python solutions seemed to be around multithreading,
signal, multiprocessing.
By the way, will signal solution work with new mod_wsgi? What are pros and
cons?
import signal
def signal_handler(signum, frame):
raise Exception("Timed out!")
signal.signal(signal.SIGALRM, signal_handler)
signal.alarm(10) # Ten secondstry:
long_function_call()except Exception, msg:
print "Timed out!"
Thank you for your time.
On Saturday, February 21, 2015 at 10:44:22 PM UTC+2, Paul Royik wrote:
>
> *Which is part of the reason why I said it isn't a good idea to be forking
> worker subprocesses from your web application to do work. When you do that,
> the starting memory usage will be whatever the process was that was
> handling the web request to begin with. Your algorithm will then add on top
> of that.*
>
> *So if you web application has very fat memory usage, that carries
> through.*
>
> *This is why I was saying that you are better off with a separate task
> execution system. Its processes would have very small amount of memory
> usage to start with as it doesn't have all your web application memory as
> well. The separate task execution system can also control how many parallel
> tasks run and so limit overall memory usage.*
>
> *So, this seems not very good solution. I have only 512 MB. How to
> implement separate task system?*
>
> *Concerning your last question. I use FTP manager like Filezilla and that
> time I reuploaded all project (even that wsgi.py wasn't changed).*
>
>
> On Saturday, February 21, 2015 at 10:00:53 PM UTC+2, Graham Dumpleton
> wrote:
>
> On 22/02/2015, at 5:33 AM, Paul Royik <[email protected]> wrote:
>
> From time to time I've got following error:
> [Sat Feb 21 12:24:27.181049 2015] [wsgi:info] [pid 5482:tid
> 139829356984064] [remote 127.0.0.1:12680] mod_wsgi (pid=5482): Force
> restart of process 'localhost:20241'.
>
> Is it normal?
>
>
> That means that the WSGI script file for your application was modified.
> This is detected and the mod_wsgi daemon process restarted automatically.
>
> This is another reason why you shouldn't be doing stuff under web
> application processes.
>
> When this occurs, if you had an algorithm worker process running at the
> time, the killing off of the daemon process running your web application
> may not result in the worker being killed and so it could keep running with
> nothing then killing it.
>
> Will have to check that though, as depends on whether when daemon process
> is killed that children also killed automatically.
>
> How do you upload code changes to your system? Are you just using ssh to
> copy your whole code base every time and so the WSGI script file
> modification time will be updated even if not changed?
>
> Graham
>
> On Saturday, February 21, 2015 at 3:14:16 PM UTC+2, Paul Royik wrote:
>
> Thank you very much.
> It works now.
> But I noticed one thing. When calculator works, memory jumps from 180 to
> 250 (and the back to 180 as expected). When I used thread method, I didn't
> see such big jump. Can you tell me why in this case the jump is so big? I
> guess, that if 10 people will do it simulaneously, then I will run out of
> memory.
>
> Anyway, this is the cleanest and best solution.
>
>
> On Saturday, February 21, 2015 at 2:26:40 PM UTC+2, Graham Dumpleton wrote:
>
> The short answer is that code will not work with the version of mod_wsgi
> you are using. It will not kill the forked process properly and would
> actually kill the WSGI application daemon process instead, leaving the
> forked process still running and thus why it consumed memory. That it was
> killing the WSGI application daemon process would have been evident in the
> error logs which is why I asked about logs.
>
> Some extra tricks were required with a special directive needing to be
> given to mod_wsgi for the version you are using to allow signal handlers to
> be overridden and the forked process would need to reset the signal
> handlers before running the job.
>
> Since it was a pain to have to use the fiddles I worked out a way in
> mod_wsgi to avoid needing to do all that. You will need to upgrade to the
> latest mod_wsgi version though, which is 4.4.9.
>
> That isn't all that is required though as that code doesn't properly wait
> on the process when terminating it. This will result in a zombie process
> hanging around, which isn't a good idea. In worst case, since terminate()
> only tries SIGTERM, it may not go away.
>
> Now you can't simply go run 'pip install -U mod_wsgi' because of the issue
> we had with that breaking the mod_wsgi-httpd install. That issue has also
> now been fixed though. What you therefore want to do is reinstall both
> mod_wsgi-httpd and mod_wsgi.
>
> So if I got it right, you should be able to do.
>
> pip2.7 -v -v -v --log $HOME/pip.log install -U --user --build
> $HOME/tmp/pip mod_wsgi-httpd
> pip2.7 install -U --user mod_wsgi
>
> I am hoping that doesn't effect your running site. To be safe you may want
> to shut it down when doing the update.
>
> Once upgraded, use the following code. Remove print statements when happy.
> I would get rid of the force_kill flag as you probably always want to be
> force killing it and escalating that to SIGKILL to definitely get rid of it.
>
> import multiprocessing
> import signal
> import time
> import os
>
> class TimeoutException(Exception):
> pass
>
> class RunableProcessing(multiprocessing.Process):
> def __init__(self, func, *args, **kwargs):
> self.queue = multiprocessing.Queue(maxsize=1)
> args = (func,) + args
> multiprocessing.Process.__init__(self, target=self.run_func,
> args=args, kwargs=kwargs)
>
> def run_func(self, func, *args, **kwargs):
> try:
> result = func(*args, **kwargs)
> self.queue.put((True, result))
> except Exception as e:
> self.queue.put((False, e))
>
> def done(self):
> return self.queue.full()
>
> def result(self):
> return self.queue.get()
>
> def timeout(seconds, force_kill=True):
> def wrapper(function):
> def inner(*args, **kwargs):
> now = time.time()
> proc = RunableProcessing(function, *args, **kwargs)
> proc.start()
> proc.join(seconds)
> if proc.is_alive():
> print 'still alive'
> if force_kill:
> print 'kill it'
> proc.terminate()
> proc.join(3)
> print 'is it dead', proc.is_alive()
> if proc.is_alive():
> try:
>
>
> ...
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
