The short answer is that code will not work with the version of mod_wsgi you
are using. It will not kill the forked process properly and would actually kill
the WSGI application daemon process instead, leaving the forked process still
running and thus why it consumed memory. That it was killing the WSGI
application daemon process would have been evident in the error logs which is
why I asked about logs.
Some extra tricks were required with a special directive needing to be given to
mod_wsgi for the version you are using to allow signal handlers to be
overridden and the forked process would need to reset the signal handlers
before running the job.
Since it was a pain to have to use the fiddles I worked out a way in mod_wsgi
to avoid needing to do all that. You will need to upgrade to the latest
mod_wsgi version though, which is 4.4.9.
That isn't all that is required though as that code doesn't properly wait on
the process when terminating it. This will result in a zombie process hanging
around, which isn't a good idea. In worst case, since terminate() only tries
SIGTERM, it may not go away.
Now you can't simply go run 'pip install -U mod_wsgi' because of the issue we
had with that breaking the mod_wsgi-httpd install. That issue has also now been
fixed though. What you therefore want to do is reinstall both mod_wsgi-httpd
and mod_wsgi.
So if I got it right, you should be able to do.
pip2.7 -v -v -v --log $HOME/pip.log install -U --user --build $HOME/tmp/pip
mod_wsgi-httpd
pip2.7 install -U --user mod_wsgi
I am hoping that doesn't effect your running site. To be safe you may want to
shut it down when doing the update.
Once upgraded, use the following code. Remove print statements when happy. I
would get rid of the force_kill flag as you probably always want to be force
killing it and escalating that to SIGKILL to definitely get rid of it.
import multiprocessing
import signal
import time
import os
class TimeoutException(Exception):
pass
class RunableProcessing(multiprocessing.Process):
def __init__(self, func, *args, **kwargs):
self.queue = multiprocessing.Queue(maxsize=1)
args = (func,) + args
multiprocessing.Process.__init__(self, target=self.run_func, args=args,
kwargs=kwargs)
def run_func(self, func, *args, **kwargs):
try:
result = func(*args, **kwargs)
self.queue.put((True, result))
except Exception as e:
self.queue.put((False, e))
def done(self):
return self.queue.full()
def result(self):
return self.queue.get()
def timeout(seconds, force_kill=True):
def wrapper(function):
def inner(*args, **kwargs):
now = time.time()
proc = RunableProcessing(function, *args, **kwargs)
proc.start()
proc.join(seconds)
if proc.is_alive():
print 'still alive'
if force_kill:
print 'kill it'
proc.terminate()
proc.join(3)
print 'is it dead', proc.is_alive()
if proc.is_alive():
try:
print 'force kill'
os.kill(proc.pid, signal.SIGKILL)
except Exception:
pass
proc.join(1)
print 'alive', proc.is_alive()
runtime = int(time.time() - now)
raise TimeoutException('timed out after {0}
seconds'.format(runtime))
assert proc.done()
success, result = proc.result()
if success:
return result
else:
raise result
return inner
return wrapper
def term(*args):
print 'term'
time.sleep(4.0)
print 'done'
@timeout(5.0)
def func():
signal.signal(signal.SIGTERM, term)
print 'sleep'
time.sleep(10)
def application(environ, start_response):
status = '200 OK'
output = b'Hello World!'
func()
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
On 21/02/2015, at 10:23 PM, Paul Royik <[email protected]> wrote:
> I used this code:
> class TimeoutException(Exception):
> pass
>
>
> class RunableProcessing(multiprocessing.Process):
> def __init__(self, func, *args, **kwargs):
> self.queue = multiprocessing.Queue(maxsize=1)
> args = (func,) + args
> multiprocessing.Process.__init__(self, target=self.run_func,
> args=args, kwargs=kwargs)
>
> def run_func(self, func, *args, **kwargs):
> try:
> result = func(*args, **kwargs)
> self.queue.put((True, result))
> except Exception as e:
> self.queue.put((False, e))
>
> def done(self):
> return self.queue.full()
>
> def result(self):
> return self.queue.get()
>
>
> def timeout(seconds, force_kill=True):
> def wrapper(function):
> def inner(*args, **kwargs):
> now = time.time()
> proc = RunableProcessing(function, *args, **kwargs)
> proc.start()
> proc.join(seconds)
> if proc.is_alive():
> if force_kill:
> proc.terminate()
> runtime = int(time.time() - now)
> raise TimeoutException('timed out after {0}
> seconds'.format(runtime))
> assert proc.done()
> success, result = proc.result()
> if success:
> return result
> else:
> raise result
> return inner
> return wrapper
>
> There was nothing suspicious in error logs, just request-timeout from tme to
> time.
>
> On Saturday, February 21, 2015 at 11:44:29 AM UTC+2, Graham Dumpleton wrote:
> On 21/02/2015, at 8:04 PM, Paul Royik <[email protected]> wrote:
>
>> First of all I found a post, where you said, that it is not a good idea to
>> use multiprocessing with mod_wsgi.
>> Secondly, library that I use is not very compatible with multiprocessing
>> (pickling issues).
>
> If using a queue, the only data that should need pickling is the response
> data. If the classes from your library can not be pickled all you need to do
> is convert them to more basic data structures so they can be returned. Since
> you are likely going to have to convert the result into something that can be
> rendered into your page response if embedding results in a web page, you
> could even render the page in the separate process and just return the text
> of the page to then be returned from the web request.
>
>> Thirdly, very fust memory overflow, in 20 minutes memory usage grows to 7 GB.
>
> So it sounds like you didn't really understand why it failed.
>
> Was there anything strange in the way of messages in the Apache error log?
>
> What was the actual code you were using with the multiprocess module?
>
> That is which of the techniques from:
>
> https://docs.python.org/2/library/multiprocessing.html
>
> were you using, but then also how did you modify those to accommodate trying
> to kill the process when your timeout was reached and also ensure that the
> child process was properly reaped and didn't just hang around as a zombie
> process?
>
> Graham
>
>> On Saturday, February 21, 2015 at 9:23:18 AM UTC+2, Graham Dumpleton wrote:
>>
>> On 21/02/2015, at 9:52 AM, Paul Royik <[email protected]> wrote:
>>
>> > Hello.
>> > Is there anyway to stop thread using subprocess module.
>>
>> The subprocess module has got nothing to do with in process threading.
>>
>> > As far as I know multiprocessing is not applicable for mod_wsgi.
>>
>> Explain what the problem was when you attempted to use the multiprocessing
>> module.
>>
>> I occasionally have seen people say that multiprocessing doesn't work under
>> mod_wsgi. I have never actually ever had some one lodge a bug report
>> indicating what the problem is, nor even use the mailing list to explain
>> what they were doing with a very simple code example, and explain what
>> didn't work.
>>
>> > I returned to thread solution, because there is no other way to stop
>> > function.
>>
>> If you couldn't get the timeout based method within the algorithm self
>> checking, it will not work if in a thread.
>>
>> Ultimately, you should not even be attempting to do what you are in the web
>> server processes, or even as sub processes of the web application. What
>> would be regarded as best practice for this sort of thing is a separate task
>> queueing service which has the capabilities for using separate processes for
>> task execution and a supervisor system that can kill processes after running
>> for a set time and not completing. Such a system can also control the rate
>> at which parallel tasks are allowed to run to control memory usage.
>>
>> Now before you jump up and down and say 'how, how, how', explain what you
>> were trying to do with multiprocessing and how it failed.
>>
>> Graham
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
