I used this code:
class TimeoutException(Exception):
pass
class RunableProcessing(multiprocessing.Process):
def __init__(self, func, *args, **kwargs):
self.queue = multiprocessing.Queue(maxsize=1)
args = (func,) + args
multiprocessing.Process.__init__(self, target=self.run_func, args=args,
kwargs=kwargs)
def run_func(self, func, *args, **kwargs):
try:
result = func(*args, **kwargs)
self.queue.put((True, result))
except Exception as e:
self.queue.put((False, e))
def done(self):
return self.queue.full()
def result(self):
return self.queue.get()
def timeout(seconds, force_kill=True):
def wrapper(function):
def inner(*args, **kwargs):
now = time.time()
proc = RunableProcessing(function, *args, **kwargs)
proc.start()
proc.join(seconds)
if proc.is_alive():
if force_kill:
proc.terminate()
runtime = int(time.time() - now)
raise TimeoutException('timed out after {0}
seconds'.format(runtime))
assert proc.done()
success, result = proc.result()
if success:
return result
else:
raise result
return inner
return wrapper
There was nothing suspicious in error logs, just request-timeout from tme to
time.
On Saturday, February 21, 2015 at 11:44:29 AM UTC+2, Graham Dumpleton wrote:
>
> On 21/02/2015, at 8:04 PM, Paul Royik <[email protected] <javascript:>>
> wrote:
>
> First of all I found a post, where you said, that it is not a good idea to
> use multiprocessing with mod_wsgi.
> Secondly, library that I use is not very compatible with multiprocessing
> (pickling issues).
>
>
> If using a queue, the only data that should need pickling is the response
> data. If the classes from your library can not be pickled all you need to
> do is convert them to more basic data structures so they can be returned.
> Since you are likely going to have to convert the result into something
> that can be rendered into your page response if embedding results in a web
> page, you could even render the page in the separate process and just
> return the text of the page to then be returned from the web request.
>
> Thirdly, very fust memory overflow, in 20 minutes memory usage grows to 7
> GB.
>
>
> So it sounds like you didn't really understand why it failed.
>
> Was there anything strange in the way of messages in the Apache error log?
>
> What was the actual code you were using with the multiprocess module?
>
> That is which of the techniques from:
>
> https://docs.python.org/2/library/multiprocessing.html
>
> were you using, but then also how did you modify those to accommodate
> trying to kill the process when your timeout was reached and also ensure
> that the child process was properly reaped and didn't just hang around as a
> zombie process?
>
> Graham
>
> On Saturday, February 21, 2015 at 9:23:18 AM UTC+2, Graham Dumpleton wrote:
>>
>>
>> On 21/02/2015, at 9:52 AM, Paul Royik <[email protected]> wrote:
>>
>> > Hello.
>> > Is there anyway to stop thread using subprocess module.
>>
>> The subprocess module has got nothing to do with in process threading.
>>
>> > As far as I know multiprocessing is not applicable for mod_wsgi.
>>
>> Explain what the problem was when you attempted to use the
>> multiprocessing module.
>>
>> I occasionally have seen people say that multiprocessing doesn't work
>> under mod_wsgi. I have never actually ever had some one lodge a bug report
>> indicating what the problem is, nor even use the mailing list to explain
>> what they were doing with a very simple code example, and explain what
>> didn't work.
>>
>> > I returned to thread solution, because there is no other way to stop
>> function.
>>
>> If you couldn't get the timeout based method within the algorithm self
>> checking, it will not work if in a thread.
>>
>> Ultimately, you should not even be attempting to do what you are in the
>> web server processes, or even as sub processes of the web application. What
>> would be regarded as best practice for this sort of thing is a separate
>> task queueing service which has the capabilities for using separate
>> processes for task execution and a supervisor system that can kill
>> processes after running for a set time and not completing. Such a system
>> can also control the rate at which parallel tasks are allowed to run to
>> control memory usage.
>>
>> Now before you jump up and down and say 'how, how, how', explain what you
>> were trying to do with multiprocessing and how it failed.
>>
>> Graham
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected] <javascript:>
> .
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
