On 21/02/2015, at 8:04 PM, Paul Royik <[email protected]> wrote:
> First of all I found a post, where you said, that it is not a good idea to > use multiprocessing with mod_wsgi. > Secondly, library that I use is not very compatible with multiprocessing > (pickling issues). If using a queue, the only data that should need pickling is the response data. If the classes from your library can not be pickled all you need to do is convert them to more basic data structures so they can be returned. Since you are likely going to have to convert the result into something that can be rendered into your page response if embedding results in a web page, you could even render the page in the separate process and just return the text of the page to then be returned from the web request. > Thirdly, very fust memory overflow, in 20 minutes memory usage grows to 7 GB. So it sounds like you didn't really understand why it failed. Was there anything strange in the way of messages in the Apache error log? What was the actual code you were using with the multiprocess module? That is which of the techniques from: https://docs.python.org/2/library/multiprocessing.html were you using, but then also how did you modify those to accommodate trying to kill the process when your timeout was reached and also ensure that the child process was properly reaped and didn't just hang around as a zombie process? Graham > On Saturday, February 21, 2015 at 9:23:18 AM UTC+2, Graham Dumpleton wrote: > > On 21/02/2015, at 9:52 AM, Paul Royik <[email protected]> wrote: > > > Hello. > > Is there anyway to stop thread using subprocess module. > > The subprocess module has got nothing to do with in process threading. > > > As far as I know multiprocessing is not applicable for mod_wsgi. > > Explain what the problem was when you attempted to use the multiprocessing > module. > > I occasionally have seen people say that multiprocessing doesn't work under > mod_wsgi. I have never actually ever had some one lodge a bug report > indicating what the problem is, nor even use the mailing list to explain what > they were doing with a very simple code example, and explain what didn't > work. > > > I returned to thread solution, because there is no other way to stop > > function. > > If you couldn't get the timeout based method within the algorithm self > checking, it will not work if in a thread. > > Ultimately, you should not even be attempting to do what you are in the web > server processes, or even as sub processes of the web application. What would > be regarded as best practice for this sort of thing is a separate task > queueing service which has the capabilities for using separate processes for > task execution and a supervisor system that can kill processes after running > for a set time and not completing. Such a system can also control the rate at > which parallel tasks are allowed to run to control memory usage. > > Now before you jump up and down and say 'how, how, how', explain what you > were trying to do with multiprocessing and how it failed. > > Graham > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
