Thanks Graham. I have put in place a plan to test with 1ProcessX3Threads,2ProcessX3Threads,4ProcessX3Threads and likewise for 6,10,20,40 threads. Again for me benchmark is the CPU usage with response time. Anything above 75% CPU, will try to mark it as a comfort zone....in addition planning to review how load average varies with the above.
Will share what i end up.. thanks On Monday, June 5, 2017 at 6:26:49 AM UTC+5:30, Graham Dumpleton wrote: > > > On 5 Jun 2017, at 2:04 AM, Vishnu Prasad <[email protected] > <javascript:>> wrote: > > Thanks Graham. Got my code ready @ > https://github.com/vishnuprasadh/flaskdynamicimgprocessor. Will Perf test > and update, was a very wise guidance from you. > > > Depending on how the image library being used is implemented, using a high > number of threads as you have in: > > WSGIDaemonProcess imgprocessor user=vph home=/var/www/imgprocessor > threads=40 > > may be a bad idea. > > The problem is that the global interpreter lock (GIL) in Python means that > only one piece of Python code can run at a time. > > If the image library being used isn't able to release the GIL while doing > the image resize, and use C level operations only which don't need to lock > the Python object, then you can't have multiple image resize operations > running in parallel. Being a CPU bound activity also doesn't help your > course. > > For CPU bound activities, especially where the GIL cannot be released and > the operation can take a while, you should favour use of multiple processes > over threads as then you can get proper concurrency by virtue of the CPU > intensive task running on separate processes. > > Use of a high number of threads should only be done when you have a I/O > bound application. Even then, running 40 threads rings alarm bells. Usually > you still want to run a lot less than that if can. > > Also, as far as testing performance, ensure you aren't testing by simply > overloading your web server. That is completely the wrong way of going > about it. > > I would recommend you only supply as much traffic as would cause a single > web application process to be within the 40-60% CPU range if using Python > code and so limited by the GIL. If you are reaching more than that, you > should be creating more processes. > > Note that even if you put threads at 40 and do overload your application, > it doesn't necessarily mean that all threads are being used. Most likely in > a CPU intensive application, most threads sit there unused and so wasting a > bit of memory. That is why need to monitor CPU usage of processes, but also > the thread utilisation factor. > > I did a talk about this at regional PyCon some time back. See: > > * https://www.youtube.com/watch?v=SGleKfigMsk > > Lastly, neither of these is needed in the configuration: > > WSGIScriptReloading On > Options +ExecCGI > > Script reloading is on by default. ExecCGI is only needed if using > AddHandler, not when using WSGIScriptAlias. > > And: > > Order allow,deny > Allow from all > Require all granted > > is using both Apache 2.2 and 2.4 methods of granting access. I wouldn't > show both. If need to, use: > > <IfVersion < 2.4> > Order allow,deny > Allow from all > </IfVersion> > > <IfVersion >= 2.4> > Require all granted > </IfVersion> > > Graham > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
