Thanks Justin! Yes my box has hyperthreading enabled. I discovered that leaving 1 physical core free ( 2 logical core free ) offers the best performance with spawning multiple instances of oiio process.
I am not blocked, but I was wondering if other people had different experience with regards to speeding up OIIO as python based image processing tool. Cheers Dave. Dave Lajoie R&D Director | Directeur R&D [image: Digital-District] <http://www.digital-district.fr> 5605 Avenue de Gaspé, Suite 408 | Montréal, QC H2T 2A4 Tel (514) 360-3253 ext. 130 On Mon, Sep 1, 2014 at 12:59 AM, Justin Israel <[email protected]> wrote: > I can't comment at all on how the OIIO python bindings specifically behave > in a threaded usage, but from a general standpoint AFAIK the library would > have to be dropping the GIL when it can, in order for calls to be able to > yield to other threads. That would mean that under the condition that it > wasn't dropping the GIL, your multi-threaded calls into oiio would end up > blocking on each other and you would end up with a semi-serial result. > > The reason Python threading works better with IO-bound calls is that it > can release the GIL and let other threads run for a time slice. I added > similar GIL-dropping to one of my Cython bindings to a C++ library so that > multiple threaded calls into the library could execute concurrently. > > Do you have 24 physical cores in your box? Or are those hyper-threaded > logical cores (12 physical cores)? I would think, if using the python > multiprocessing approach, you would want to do something like a 12 process > pool, and let each task have 2 thread? > > Again, this was totally a generic python response :-) > > > > On Mon, Sep 1, 2014 at 12:35 PM, Dave Lajoie < > [email protected]> wrote: > >> Hello Guys! >> >> I have been searching the archive and I could not find anything on this. >> Basically I am using open image IO as slate generation tool (among other >> things of course). It works great! I am looking at speeding up the process, >> so I have attempted to call my python function using various threading >> modules, and I am not getting a lot of speed up. I am using nthread=0 to >> ensure the call uses all available threads/cores. >> >> I do know that Python GIL is well known to be great at certain type of >> multi-threading calls, especially anything IO bound offers best >> performance. However I was wondering if there are any know issues with >> python multithreading, and/or if there was any list of do's and don'ts. >> >> Currently the only way I could get all the cores to be utilized to 100% >> was to spawn a separate python process, which will process a single frame, >> so on my box I would have 24 concurrent python subprocesses invoking OIIO. >> The machine grinds almost to halt ( which is good in this case), but I was >> wondering what are the other things I don't know regarding this subject. :) >> >> Any help would be appreciated. >> Best >> Dave Lajoie >> R&D Director Digital District Montreal >> >> >> >> _______________________________________________ >> Oiio-dev mailing list >> [email protected] >> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org >> >> > > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > >
_______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
