Ideally threads in python work great, given the code in the threaded code is a c-extension and or some subprocess or AIO like call. Pure thread python code does not scale because of the GIL.
Given my work At Intel on threaded/MPI systems. Multiprocessing is always slower then than threading. There is more overhead and work to be done to communicate a simple state change that from a CPU point of view is magnitudes slower than what would happen in a thread. Certain patterns scale well with multiprocessing library that allow a system to go across different machine. These pattern work well in python multiprocessing library as it allow for reduced communication and scaling of the CPU resources on a given box. SCons however has a global DAG state that has to be shared. The one way this could work is if the tasking system could use it. The issues is how we share state on what is building or finished, output text across different processes, dealing with pure python builders, etc. For me is not that this could not be useful… the issue is that it is not as simple as one would hope. Jason From: Russel Winder Sent: Sunday, November 22, 2015 3:29 PM To: SCons developer list Cc: Sarah Mount Subject: Re: [Scons-dev] Multiprocessing instead of threads? On Fri, 2015-11-20 at 19:51 -0800, Bill Deegan wrote: > Greetings, > > Would there be value in using python's multiprocessing package > instead of > threads? As any consultant will tell you: it depends. Threads in Python suck due to the GIL. Conversely, threads work fine for concurrency due to the GIL, assuming you have no interest whatsoever in parallelism. Parallelism in Python currently requires use of multiprocessing. It works but there is an overhead. And the synchronization infrastructure of multiprocessing is really rather naïve and under-performant. Sarah Mount has needed a far better low level process communication synchronization and communication infrastructure for Python-CSP. I said I would investigate. To date I have failed. Perhaps in 2016-02… > Anyone have experience using this package? Yes. ! -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:[email protected] 41 Buckmaster Road m: +44 7770 465 077 xmpp: [email protected] London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
