On Mon, 2015-11-23 at 07:48 -0600, Jason Kenny wrote: > Ideally threads in python work great, given the code in the threaded > code is a c-extension and or some subprocess or AIO like call. Pure > thread python code does not scale because of the GIL.
Well given that threads are no longer the things they were in the 1970s, that they now map to kernel threads, and we have fibres that are now what threads used to be then… Python and the GIL suck at anything multi-threaded unless you do a very specific form of blocking IO. > Given my work At Intel on threaded/MPI systems. Multiprocessing is > always slower then than threading. There is more overhead and work to > be done to communicate a simple state change that from a CPU point of > view is magnitudes slower than what would happen in a thread. Certain > patterns scale well with multiprocessing library that allow a system > to go across different machine. These pattern work well in python > multiprocessing library as it allow for reduced communication and > scaling of the CPU resources on a given box. SCons however has a > global DAG state that has to be shared. The one way this could work > is if the tasking system could use it. The issues is how we share > state on what is building or finished, output text across different > processes, dealing with pure python builders, etc. > > For me is not that this could not be useful… the issue is that it is > not as simple as one would hope. The moral of this story is do not even contemplate using Python for anything where parallelism in a single process is even remotely a good idea. D, Chapel, even C++, anything but Python. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:[email protected] 41 Buckmaster Road m: +44 7770 465 077 xmpp: [email protected] London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
