Jason Kenny wrote: > I have thought about it myself > > It might have value if we can separate the task logic in some way to allow > a way to reduce data syncing on the node. We have to remember that we have > a big global data structure that we use to control the build state. As > item get done we do more work in a safe way. In a thread model this works > great as we are in the same process space. In multithreading we have to > sync this data across processes. This is a magnitudes slower than the > threaded case. So the trick is can we use multiprocessing for us to speed > up python based tasked ( vs the ones that are commandline only). I don’t > have a clear answer but have thought of two ideas..
What about shared memory? > > 1) Can we provide a better interface to making builders or actions of pure > python code so that we could make a temp script that we call as a separate > python process. This would allow us to reduce contention on the GIL to > logic related to build state, vs builder logic 2) Can we do something to > change the task logic to reduce communication to a master “process” that > keeps state and delegates out actions to a sperate process > > If we would like to make an improvement I am very sure that if we made the > build off of builders vs nodes this would greatly speed up the build > logic. This was the basic premise of the Greg Noels TNG work. I can say > more this in a different thread .. don’t want to side track this thread. > > I think we should also note make more processes will require more memory > overall, vs threads. Howe much depends on details that are not completely > clear to me at the moment. > > Jason > > > Sent from Mail for Windows 10 > > > > From: Bill Deegan > Sent: Friday, November 20, 2015 9:52 PM > To: SCons developer list > Subject: [Scons-dev] Multiprocessing instead of threads? > > > Greetings, > Would there be value in using python's multiprocessing package instead of > threads? Anyone have experience using this package? > -Bill _______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
