On Mon, Nov 23, 2015 at 7:57 PM, Neal Becker <[email protected]> wrote:
> Jason Kenny wrote: > > > I have thought about it myself > > > > It might have value if we can separate the task logic in some way to > allow > > a way to reduce data syncing on the node. We have to remember that we > have > > a big global data structure that we use to control the build state. As > > item get done we do more work in a safe way. In a thread model this works > > great as we are in the same process space. In multithreading we have to > > sync this data across processes. This is a magnitudes slower than the > > threaded case. So the trick is can we use multiprocessing for us to speed > > up python based tasked ( vs the ones that are commandline only). I don’t > > have a clear answer but have thought of two ideas.. > > What about shared memory? > Threads automatically benefit here as they are shared memory by default, but with extra work, we could also do shared memory with subprocesses. > > > > > 1) Can we provide a better interface to making builders or actions of > pure > > python code so that we could make a temp script that we call as a > separate > > python process. This would allow us to reduce contention on the GIL to > > logic related to build state, vs builder logic 2) Can we do something to > > change the task logic to reduce communication to a master “process” that > > keeps state and delegates out actions to a sperate process > > > > If we would like to make an improvement I am very sure that if we made > the > > build off of builders vs nodes this would greatly speed up the build > > logic. This was the basic premise of the Greg Noels TNG work. I can say > > more this in a different thread .. don’t want to side track this thread. > > > > I think we should also note make more processes will require more memory > > overall, vs threads. Howe much depends on details that are not completely > > clear to me at the moment. > > > > Jason > > > > > > Sent from Mail for Windows 10 > > > > > > > > From: Bill Deegan > > Sent: Friday, November 20, 2015 9:52 PM > > To: SCons developer list > > Subject: [Scons-dev] Multiprocessing instead of threads? > > > > > > Greetings, > > Would there be value in using python's multiprocessing package instead of > > threads? Anyone have experience using this package? > > -Bill > > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
