I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec.
-- Gary Oberbrunner (sent from my Android) On Apr 1, 2014 11:52 AM, "Kenny, Jason L" <[email protected]> wrote: > > > Hi guys, > > > > I just got a patch to Parts internal at Intel to fix some issues with > subprocess. I find myself sort of surprised by this, and honestly felt that > this seems to be an issue that should be looked at in Scons as well. > > > > The problem is this. We have been building a given huge product here at > Intel on RHEL 4.8. IT is old and finally time has come to start moving to a > newer system. What was found when we moved to a newer RHEL 5.10 ( I know > bleeding edge J ) we found that the build was twice as slow. Looking into > it more it was found that it was the use of fork() in subprocess. It was > found that posix_spawn() fixed this. Attached is a monkey patch that is up > for internal review to modify subprocess module a little to fix this for > Unix based systems. > > > > Now I am not an expert on Linux internal details. However I thought fork() > was the way process got spawned on Linux. Ie fork() was the win32 > CreateProcess(). I am clearly wrong, but this seems to be a problem I would > not expect to have seen on Linux. The root of the problem seems to be that > the system takes a long time to setup the fork when you have a large build > ( and we have a very large build ) because of the memory usage (2-6GB) > needed by SCons alone. Using a different method to create a process that > does not try to "clone" memory seems to be the fix that gets the build > times to be about ~2x+ faster. > > > > Does anyone else have input on this? IF this is a good patch, it seem that > we will want to apply it to SCons, for a speed boost. > > > > Thanks > > Jason > > > > > > Also I attached a test file that to show fork() is the problem. You can > see the difference importing the stubprocess.py file and running the test > again. > > _______________________________________________ > Scons-dev mailing list > [email protected] > http://two.pairlist.net/mailman/listinfo/scons-dev > >
_______________________________________________ Scons-dev mailing list [email protected] http://two.pairlist.net/mailman/listinfo/scons-dev
