At least in pypy a json sconsign would be quite fast compared to pickle. On Fri, Jul 21, 2017, 11:39 Andrew C. Morrow <[email protected]> wrote:
> > Hi scons-dev - > > The following is a revised draft of an email that I had originally > intended to send as a follow up to > https://pairlist4.pair.net/pipermail/scons-users/2017-June/006018.html. > Instead, Bill Deegan and I took some time to expand on my first draft and > add some ideas about how to address some of th e issues. We hope to migrate > this to the wiki, but wanted to share it here first for feedback. > > ---- > > Performance is one of the major challenges facing SCons. When compared > with other current options, particularly Ninja, in many cases performance > can lag significantly. That said other options by and large lack the > extensibility and many features of SCons. > > Bill Deegan (SCons project co-manager) and I have been working together to > understand some of the issues that lead to poor SCons performance in a real > world (and fairly modestly sized) C++ codebase. Here is a summary of some > of our findings: > > > - > > Python code usage: There are many places in the codebase where while > the code is correct, performance based on cpython’s implementation can be > improved by minor changes. > - > > Examples > - > > Using for loops and hashes to uniquify a list. Simple change in > Node class yielded approximately 15% speedup for null build > - > > Using if x.find(‘some character’) >=0 instead of is ‘some > character’ in x (timeit benchmark shows a 10x speed difference) > - > > Method to address > - > > Profile the code looking for hotspots with cprofile and > line_profiler. Then look for best implementations of code. (Use > timeit if > useful to compare implementations. There are examples of such in the > bench > dir (see: > > https://bitbucket.org/scons/scons/src/68a8afebafbefcf88217e9e778c1845db4f81823/bench/?at=default > ) > - > > Serial DAG traversal: SCons walks the DAG to find out of date targets > in a serial fashion. Once it finds them, it farms the work out to other > threads, but the DAG walk remains serial. Given the proliferation of > multicore machines since SCons’ initial implementation, a parallel walk of > the DAG would yield significant speedup. Likely this would require > implementation using the multiprocessing python library (instead of > threads), since the GIL would block real parallelism otherwise. Packages > like Boost where there are many header files can cause large increases in > the size of the DAG, exacerbating this issue. There are two serious > consequences of the slow DAG walk: > - > > Incremental rebuilds in large projects. Typical developer workflow > is to edit a file, rebuild, test. In our modestly sized codebase, we see > the incremental time to do an ‘all’ rebuild for a one file change can > reach > well over a minute. This time is completely dominated by the serial > dependency walk. > - > > Inability to saturate distributed build clusters. In a > distcc/icecream build, the serial DAG walk is slow enough that not > enough > jobs can be farmed out in parallel to saturate even a modest (400 cpu) > build cluster. In our example, using ninja to drive a distributed full > build results in an approximately 15x speedup, but SCons can only > achieve a > 2x speedup. > - > > Method to address: > - > > Investigate changing tree walk to generator > - > > Investigate implementing tree walk using multiprocessing library > - > > The dependency graph is the python object graph: The target dependency > DAG is modeled via python Node Object to Node Object linkages (e.g. a list > of child nodes held in a node). As a result, the only way to determine > up-to-date-ness is by deeply nested method calls that repeatedly traverse > the Python object graph. An attempt is made to mitigate this by memoizing > state at the leaves (e.g. to cache the result of stat calls), but this > still results in a large number of python function invocations for even the > simplest state checks, where a result is already known. Similarly, the lack > of global visibility precludes using externally provided change information > to bypass scans. > - > > See above re generator > - > > Investigate modeling state separately from the python Node graph > via some sort of centralized scoreboarding mechanism, it seems likely > that > both the function call overhead could be eliminated and that local > knowledge could be propagated globally more effectively. > - > > CacheDir: There are some issues listed below. End-to-end caching > functionality of SCons, including generated files, object files, shared > libraries, whole executables, etc., is one of its great strengths, but its > performance has much room for improvement. > - > > Existing bug(s) when combining CacheDir with MD5-Timestamp devalues > CacheDir. > - > > Bug: http://scons.tigris.org/issues/show_bug.cgi?id=2980 > - > > Performance issues: > - > > CacheDir re-creates signature data when extracting nodes from > the Cache, even though it could have recorded the signature when > entering > the objects into the cache. > - > > Method to address > - > > Store signatures for items in cachedir and then use them > directly when copying items from Cache. > - > > Fix the CacheDir / MD5-Timestamp integration bug > - > > SConsign generation: The generation of the SConsign file is > monolithic, not incremental. This means that if only one object file > changed, the entire database needs to be re-written. It also appears that > the mechanism used to serialize it is itself slow. Moving to a faster > serialization model would be good, but even better would be to move to a > faster serialization model that also admitted incremental updates to single > items. > - > > Method to address: > - > > Replace sconsign with something faster than the current > implementation, which is based on Pickle. > - > > And/or Improve sconsign with something which can incrementally > only write that which has changed. > - > > Configure check performance: Even cached Configure checks seems slow, > and for a complexly configured build this can add significant startup cost. > Improvements here would be useful. > - > > Method to address: > - > > Code inspection, look for improvements > - > > Profile > - > > Variable Substitution: Currently variable substitution, which is > largely used to create the command lines run by SCons, uses an appreciable > percentage (approximately 18% for a null incremental build) of SCons’ CPU > runtime. By and large much of this evaluation is duplicate (and thus > avoidable work). For the moderate sized build discussed above there are > approximately 100k calls to evaluation substitutions. There are only 413 > unique strings to be evaluated. Consider that the CXXCOM variable is > expanded 2412 times for this build. The only variables which are guaranteed > unique are the SOURCES and TARGETS, all others could be evaluated once and > cached. > - > > Prior work on this item: > - > > > > https://bitbucket.org/scons/scons/wiki/SubstQuoteEscapeCache/Discussion > - > > Working doc on current and areas for improvement: > - > > > > https://bitbucket.org/scons/scons/wiki/SubstQuoteEscapeCache/SubstImprovement2017 > - > > Method to address: > - > > Consider pre-evaluating Environment() variables where > reasonable. This could use some sort of copy-on-write between cloned > Environments. This pre-evaluation would skip known target specific > variables (TARGET,SOURCES,CHANGED_SOURCES, and a few others), so > minimally > the per command line substitution should be faster. > > > Bill and I would appreciate any feedback or thoughts on the above items, > or suggestions for other areas to investigate. We are hoping that by > addressing some or all of these items, the runtime overhead of SCons could > be brought down significantly enough to re-render it competitive with other > build systems. We hope to begin work on the above items once SCons 3.0 has > shipped. > > Thanks, > Andrew > > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
