Hi All - With the SCons 3.0 release and GitHub migration completed (huge congratulations to everyone involved), I'd like to revisit the discussion from the summer about SCons performance.
This email is a follow up to http://two.pairlist.net/pipermail/scons-dev/2017-July/004371.html While there was a lot of good discussion in that thread, it drifted into several interesting side conversations. So I would like to narrow the scope of discussion back to performance. I would also like to propose that the next SCons release, (possibly named 3.1), focus primarily on SCons performance, to the exclusion of feature evolution or refactoring. As was noted in the earlier thread, SCons has many unique, flexible, and powerful features, and those features come with a performance cost. I think it unrealistic to expect that SCons will ever match the performance of, say, Ninja, and I think that is just fine. But I also believe that SCons could be made much faster than it currently is without compromising its power or flexibility, and I further think that failing to improve its performance jeopardizes its future. There are many competitors to SCons emerging, and the current SCons performance for incremental rebuilds is so poor as to make a busy developer reach for almost anything else to make the bleeding stop. I would like to propose the following as highlight areas for improvement in a performance focused SCons release: - Profile and fix low-hanging performance issues in no-op builds: Bill has found a few using the MongoDB build as a testbed, but we know there are more. Aggressively profile and address these issues as they are found. - Tune for performance over resource utilization: There are places where information is discarded in an effort to prune resource utilization, but developer and CI machines are typically generously provisioned. Identify locations where SCons has been optimized for resource utilization over performance, and reverse. - Implement variable Substitution caching: In complex builds we have discovered that a large fraction of time is spent repeatedly evaluating the same Substs. Caching these results when safe should offer significant speedups. This has a larger visible impact on incremental builds where the recreation of command lines to compare against existing built objects can eat up to 50% of the run time. - Re-write the SConsign mechanism to use a new opt-in format that avoids the cost of pickle, and can do incremental updates. This would speed up both the load of the SConsign data out of the gate, and also dramatically improve the end-of-build delay encountered in large builds as the entire SConsign file is re-written, even though only a few files changed. - Avoid re-computing MD5 signatures when extracting items from the SCons Cache. The end-to-end caching behavior of SCons is one of its highlights, but enabling it causes signature re-computation when items are pulled out of the cache, undermining its performance. The cache should be augmented to cache not just the objects themselves, but the signatures that they possess. When objects are extracted from the cache, the required signature comes with and does not need to be re-computed. - Fix the issue where the MD5-Timestamp decider cannot be safely combined with the SCons Cache without spurious rebuilds <http://scons.tigris.org/issues/show_bug.cgi?id=2980>. This again defeats the otherwise fantastic behavior of the cache by requiring users to MD5 every file when building with the cache. By fixing this issue, and implementing the above CacheDir improvements, the SCons cache will both be intrinsically fast, and not force a degradation in the normal build path. - Implement a parallel TaskMaster, such that the DAG walk can utilize multiple cores. While this is clearly the most technically challenging proposal, the benefits would be huge. I believe that a SCons 3.1 release that addressed the above issues would put SCons in a position to again become competitive with the landscape of build systems. I’m looking forward to hearing your thoughts, but I would appreciate it if we could try to keep focused on performance and its prioritization in the SCons release roadmap, rather than other discussions of potential future SCons features or refactorings. Thanks, Andrew
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
