Do we have a branch to test out? I pretty sure that this will break Parts ( as I monkey patch a lot of items on the nodes objects) However I would like to see how bad it is, I sure a small tweak will allow me to work around the issues ( ie add __dict__ to the __slot__ definition) I making a drop of Parts 0.10.2 this week, and I was able to reduce a large amount of memory on my end, I would very much like to see the memory difference with these fixes.
Jason From: [email protected] [mailto:[email protected]] On Behalf Of William Deegan Sent: Monday, December 03, 2012 12:43 PM To: SCons developer list Subject: Re: [Scons-dev] Optimizing SCons... All, On Dec 2, 2012, at 8:58 AM, Gary Oberbrunner <[email protected]<mailto:[email protected]>> wrote: This looks very interesting. Speed and memory use are two "hot button" issues for many SCons users. What does it do that would break existing projects? Is it the not storing of full paths? (When were slots introduced? 2.2? In that case we're fine on that front.) One of the SCons buildbot builders runs a set of memory and timing benchmarks -- maybe we could get your version run through that. I suspect making a branch is the best way to start that -- what do you think, Bill? Sounds good. Though for some reason since we killed off the bad merge head buildbot seems to be jammed up. I'll try and take a look at that over the next few days. -Bill On Sun, Dec 2, 2012 at 9:01 AM, Dirk Bächle <[email protected]<mailto:[email protected]>> wrote: Hi, over the last days I created a patch that aims at reducing SCons' memory footprint, especially for large (in terms of number of files) C/C++ projects. It can be applied to the current latest revision (b496d47c4efb) and is attached as archive to this email. The amount of changes is split into the basic steps: 1) Make Node classes use slots. 2) Make SigInfo classes use slots. 3) Make Executor and Batch use slots and stop caching full paths in File nodes. I had to rewrite the memoizer count framework to use decorators instead of the original metaclass approach. Additionally, I had to correct a lot of tests and Tools to ensure that they still pass without any fails (at least for those tests that I can run locally under Linux, some further adaptions might be necessary). Here are some numbers for the testsuite that I compiled (it consists of several real-life SCons projects): Project | before | after | ================================= ascend 96MB 82MB bombono 120MB 104MB lumiera 114MB 101MB mapnik 148MB 129MB sconsbld 540MB 377MB ================================= They list the maximum of allocated memory for a clean build, before and after the optimizations. For the first four projects the results don't show that much of an improvement. But this is because they are still relatively small, compared to the basic overhead that is needed while parsing the SConstruct/SConscript files for example. The last (sconsbld) is a benchmark, created by a Perl script, with a number of 12000 source files, and 12000 includes...resulting in 12000 object files and 600 executables. Increasing the number of files seems to drive up the percentage of memory you can save. Doubling them up for "sconsbld" (24000 source and include files) gives 1579MB vs 1052MB. I also used cProfile and simple timing of the single runs with the "time" command, to ensure that no speed penalties are involved with the changes. Good news is that all runs, clean builds as well as updates, tend to get a little faster...probably due to the usage of slots. So, that's what I have right now. My questions are: - Is this of any interest (or is it still "too early" for optimizations ;) )? - If yes, what could be the next steps? I didn't simply upload this as a pull request, because the changes would definitely break existing projects and custom Tools. So, either some form of deprecation cycle or a separate branch/repository would be needed (default vs optimized). Comments anyone? Best regards, Dirk _______________________________________________ Scons-dev mailing list [email protected]<mailto:[email protected]> http://two.pairlist.net/mailman/listinfo/scons-dev -- Gary _______________________________________________ Scons-dev mailing list [email protected]<mailto:[email protected]> http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________ Scons-dev mailing list [email protected] http://two.pairlist.net/mailman/listinfo/scons-dev
