On Dec 3, 2012, at 9:57 PM, anatoly techtonik <[email protected]> wrote:
> On Sun, Dec 2, 2012 at 5:01 PM, Dirk Bächle <[email protected]> wrote: > Hi, > > over the last days I created a patch that aims at reducing SCons' > memory footprint, especially for large (in terms of number of > files) C/C++ projects. > It can be applied to the current latest revision (b496d47c4efb) > and is attached as archive to this email. > > It will be easier to review this on rietveld. Are you unable to review it on bitbucket? > > The amount of changes is split into the basic steps: > > 1) Make Node classes use slots. > 2) Make SigInfo classes use slots. > 3) Make Executor and Batch use slots and > stop caching full paths in File nodes. > > Do you have stats what objects contribute to the most amount of memory being > taken and what are the dependencies of these objects from the inputs? It is > interesting to see what is the major reason to SCons memory growth. Maybe we > should bzip strings on the fly. =) > > Does this slot optimization makes sense under pypy? I think it is a way to go > anyway if you want to optimize and speed up anything. > > I had to rewrite the memoizer count framework to use > decorators instead of the original metaclass approach. > Additionally, I had to correct a lot of tests and Tools > to ensure that they still pass without any fails > (at least for those tests that I can run locally under Linux, > some further adaptions might be necessary). > > Here are some numbers for the testsuite that I compiled > (it consists of several real-life SCons projects): > > Project | before | after | > ================================= > > ascend 96MB 82MB > bombono 120MB 104MB > lumiera 114MB 101MB > mapnik 148MB 129MB > sconsbld 540MB 377MB > > ================================= > > They list the maximum of allocated memory for a clean build, before and > after the optimizations. > > Impressive. But still 540MB of memory is not much for the most complex build. It's still quite a bit of memory, so reducing it and getting the speedup is a worthy effort. > > So, that's what I have right now. My questions are: > > - Is this of any interest (or is it still "too early" for > optimizations ;) )? > - If yes, what could be the next steps? Next steps would be running these code changes on a number of different system configurations. Taking a look at Mark's suggestion about turning File Node attributes into properties perhaps? > > I'd say - yes. It is too early for me. I don't feel that SCons code layout is > clean enough - there is still magic that could be less magic. Not all tags > are present in repository to compare. No pictures in documentation, and no > working bug tracker. > > The approximate wishlist: > [ ] make "import * from SCons" possible - it could be convenient to use SCons > like Fabric So you want to encapsulate SCons' running? This is a very infrequently requested feature, so I'd put this after most of the other items on your list. > [ ] create/find docbook template for Sphinx to keep docs old-style, but > easier to update (esp. on Windows) I think there was a user who was working on revamping the doc toolchain for SCons (Who that is is escaping me at the moment). > [ ] think about Python 3 I floated the question of the importance of this on the users mailing list a few minutes ago. > [ ] see what components are there, how are they decoupled and interchangeable > [ ] issues to a new tracker Not really a priority. We've discussed this a number of times in the past. What tracker would you suggest moving to? > [ ] restore tags What tags are missing? > > I didn't simply upload this as a pull request, because the changes > would definitely break existing projects and custom Tools. > So, either some form of deprecation cycle or a separate branch/repository > would be needed (default vs optimized). > > Comments anyone? > -Bill SCons Project Co-Manager.
_______________________________________________ Scons-dev mailing list [email protected] http://two.pairlist.net/mailman/listinfo/scons-dev
