Hi Anatoly,
On 04.12.2012 06:57, anatoly techtonik wrote:
[...]
It will be easier to review this on rietveld.
I don't think we have reached full review state yet. As the comments and
suggestions (regarding properties, for example) show, the patch still
might need some work.
After getting more input from other users, I can upload the single
commits to a branch/repository somewhere and then also a review can
happen in the form of a normal pull request.
[...]
Do you have stats what objects contribute to the most amount of memory
being taken and what are the dependencies of these objects from the
inputs? It is interesting to see what is the major reason to SCons
memory growth. Maybe we should bzip strings on the fly. =)
Does this slot optimization makes sense under pypy? I think it is a
way to go anyway if you want to optimize and speed up anything.
The main portions of SCons are spent on:
1.) memoizing values/results (=caching), and
2.) Nodes.
By reducing the caching, you slow SCons down...I didn't want this. So I
took care of the nodes instead.
I found this out by:
ad 1.) removing all the _memo dicts (= no caching), and
ad 2.) adding my own Node class with the same memory layout. For
every Node that got created, I added
a copy...and then gradually removed attributes from this
"shadow" node class.
One of the basic problems is, that we always have to keep the whole tree
with all its nodes until the very end. It would help a lot if we could
throw away visited nodes, but this is not possible as long as we allow
the DAG to change during the build, while still correctly tracking
implicit dependencies.
[...]
sconsbld 540MB 377MB
=================================
They list the maximum of allocated memory for a clean build,
before and
after the optimizations.
Impressive. But still 540MB of memory is not much for the most complex
build.
As I wrote in my email, I also compared a project that used about 1.5GB
with the current default SCons version. I can't test anything larger
than that, because I only have a 2GB machine. If you have a bigger
project available just take the patch and run it through, with and
without slots. That's what this thread is all about... ;)
So, that's what I have right now. My questions are:
- Is this of any interest (or is it still "too early" for
optimizations ;) )?
- If yes, what could be the next steps?
I'd say - yes. It is too early for me. I don't feel that SCons code
layout is clean enough - there is still magic that could be less
magic. Not all tags are present in repository to compare. No pictures
in documentation, and no working bug tracker.
My very personal impression is: the SCons design is very good already. I
had to delve deeper into it while compiling my patch, and I'm liking it
in its current state. It will prove to be very hard to come up with a
different design, which saves significantly more memory while keeping
all the functionality and level of flexibility that we have now.
SCons will always be sort of a memory hog, unless you cut out major
features like the dependency tracking or the interactive mode. We can
only try to trim down the inherently humongous memory consumption and
that's what I wanted to accomplish.
Regards,
Dirk
_______________________________________________
Scons-dev mailing list
[email protected]
http://two.pairlist.net/mailman/listinfo/scons-dev
