Parts check a number of things to see if a Part might be out of date,
and if it could be, we load it, else we don’t read the file, and
instead load cached information if and only if there is a component
that depends on it.
I.e. Parts checks all sources, implicit dependencies, targets etc. by
itself.
I mean - not using SCons standard logic. And it works faster than if
SCons checked all these things.
Right?
Then what is the secret of Parts? How does it check all signatures much
faster than SCons?
For me it sounds like if I split a big project to many totally
independent small projects (SConstruct`s) and write a shell script:
| scons -C subprj1||&& \||
|| scons -C subprj2 && \||
|| scons -C subprj3 && \||
|| ...||
|| scons -C subprj200||
|
Then this shell script would work faster then one big project, but it
would be still significantly slower than Parts.
Right?
Thank you.
Best regards,
Constantine.
In general Parts uses the notion of a component ( or a file that
defines something) to make a “component” Dag. It checks to see if
anything modified that state ( parts file changed, Sconstruct changed,
configuration/toolchains was modified, source or output files for
component changed etc… which are mostly a Scons timestamp-MD5 check.
If any component is out of date or might be out of sync we mark it as
such and load any dependent component ( no need to check them as we
know there is “some” modification). This loads the items into SCons to
be processed. The resulting time to load and process is *much* smaller
on average.
Currently Parts is looking at removing our “mapper” logic, which is an
abuse of the subst() system in SCons to share data between components,
and moving to what is called “continuous loader” which will load the
Components in dependency order (making data sharing simpler). This
should also make the current logic for checking state simpler and faster.
From there the idea to look at is to store the actions defined in the
Component and replay them as part of the rebuild, as this can be done
once we know we have a sub tree that has nothing new added to it.
While those items build we continue to load more data ( as the build
would be waiting for build actions to complete). Given the common case
for a incremental build this should work out. As any relationships
defined explicitly or implicitly will not change. The most common
change will be in the source files, which will be seem as part of a
change that effect loading a component build file, or by the component
build file being changed. At any rate the dependent information gets
loaded as needed as processed to see what is safe or not safe to say.
This idea need more flushing out as there are details that could mess
up a correct build, or it might show to not speed anything up enough.
Jason
*From:*Scons-dev [mailto:[email protected]] *On Behalf Of
*Constantine
*Sent:* Sunday, January 25, 2015 3:24 AM
*To:* SCons developer list; Tom Tanner
*Subject:* Re: [Scons-dev] SCons-like build system
Do I understand correctly?
That the main idea is that using the Parts a big project is split into
many small parts.
And then SCons builds these parts more-or-less independently.
I.e. SCons processes many small DAGs instead of one big DAG.
Thank you.
Best regards,
Constantine.
On 01/23/15 18:03, Kenny, Jason L wrote:
I understand this problem. This is one main thing Parts tries to
address. It uses information about the different component to
figure out what not to load. This requires the build to be laid
out in some sort of component based like build, vs a recursive
make, which may not be a big deal, depending on how your project
is laid out. The result for example of a massive build I have here
is a “do nothing” build takes ~7-20 second vs the normal ~15-20
minutes it would normally take. Incremental build also are reduced
to minutes ( depending on what changed of course)
I honestly believe this can be better yet.. but we requires more
work. The main issue with spawn issue that we pushed, was found
when a build for this large product moved from rhel4 to rhel5. The
time increased from 2 hours to 4+. It was the spawning issue.
Something changed that really made this worse. Once we had this
fixed rhel5 build went to 2 hours and the old rhel4 based build
went to ~1.5 hours.
Jason
*From:*Scons-dev [mailto:[email protected]] *On Behalf
Of *Tom Tanner (BLOOMBERG/ LONDON)
*Sent:* Thursday, January 22, 2015 2:49 AM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: [Scons-dev] SCons-like build system
Having been poking around in this, I see that 'posix_spawn' cleans
to be availble on both aix and solaris (at least they have man
pages). Which is a relief for me. However, my current grief is
that a 'do nothing' build is still taking 20 minutes to traverse
the tree. *part* of this is because I check the md5sum on each
file to make sure someone hasn't gone and edit/patched/hacked a
generated target (which happens more often than is desirable), but
that isn't the bulk of the time taken.
I suspect I could leverage the fact that we use git to get the
sha1 of certain repositories that are known not to be changed by
us and if we find a file therein, to use the combined sha1 of the
repos might improve that at the cost of potentially spurious
rebuilds if one of the repos changed but the others didn't as i
wouldn't be reading all the files.
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
