Sounds good... Technically I can open up what I have for the Scons nodes
rewrite as well.. I just trying to get the interfaces fixed and clear and clean.
I have with Parts two node trees. One for the SCons nodes, and one for the
component nodes. Ideally I would want to merge these together as one, with one
node manager, not many ( like we have in Scons). We need to get Symlinks in the
node tree for file based nodes. I have it working in Parts (yaaa...) It works
well, and allows us to define a gnu style symlink setup in a cross platform
way. I believe one thing I found with this is that we only need symlink for
file. Directory links are a can of worms.
For subst() stuff it is the memory size. There is a set of special objects that
are buggy and wasteful in memory. There is very little caching going on here.
This can be improved.
I should add from what I learned in what I have for Scons speed up with Parts
there are few things that can be done in SCons. So some thoughts here to think
about.
I store in my parts cache for a given cache key the same info SCons stores,
plus a lot more. For some reason the size I was seeing stored is less that the
master scons db file. I looked at it a little and honestly got lost in some
code that tries to mess with the pickled data.
>From that one area of speed in SCons is that all scons files are executed when
>they are read. From a reclusive makefile point of view this might be needed,
>but when you have a more component load model ( as in Parts) this is a BIG
>waste if we don't need to build that component. Currently Parts tries to
>figure out what it needs to load, and loads only that. This gives a huge speed
>up. When mixed in to some no depends/unsafe loading logic it is really fast (
>and useful is you are only trying to rebuild you top-level component... no
>need to load all the dependent components as these are already built and we
>know we did not modify them, or have the system to check to see if they are)
>This is the main reason why I would like to have the ability have a new format
>for Parts as this would allow for delayed execution.. ie something like ( very
>simple example):
PartName("foo")
PartVerson("1.2"
@config
def config(env):
... setup.. test stuff..
@emit:
Def emit(env)
Files=env.Glob(*.c)
Env.SharedLibrary("$PARTNAME_$PARTVERSION",Files)
In this case the need to configure a component or call emit ( lack of better
name) can be delayed unit we need to do it. For example here the Glob() call is
slow, as well as the SharedLibrary() call. The setting a name or adding a
CPPFLAG is fast.
I also believe that we can do a lot more with the DB/cache files. Given you
skip the python only builder issues. Most actions are command line ones. These
actions can be stored in the DB. Doing a set of Checks like Parts.. Scons can
easily detected ( in less than a second) if any files that define context of
what we would build. Given that is not changed, we load information about what
we last built instead of load it all from files and executing code to make a
tree of what to do again and instead just start building the nodes from the
stored action string without loading any build files. This is ideally the most
common case for developers as they are modifying code, not adding/removing file
or modifying build files. ( even in the case of adding removing files, we could
do a lot to reduce what is loaded)
I should say that Given some tweaks, I do believe SCons can do a correct safe
build much faster, vs doing trick to get speed up but with risk of an broken
build.
I should add that I have seen a lot of SCons files are more scripts. They call
execute() to build stuff vs letting SCons do what it does best. These seems to
be happen because the developer does not understand what SCons does. They still
used SCons because the Environment object provides and easy to under model to
control different tools chains and cross build in a single pass. This notion
seems to be much hard to deal with in other many other systems.
I am sure I have more stuff/experience to add.. but I need to get some work
done :-)
Jason
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Dirk Bächle
Sent: Monday, October 28, 2013 1:45 PM
To: SCons developer list
Subject: Re: [Scons-dev] Schedule for redesign?
Hi Jason,
and thanks for chiming in.
On 28.10.2013 17:07, Kenny, Jason L wrote:
> I hope I would not be a wrench in engine :-)
>
> Honestly The issue that this is work and I have to do this at home is
> the main reason I have not pushed anything at this point. ( And having
> twin 3 years-old means I don't have the go power at the end of the
> day...)
>
> I would agree with what is being stated. I would go a bit farther however.
I understand that you have done some work on redesigning/refactoring parts of
SCons, and you certainly don't want to lose it for nothing. So, I'd like to
make the following suggestion: I'll factor out some attributes of the Node
class into their own module and commit these changes (together with the other
rewritings I mentioned) to a named branch in my personal "experimental" SCons
fork.
Then you can have a look at it and check whether you'd be able to base your
further work on that. If not, you can tell me what should be changed such that
you can continue properly with your work. If I get your "okay", the changes
find their way into the SCons core and you can take it from there whenever you
find the time. How does that sound?
> I would redesign the Node API. But also redo the executioner logic. Ideally
> we want I have learned is that the task logic needs to be updated. Greg Noel
> was on the correct path with TNG in that we want to execute "builders" not
> nodes. We also need the API ( internal) to decouple the relationships with
> the tasks and the nodes.
>
> I would also look at fixing up the subst engine. I found that this is another
> area of memory waste at the moment ( and not in the good way). This might be
> more of an issue with Parts as I currently use the subst engine to share data
> between different "parts"/ components. Until I do a new part file format and
> or a "continuous loader" I have no way to pass data correctly.
During my investigations I tried to speedup subst() by caching some
pre-expanded strings, but it didn't work out. My current impression is that you
either have to compile it (including optimizations by hand, simply running
Cython over it is not enough!) or reduce functionality (like cutting down the
support for callables) to get it faster.
But don't let me stop you from trying... ;)
Best regards,
Dirk
_______________________________________________
Scons-dev mailing list
[email protected]
http://two.pairlist.net/mailman/listinfo/scons-dev
_______________________________________________
Scons-dev mailing list
[email protected]
http://two.pairlist.net/mailman/listinfo/scons-dev
