Hi Bill
Given Parts makes components I needed a way to share data. I tried the Export
Import logic, but had issues with it. In the end I used Import to pass in the
env and other functions I add in Parts.
This is why all .parts files start with Import(‘*’). I could get around this my
taking more control, but this seemed to be counter productive to my hope of
getting Parts logic into SCons.
So to pass data around and to request what data is needed we have a DependsOn()
function ( and a Exportxxx()/SDKxxx functions that push values to be sharable”
that allows the user define they depend on a component with certain requires,
like version or other features ( given more than on Part is defined in the
SConstruct as it common if you cross build in a single pass). I this call you
can define some values that you require from that component via the REQ object.
Certain values are pass automatically such as CPPPATH, and LIBS.. ie values you
need to compile against the component, while other values need to be explicitly
requested, such as CPPFLAGS or custom values that may have been export for the
Component. When you DependOn([Component(A)) Parts will add some strings to map
implicated values. These may not get expanded at all and at worse waste some
space in the environment. Explicted value defined by the REQ meta object also
get mapped. For this e-maili will keep it simple and leave out some features.
The main point is that the string we add is something like
“{PARTIDEXPORTS(‘sub1@version:1.*’,’build’,’LIBS’,’1’)}”
This call a parts.mapper object ( we have a few defined in Parts). This object
will look up the needed object and get the requested data in the dependent
component and add it to the environment variable in question. This subst
happens when SCons subst the code. We do some tricks to avoid extra look ups
and to replace values that are known. This way Part A can depend on Part B and
C and we can add to the LIBS and LIBSPATH variable the needed information
without the developer bothering to do this. What is also nice about this is
that if B was making a Boo.lib files and it changes to a BOB1.2.lib file, the
only change needed is the in B.part files. The change will auto propagate
correctly up the depends chain and everything will just rebuild. As I stated
this happens via calling an python object in the subst() call and having the
code correctly return a value and or a value and update in place a value in the
a list ( as is common with CPPFLAGS, LIBS, etc..) In the end you do a build (
and you can look at samples in Parts to see this happening) when the part file
is done being read we might see something like:
Given a setup such as
a->[B,C]
B->[]
C->[D]
Env[‘LIBS’] -> [‘a.lib’,’ {PARTIDEXPORTS(‘B@version:2.*’,’build’,’LIBS’,’1’)}’,
{PARTIDEXPORTS(‘C@version:1.*’,’build’,’LIBS’,’1’)}”]
But when the subst for like “LINKCOM” happens it will get for the $LIBS a list
like -> [‘a.lib’,’b.lib’,’c.lib’, ’d.lib’]. technically this happens with the
Scanner first for items LIBS, LIBPATH, CPPPATH, so when the command subst
happens we already replaced ourselves to prevent redundant logic from happening.
If you look at the history this was simple at first, but then Gary complained I
made it to C\C++ specific so it could never get into SCons, I tweaked to make
it more general, then I needed to make it better to allow custom values. In
this end other tweaks have been made. The issue is that this code goes off a
lot and takes time to process in Large builds. Work has been done to replace
the SCons version of _concat_ixes(), and some other objects to make them not
faster. The last major update we had made was to try to fix an issue in which
we had a Vtune with some messed up depends circular depends in which the mapper
code would flip an include path around with mid-level dependent components,
cause a rebuild because SCons saw a change in the environment. This code has
been a sore spot for me to a degree as it never quite worked right in certain
complex cases was done to delay subst values because we would not know at load
time which variant of Part C you needed to get value from until we loaded all
the components. The current form is a bit complex and ugly honestly. There are
some odd work around for some odd behavior with CmdStringHolder class.
Given what I have learned…. The plan is to replace this given I get time to do
this via implementing:
1. A new Parts format using @decorators to delay process functions ( ie make
the Parts file loading not an exec off a lot of python code, but a register
callback) This will allow me to load all the “parts” and have needed
information such as name, version and other platform information before I try
to process what the component will emit as build actions.
2. For the existing format implement a “continuous loader”.. or load a
Parts file until I get to a DependsOn statement. At the point I pause and start
loading another Part until all the primary “top” level Parts are loaded. At
that point I know enough information to know what is at the bottom and what is
at the top…
In both of these cases this allows me to delay “processing” the expensive
stuff, such as env.Program([list of 1000 files]) until I really need to because
I know it is the build “target” list. I also can on first pass ( ie no DB) can
load the Parts in order and insert the needed values directly. Ie add “B.lib”
or “b/includes” to the needed item. I don’t need to subst and code. At most I
might need to move some values around, such as libs to make sure the linker is
happy and we avoid double ( or more) inserts of the same values. This would
allow Parts to start SCons up much faster as we can:
1. Avoid loading all the components if the target is to not build “all” or
“.”
2. Avoid a lot of subst() calls during the build allowing SCons, while
allowing me a nice way to share data in an organized way. This will also allow
me to share objects between components vs only strings. This will be good as it
will allow a “part” to define a builder and export it to be shared by other
components. Which is a more common request than I though. I have ugly work
around at the moment to do this today.. but I rather not speak about them 😊
3. Independent of mappers, this allows for other speed ups such as possible
avoiding of code execution such as env.Builder([lots of objects]) what would
not be called or better yet avoid calls to SCons Glob() or the Parts Pattern()
which scan the disk for files. This was found in profiling to be one of the
biggest reasons why it took time to load a part. We had cases such as boost in
which the a few scans happened ( one for headers to export and few different
ones for sources to build different boost components) which would resulted in
the file taking 30 second to load not average .02 – .5 seconds on a laptop
spinning disk the other parts generally took that did not scan large number of
files. ( this is one reason for the use cache better argument I made before, as
we should only need to scan a directory at most once, and then only again on a
change. I had prototype this in Parts once and it showed a big improvement in
load time for the read phase.. never was finished it ☹, but for me the point is
that once we know what it is we should save it and not reproduce calculating it
until something says we need too. File and directory nodes, and subst string
values I had used to pass libs, paths and other values between components are
great examples of this )
I hope this helps… let me know if you have other questions.
You can also install Parts with SCons on a box and run a some of my samples..
add a --trace=mappers should give you a big dump of data. ( ie add --verbose
and --trace will dump everything… adding the [category] printed with a verbose
or trace message to the –trace or –verbose separated with commas will allow you
to filter to a smaller set of data. It will help show what is going one and how
Parts shares data
Jason
In case you want a simple example of what this might look like without playing
with code
# hello part depends on print_msg parts
PartName('hello')
env.DependsOn([Component('print_msg','1.*')])
ret=env.Program("hello","hello.c")
env.InstallTarget(ret)
------------
# the print_msg.part
PartVersion('1.0.0')
PartName('print_msg')
#files
cpp_files=['print_msg.c']
…
outputs=env.SharedLibrary('print_msg',cpp_files)
#This will auto export the print_msg.lib file as well other stuff related to
install sandbox and possible package creation
env.InstallTarget(outputs)
# this exports the print_msg.h include path
env.SdkInclude(['print_msg.h'])
---
#SConstruct
from parts import *
Part('hello/hello.parts')
Part('print_msg/print_msg.parts')
From: Bill Deegan [mailto:[email protected]]
Sent: Monday, July 24, 2017 5:40 PM
To: SCons developer list <[email protected]>; Jason Kenny <[email protected]>
Subject: Variable Substitution
Jason,
You mentioned:
Variable Substitution:
I abuse this in Parts to share data in a lazy fashion between components. It
has been a sore point for me, given reason stated below. We have done some work
to address the items by reusing states better. I can say there are some issues
with the current code that causes memory bloat and wasted time. I don’t want to
dwell on this, but will say that this is the second biggest item in my mind
that would have a big impact to overall time to the user. I know I want to
change the load logic in Parts to avoid using the substitution engine as much
as possible.
Can you expand upon how you abuse variable substitution?
Is it about storing information for later use that will never really be
expanded by Subst()?
Or something else?
Thanks,
Bill
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
