Hi Bill
First sorry I double posted this.. the first case I did not finish, fumbled a
key and posted before I was ready… This version should be “better”.
Given Parts makes components I needed a way to share data. I tried the Export
Import logic, but had issues with it. In the end I used Import to pass in the
env and other functions I add in Parts.
This is why all .parts files start with Import(‘*’). I could get around this by
taking more control, but this seemed to be counter productive to my hope of
getting Parts logic into SCons.
So to pass data around and to request what data is needed we have a DependsOn()
function ( and a Exportxxx()/SDKxxx functions that push values to be shareable”
that allows the user define they depend on a component with certain requires,
like version or other features ( given more than on Part is defined in the
SConstruct as it common if you cross build in a single pass). I this call you
can define some values that you require from that component via the REQ object.
Certain values are passed automatically such as CPPPATH, and LIBS.. ie values
you need to compile against the component, while other values need to be
explicitly requested, such as CPPFLAGS or custom values that may have been
exporting from the Component. When you DependOn([Component(A)) Parts will add
some strings to map implicated values. These may not get expanded at all and at
worse waste some space in the environment. Explicit value defined by the REQ
meta object also get mapped. For this e-mail, I will keep it simple and leave
out some features.
The main point is that the string we add is something like
“{PARTIDEXPORTS(‘sub1@version:1.*’,’build’,’LIBS’,’1’)}”
This call a parts.mapper object ( we have a few defined in Parts). This object
will look up the needed object and get the requested data in the dependent
component and add it to the environment variable in question. This subst
happens when SCons subst the code. We do some tricks to avoid extra look ups
and to replace values that are known. This way Part A can depend on Part B and
C and we can add to the LIBS and LIBSPATH variable the needed information
without the developer bothering to do this. What is also nice about this is
that if B was making a Boo.lib files and it changes to a BOB1.2.lib file, the
only change needed is the in B.part files. The change will auto propagate
correctly up the depends chain and everything will just rebuild. As I stated
this happens via calling a python object in the subst() call and having the
code correctly return a value and or a value and update in place a value in the
list ( as is common with CPPFLAGS, LIBS, etc..) In the end, you do a build (
and you can look at samples in Parts to see this happening) when the part file
is done being read we might see something like:
Given a setup such as
a->[B,C]
B->[]
C->[D]
Env[‘LIBS’] -> [‘a.lib’,’ {PARTIDEXPORTS(‘B@version:2.*’,’build’,’LIBS’,’1’)}’,
{PARTIDEXPORTS(‘C@version:1.*’,’build’,’LIBS’,’1’)}”]
But when the subst for like “LINKCOM” happens it will get for the $LIBS a list
like -> [‘a.lib’,’b.lib’,’c.lib’, ’d.lib’]. technically this happens with the
Scanner first for items LIBS, LIBPATH, CPPPATH, so when the command subst
happens we already replaced ourselves to prevent redundant logic from happening.
The mapper code in all the glory can be seen here:
https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/mappers.py?at=master&fileviewer=file-view-default
If you look at the history this was simpler code at first, but then Gary
complained I made it to C\C++ specific so it could never get into SCons, I
tweaked to make it more general, then I needed to make it better to allow
custom values. Other tweaks have been made as well over time to address bugs
or feature issues. The issue is that this code goes off a lot and takes time to
process in Large builds. Work has been done to replace the SCons version of
_concat_ixes(), and some other objects to make them not faster. The last major
update we had made was to try to fix an issue in which we had a Vtune with some
messed up depends circular depends in which the mapper code would flip an
include path around with mid-level dependent components, cause a rebuild
because SCons saw a change in the environment. This code has been a sore spot
for me to a large degree as it never quite worked right in certain complex
cases was done to delay subst values because we would not know at load time
which variant of Part C you needed to get value from until we loaded all the
components. The current form is a bit complex and ugly honestly. There is some
odd work around for some odd behaviors with CmdStringHolder class.
Given what I have learned... The plan is to replace this given I get time to do
this via implementing:
1. A new Parts format using @decorators to delay process functions ( ie make
the Parts file loading not an exec off a lot of python code, but a register
callback) This will allow me to load all the “parts” and have needed
information such as name, version, and other platform information before I try
to process what the component will emit as build actions.
2. For the existing format implement a “continuous loader”.. or load a Parts
file until I get to a DependsOn statement. At the point, I pause and start
loading another Part until all the primary “top” level Parts are loaded. At
that point, I know enough information to know what is at the bottom and what is
at the top… This can be done with threads in earlier versions of python or with
python3 (3.5 in my head) asyncio and yield “task” logic.
In both of these cases, this allows me to delay “processing” the expensive
stuff, such as env.Program([list of 1000 files]) until I really need to because
I know it is the build “target” list. I also can on first pass ( ie no DB) can
load the Parts in order and insert the needed values directly. Ie add “B.lib”
or “b/includes” to the needed item. I don’t need to subst and code. At most I
might need to move some values around, such as libs to make sure the linker is
happy and we avoid double ( or more) inserts of the same values. This would
allow Parts to start SCons up much faster as we can:
1. Avoid loading all the components if the target is to not build “all” or
“.”
2. Avoid a lot of subst() calls during the build allowing SCons, while
allowing me a nice way to share data in an organized way. This will also allow
me to share objects between components vs only strings. This will be good as it
will allow a “part” to define a builder and export it to be shared by other
components. Which is a more common request than I thought. I have ugly work
around at the moment to do this today.. but I rather not speak about them 😊
3. Independent of mappers, this allows for other speed ups such as possible
avoiding of code execution such as env.Builder([lots of objects]) what would
not be called or better yet avoid calls to SCons Glob() or the Parts Pattern()
which scan the disk for files. This was found in profiling to be one of the
biggest reasons why it took the time to load a part. We had cases such as boost
in which a few scans happened ( one for headers to export and few different
ones for sources to build different boost components) which would result in the
file taking 30 seconds to load not average .02 – .5 seconds on a laptop
spinning disk the other parts generally took that did not scan large number of
files. ( this is one reason for the use the cache better argument I made
before, as we should only need to scan a directory at most once, and then only
again for a change. I had a prototype this in Parts once and it showed a big
improvement in load time for the read phase.. never was finished it ☹, but for
me the point is that once we know what it is we should save it and not
reproduce calculating it until something says we need too. File and directory
nodes, and subst string values I had used to pass libs, paths and other values
between components are great examples of this )
I hope this helps… let me know if you have other questions.
You can also install Parts with SCons on a box and run a some of my samples..
add a --trace=mappers should give you a big dump of data. ( ie add --verbose
and --trace will dump everything… adding the [category] printed with a verbose
or trace message to the –trace or –verbose separated with commas will allow you
to filter to a smaller set of data. It will help show what is going on and how
Parts shares data
Jason
In case you want a simple example of what this might look like without playing
with code
# hello part depends on print_msg parts
PartName('hello')
env.DependsOn([Component('print_msg','1.*')])
ret=env.Program("hello","hello.c")
env.InstallTarget(ret)
------------
# the print_msg.part
PartVersion('1.0.0')
PartName('print_msg')
#files
cpp_files=['print_msg.c']
…
outputs=env.SharedLibrary('print_msg',cpp_files)
#This will auto export the print_msg.lib file as well other stuff related to
install sandbox and possible package creation
env.InstallTarget(outputs)
# this exports the print_msg.h include path
env.SdkInclude(['print_msg.h'])
---
#SConstruct
from parts import *
Part('hello/hello.parts')
Part('print_msg/print_msg.parts')
_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev
