Jason, Any chance you can point to your improved _concat_ixes(), and any other routines you’ve improved and/or a pull request per improved function?
It does look like you really abused the values. Unless you’re playing games with them in scanners or emitters.. -Bill On Mon, Jul 24, 2017 at 7:52 PM Jason Kenny <[email protected]> wrote: > Hi Bill > > > > First sorry I double posted this.. the first case I did not finish, > fumbled a key and posted before I was ready… This version should be > “better”. > > > > Given Parts makes components I needed a way to share data. I tried the > Export Import logic, but had issues with it. In the end I used Import to > pass in the env and other functions I add in Parts. > > > > This is why all .parts files start with Import(‘*’). I could get around > this by taking more control, but this seemed to be counter productive to my > hope of getting Parts logic into SCons. > > > > So to pass data around and to request what data is needed we have a > DependsOn() function ( and a Exportxxx()/SDKxxx functions that push values > to be shareable” that allows the user define they depend on a component > with certain requires, like version or other features ( given more than on > Part is defined in the SConstruct as it common if you cross build in a > single pass). I this call you can define some values that you require from > that component via the REQ object. Certain values are passed automatically > such as CPPPATH, and LIBS.. ie values you need to compile against the > component, while other values need to be explicitly requested, such as > CPPFLAGS or custom values that may have been exporting from the Component. > When you DependOn([Component(A)) Parts will add some strings to map > implicated values. These may not get expanded at all and at worse waste > some space in the environment. Explicit value defined by the REQ meta > object also get mapped. For this e-mail, I will keep it simple and leave > out some features. > > > > The main point is that the string we add is something like > “{PARTIDEXPORTS(‘sub1@version:1.*’,’build’,’LIBS’,’1’)}” > > > > This call a parts.mapper object ( we have a few defined in Parts). This > object will look up the needed object and get the requested data in the > dependent component and add it to the environment variable in question. > This subst happens when SCons subst the code. We do some tricks to avoid > extra look ups and to replace values that are known. This way Part A can > depend on Part B and C and we can add to the LIBS and LIBSPATH variable the > needed information without the developer bothering to do this. What is also > nice about this is that if B was making a Boo.lib files and it changes to a > BOB1.2.lib file, the only change needed is the in B.part files. The change > will auto propagate correctly up the depends chain and everything will just > rebuild. As I stated this happens via calling a python object in the > subst() call and having the code correctly return a value and or a value > and update in place a value in the list ( as is common with CPPFLAGS, LIBS, > etc..) In the end, you do a build ( and you can look at samples in Parts > to see this happening) when the part file is done being read we might see > something like: > > > > Given a setup such as > > > > a->[B,C] > > B->[] > > C->[D] > > > > Env[‘LIBS’] -> [‘a.lib’,’ > {PARTIDEXPORTS(‘B@version:2.*’,’build’,’LIBS’,’1’)}’, > {PARTIDEXPORTS(‘C@version:1.*’,’build’,’LIBS’,’1’)}”] > > > > But when the subst for like “LINKCOM” happens it will get for the $LIBS a > list like -> [‘a.lib’,’b.lib’,’c.lib’, ’d.lib’]. technically this happens > with the Scanner first for items LIBS, LIBPATH, CPPPATH, so when the > command subst happens we already replaced ourselves to prevent redundant > logic from happening. > > > > The mapper code in all the glory can be seen here: > https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/mappers.py?at=master&fileviewer=file-view-default > > > > If you look at the history this was simpler code at first, but then Gary > complained I made it to C\C++ specific so it could never get into SCons, I > tweaked to make it more general, then I needed to make it better to allow > custom values. Other tweaks have been made as well over time to address > bugs or feature issues. The issue is that this code goes off a lot and > takes time to process in Large builds. Work has been done to replace the > SCons version of _concat_ixes(), and some other objects to make them not > faster. The last major update we had made was to try to fix an issue in > which we had a Vtune with some messed up depends circular depends in which > the mapper code would flip an include path around with mid-level dependent > components, cause a rebuild because SCons saw a change in the environment. > This code has been a sore spot for me to a large degree as it never quite > worked right in certain complex cases was done to delay subst values > because we would not know at load time which variant of Part C you needed > to get value from until we loaded all the components. The current form is a > bit complex and ugly honestly. There is some odd work around for some odd > behaviors with CmdStringHolder class. > > > > Given what I have learned... The plan is to replace this given I get time > to do this via implementing: > > > > 1. A new Parts format using @decorators to delay process functions ( > ie make the Parts file loading not an exec off a lot of python code, but a > register callback) This will allow me to load all the “parts” and have > needed information such as name, version, and other platform information > before I try to process what the component will emit as build actions. > 2. For the existing format implement a “continuous loader”.. or load a > Parts file until I get to a DependsOn statement. At the point, I pause and > start loading another Part until all the primary “top” level Parts are > loaded. At that point, I know enough information to know what is at the > bottom and what is at the top… This can be done with threads in earlier > versions of python or with python3 (3.5 in my head) asyncio and yield > “task” logic. > > > > In both of these cases, this allows me to delay “processing” the expensive > stuff, such as env.Program([list of 1000 files]) until I really need to > because I know it is the build “target” list. I also can on first pass ( ie > no DB) can load the Parts in order and insert the needed values directly. > Ie add “B.lib” or “b/includes” to the needed item. I don’t need to subst > and code. At most I might need to move some values around, such as libs to > make sure the linker is happy and we avoid double ( or more) inserts of the > same values. This would allow Parts to start SCons up much faster as we can: > > > > > > 1. Avoid loading all the components if the target is to not build > “all” or “.” > > > 1. Avoid a lot of subst() calls during the build allowing SCons, while > allowing me a nice way to share data in an organized way. This will also > allow me to share objects between components vs only strings. This will be > good as it will allow a “part” to define a builder and export it to be > shared by other components. Which is a more common request than I thought. > I have ugly work around at the moment to do this today.. but I rather not > speak about them 😊 > 2. Independent of mappers, this allows for other speed ups such as > possible avoiding of code execution such as env.Builder([lots of objects]) > what would not be called or better yet avoid calls to SCons Glob() or the > Parts Pattern() which scan the disk for files. This was found in profiling > to be one of the biggest reasons why it took the time to load a part. We > had cases such as boost in which a few scans happened ( one for headers to > export and few different ones for sources to build different boost > components) which would result in the file taking 30 seconds to load not > average .02 – .5 seconds on a laptop spinning disk the other parts > generally took that did not scan large number of files. ( this is one > reason for the use the cache better argument I made before, as we should > only need to scan a directory at most once, and then only again for a > change. I had a prototype this in Parts once and it showed a big > improvement in load time for the read phase.. never was finished it ☹, > but for me the point is that once we know what it is we should save it and > not reproduce calculating it until something says we need too. File and > directory nodes, and subst string values I had used to pass libs, paths > and other values between components are great examples of this ) > > > > I hope this helps… let me know if you have other questions. > > > > You can also install Parts with SCons on a box and run a some of my > samples.. add a --trace=mappers should give you a big dump of data. ( ie > add --verbose and --trace will dump everything… adding the [category] > printed with a verbose or trace message to the –trace or –verbose separated > with commas will allow you to filter to a smaller set of data. It will help > show what is going on and how Parts shares data > > > > Jason > > > > In case you want a simple example of what this might look like without > playing with code > > > > # hello part depends on print_msg parts > > PartName('hello') > > env.DependsOn([Component('print_msg','1.*')]) > > ret=env.Program("hello","hello.c") > > env.InstallTarget(ret) > > > > ------------ > > # the print_msg.part > > PartVersion('1.0.0') > > PartName('print_msg') > > > > #files > > cpp_files=['print_msg.c'] > > … > > outputs=env.SharedLibrary('print_msg',cpp_files) > > > > #This will auto export the print_msg.lib file as well other stuff related > to install sandbox and possible package creation > > env.InstallTarget(outputs) > > # this exports the print_msg.h include path > > env.SdkInclude(['print_msg.h']) > > > > --- > > #SConstruct > > > > from parts import * > > Part('hello/hello.parts') > > Part('print_msg/print_msg.parts') > > >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
