I'm thinking that some combination of bugs in the bug tracking system + wiki to capture user stories, etc would be better than an email thread?
Here: https://bitbucket.org/scons/scons/wiki/browse/ToolchainRevamp On Tue, Oct 20, 2015 at 6:13 PM, Gary Oberbrunner <[email protected]> wrote: > You talk a lot about what I consider "policy": specific types of args that > would be used to configure specific types of Tools, like compilers. This is > all fine, but my goal is to build a framework where such policies can be > applied, and then get into the specifics of platforms, cross-compilers, > versions, targets and all that. Building a game asset manager, or a > visual-effects pipeline, or a document-preparation system, or a > scientific-data analysis pipeline, will have very different _specifics_ but > the same general mechanism should be usable for all, or at least that's my > goal. > > Jason, since you've already been down this path once, I'd very much > appreciate a code and design review of what I have so far. It's only 700 > lines or so, including tests. I'm especially interested in use cases Parts > can handle that my proposed system wouldn't be able to. (Again, from a > mechanism or base framework standpoint; not that I don't specifically > define a target architecture argument for instance.) > > One use case I see from your notes below is this one, which I'll capture: > As a SConscript writer, I'd like to be able to enumerate all versions of > a tool on the system so I can choose which one to use. > If there are others, please send them along. > > -- Gary > > On Tue, Oct 20, 2015 at 11:39 AM, Jason Kenny <[email protected]> wrote: > >> Hi Gary, >> >> >> >> From looking at the below this seems to be in line with what is going on >> in Parts. I had hoped that what I did here would be used /improved on in >> Scons. That is the a goal of Parts.. to improve Scons. >> >> >> >> From what you talk about below I think we are in general on the same >> page. I know when I started this I was also of the view that the “args” >> should be external. At the time Steve Knight release pushed hard that all >> value could be defined in the environment. I see some value in this, >> however I was like you in which I wanted to have something separate to >> define arguments to setup a tool. I also wanted to have a standard set of >> values. For example we should use the same values to talk about archecture, >> version, etc.. when we can as this makes it easier to use and develop new >> tools. Given what I have seen in practice, I was moving towards the IATAP >> idea from Greg. This is what I call Settings in Parts. While I have not >> finished this object yet I think it goes toward this idea better. Here we >> have an object that given values creates an environment to be used to build >> stuff. I think at a high level we want to move this way. >> >> >> >> For a design of the tools I think these are items we want to understand >> and define some way. >> >> >> >> *Platform*- Parts define a system platform object, Scons defines a >> string, for the OS, it has nothing standard for the architecture. I think >> we should agree we need to define this better in Scons. I would like to see >> something in Scons defining this. I suggest what I have in Parts as a way >> to start. I define in a given environment a HOST_PLATFORM object and a >> TARGET_PLATFORM Object. This is to extend the PLATFORM string we have now. >> The way I did in in Parts allows these items to be synced and to extended >> with new os or archecture value at run time. I think having something like >> this in SCons will allow for a better more reusable tool design. >> >> >> >> *Finders*- In Parts I have Finders and Scanners. These are different in >> what they return, but in effect are the same thing. I believe in the case >> of what Gary is talking about in Scons they serve the same purpose. The >> goal is it to return if we found something at a given resource location, >> given that being a path, registry entries, or environment variable. I >> should point out in someway this is like a autoconf test to find a >> compiler, etc. I would like to think this way is better however. I Parts I >> have finders and scanners. The difference in Parts is that a scanner >> returns back a set of value with version information, where a finder is >> much more simple “We have a match as resource X”. The reason for the >> difference is that some tools are easier to find as a set vs defining every >> known case. For example when I worked at Intel, the Intel compiler would >> make lots of internal drops as one would expect, these drops would have >> different versions. People would want to test the new drops out. Instead of >> me making a new entry for each new drop, I scanned the disk for example for >> what was installed. And returned all versions installed on disk. This >> scaled better for this tool set for users. Another example ( which I should >> improve a bit more) is the GNU tool change. People can install different >> version of the compiler on a Linux system and they often install them in >> different ways, such as under /opt or in there home directory in a standard >> way. I tried to have the Gnu logic in Parts look for these common places >> and use these version of requested. The use of a scanner logic was a lot >> easier here than making lots of custom finders. The main value of a finder >> vs scanner for me however is that a finder can be more declarative. This >> makes it easier to read and understand. For a system like SCons this is >> important for people to add new tools in a common way to be reused. >> >> >> >> *Tool args*- I don’t define this at the moment as an object. I define >> some common values, but not an object. Originally I was going to define an >> object. However Steve knight convinced me otherwise. There is some value in >> that choice. Making it an object has some different value. I not against or >> for making a tools arg like object. What I am against if a notions of a >> **kw object which was something Steve Knight basically had and what seems >> to be defined here in the new tool form Gary. The reason I have issue with >> this is that this allows the utter chaos we have in the current tools.. ie >> there is no standard way to set a given tool up. I agree a tool should be >> able to define unique items if it makes sense, but we should strive for a >> common set of basic values that we reuse in all tools. Some value like a >> version may be ignored as it does not make sense for it. We still should >> define a common set as most tool have a common ground and people need a >> common way to set them up. Sure advance cases will exists, we have a way >> with **kw to add advance stuff as it happens. For me value that are useful >> here are an idea of ( This is just a base set I found useful…not the end >> all end all) >> >> 1) *Root path* – this is the base path we use to find the tool and >> extend with extra info >> >> 2) *Version* – this is the version of the tool we want, or selector >> of the best match of the tool we want. ( ie 3.2.2 is exact where a 3 finds >> the highest version of 3 as 3.4 on the system. Yes many time we want the >> latest, or the version does not matter for the tool. This should exist, but >> does not have to be defined to get a tool to work >> >> 3) *Use_script* – for me this tells us who to setup the >> environment. I use it as a way to point a custom script to use to setup the >> environment, vs a set of known defaults or to point to a known script ( >> based on the root path) to setup the environment >> >> 4) *Target platform* – this is the target platform we want to build >> on. Not always needed, as we default to the host. Having a common want to >> talk about the target is powerful and very useful. I should not that many >> tools have different tools based on what host-target combo exists. Many >> cases we have x86 –> x64 and x64 -> x64 like tools on a platform. I never >> have seen a case in which not preferring a native client is the not the >> correct answer. This can be an important details as some may have a native >> x64 and other only have the x86->x64 case even though they have a x64 OS. >> >> >> >> *Tool environment instance*- I Parts I called this ToolInfo. This is not >> really different from what Gary talks about concreate instance of the tool. >> This is the object that is the information for a given instance of a tool >> we may want to use. Has information about what to add to the Environment. >> For me at least these object are define in a file and given finders and >> other information to expand to the environment when it is asked to fill in >> the environment with needed information to all a given tool to run/work. >> The main interface for this object at run time is an exist call and a get >> environment kv to add to the environment ( or a function to set the >> environment directly). Both of these functions would take some sort of >> tool_args. This object would use the finders and other information given to >> it to find if that instance exists. So for example it would have finders >> for getting information on the path, or shell variable to find a root >> directory to use to create the real path to the exe, which it can use to >> test that it really exists. A common case would be that you might have: >> >> >> >> Standard look up >> >> 1) Finder to path finder finds pat >> >> 2) This object uses that path and test that when we subst the full >> path value that the provided tool name exists >> >> More custom >> >> 1) Finder to path finders fails to find anything >> >> 2) Next finder in list is an environment finder, it find a the >> variable defined to it exists. >> >> 3) We use the value of this shell variable to subst the full path, >> and finds it the tools >> >> >> >> Either way this allow a way to find and setup what we need. In Parts I >> did the setup to allow for setting the environment via define defaults, or >> by calling a predefined script or a custom script. Scripts are slower to >> call, and add risk of differences (as they could be modified), but they add >> a lot of flexibility for custom setup cases while reusing the main tools >> logic. >> >> >> >> *Tool info container*- The other object needed is something to hold all >> these different tool cases. I called this ToolSetting in Parts. I did not >> make it as one global object, but many different obejcts, such as MSVC, >> GCC, etc. There might be some value to make it that way given what Gary >> proposes that I think would be an improvement on my design. The main point >> of this object for me is that this is the guy we register a set of tool >> information with, ie the exact platform requirements ( ie host and target >> it is for), and the tool info itself. This interface is a simple register >> at load time. If done right/well I should allow for an easy declarative >> definition, that should make it easy for other to read, copy and paste and >> tweak if needed. The other function is a query function to get a tool >> instance, based on some tool_args and to test for exist of some tool based >> on tool args. The only major function I would add in Parts that I don’t >> have in this case is a function to return all know information. This can be >> great to allow people to easily test via a simple command what the build >> system knows about. In Parts have this working via a hack of defining >> something we know does not exist to get an error message about what it >> could not find, and what are valid option the systems knows about.. just to >> note what I would change here ( minus the internally ugly but working impl >> I have at the moment). I also think this is a place I would add logic to >> cache known information about known tools to avoid re-looking it up for >> startup time improvements as a polish step. Would be useful for the generic >> check a lot of tools toolchain setups as these could be defined. >> >> >> >> *Toolchain* – This is a key feature that we need to have added. I think >> core ideas of easily defining a toolchain in some way is critical. Some >> quick thoughts. In Parts I never really got this done. What I do have is a >> way to define something in Files under a toolchain directory. What is great >> about this is that it easy for users to define there own changes, local to >> there own build or systems. That is a plus. What is not good. I did not >> define a good interface for this ( lack of time and I never got back to it >> correctly…) . This mean I cannot easily define a toolchain in a Sconstruct. >> I hope to address this with the Setting (IAPAT) object going forward. It is >> needed that user can define a toolchain in a the Sconstruct, in independent >> files and possibly have some control on the command line. On the command >> line I think user really need and want a way to influence a toolchain >> setup, such as use version X of a tool or use this path instead to find the >> tool, vs what is defined by default. In Parts I allow this to a degree with >> –toolchain (--tc for short). The argument is in a tool[_version],[…] >> format where tool can be a tool or toolchain. I would redo this ideally to >> have more of a --tc=toolchain[@key:value[@key:value],[...]] like format to >> pass in a kv value dict to help influence a toolchain setup. Beyond this we >> need a good way to define a toolchain. There are a lot of ideas out there >> for this. Common on include some sort of anyof(), oneof() combination to >> define more complex toolchains with fallbacks. Or having an idea of a >> common tool type such as CC and all tool define themselves as a subset of a >> “master” tool concept. I really open to ideas here myself. >> >> >> >> >> >> Ideas we can layer on top of this >> >> >> >> Once we do a better tools setup. We can layer an idea on top of it like >> configuration to allow applying common flags to the environment after the >> tools are configured. This is great as it easily allows common values to be >> used as a whole and easily changed to try a new set, without having to >> modify the build scripts. In the case of Parts this has been great for >> complex build and cross build cases as uses can easily define common flags, >> etc for a given tool/version/etc to be used based on the tool chain a >> environment is using. This should be separate at this point.. how I just >> point it out as it a layer to think about as it would reuse tool >> information in a toolchain to provide more value. Something to think about >> in the design of building up the tool correctly. >> >> >> >> >> >> I would hope that the concepts I talked about here are something we can >> find common ground on and talk about defining some common name for. I know >> this is a large e-mail, However I like very much some thoughts about what I >> suggest at high level to form a base of communication for use on the tool >> design. I am not saying my impl in Parts is what I would like it to be, but >> I think it does so common design points we want to discuss and the value of >> a reasonably easy to use and reusable system can have. >> >> >> >> Jason >> >> >> >> >> >> >> >> Sent from Mail <http://go.microsoft.com/fwlink/?LinkId=550986> for >> Windows 10 >> >> >> > > > > -- > Gary > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev > >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
