Re: [m5-dev] what scons can do
(inline) One psychological solution would be to reduce the amount of output from SLICC, making it a less obvious target for annoyance when you're waiting for a build... Along this same line of thought, it may be easier in the interim to print out something like Continuing build ... or SLICC done or any type of progress message after the SLICC stuff is done. -- - Korey ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
-Original Message- From: m5-dev-boun...@m5sim.org [mailto:m5-dev-boun...@m5sim.org] On Behalf Of nathan binkert Sent: Thursday, April 21, 2011 5:53 PM To: M5 Developer List Subject: Re: [m5-dev] what scons can do Maybe so... I think there's a subconscious impression that it takes a while because there's a phase in the build that takes a noticeable amount of time and that's all the output you see. If in fact that delay is 10% running SLICC and 90% scons doing other stuff silently then I agree it's not such a big deal. I think it's more like 1%/99% :) Is that 1%/99% a statement of a clean build for m5.fast? I think the much more common case is you edit one .cc file and rebuild. In that situation, it sure seems like a lot more than 1% of the time is spent by scons regenerating and reanalyzing SLICC files. Whatever it may be, it sure would be great if we could speed things up. I'm happy to help however I can. Brad ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Is that 1%/99% a statement of a clean build for m5.fast? I think the much more common case is you edit one .cc file and rebuild. In that situation, it sure seems like a lot more than 1% of the time is spent by scons regenerating and reanalyzing SLICC files. Whatever it may be, it sure would be great if we could speed things up. I'm happy to help however I can. On a clean build maybe it's 1/10. It doesn't regenerate the SLICC files, it only parses them which only takes half a second (or probably less). It's other SCons internal machinery that's so darned slow. Nate ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Quoting nathan binkert n...@binkert.org: Anyway, it seems like it would be useful to be able to have multiple binaries that can be built by scons, specifically the utility stuff and unit tests. That way we could avoid having a hodge podge of small build systems which are either isolated or not in not quite the right ways. I know some of Nate's recent changes suggested this was going to get easier. Could you quickly summarize what that's all about, Nate? What changes are you thinking of? I'm not sure that anything I've done makes this easier or harder. I think it's just a matter of creating SConscript files that the SConstruct file sources. A mechanism where I can say binary foo needs file bar, and then when I tell it to build foo it build's it with bar, and bar doesn't get mixed in to other things. I think you're new widget that was like mercurial patch guards would help with that, right? You could have a guard for each extra target. Also, I was thinking about how to handle the dependencies/generated files/custom language issue a little while back, and what I kept coming back to were schemes where scons would use a cache of dependency information which it would regenerate if any of the input files which determined outputs and/or dependencies changed. The problem is that scons would need to run once and possibly regenerate its cache, and then run again to actually run. Is this sort of multi-pass setup possible somehow without major hacks? I've looked into this in depth and I think that SCons just sucks for this sort of thing. I've seen a few different proposals for dealing with this, but they all seem to suck. Basically SCons builds the dependence graph up front and then walks it. It doesn't seem to be able to build it on the fly. (WAF is different in this regard.) That said, there are hacks out there to get around this, but I haven't managed to get them to work and I'm not sure if they're fragile or not. That's unfortunate. When you run for the first time, scons would see that foo.isa.dep doesn't exist. During it's build phase, it would run foo.isa through the system and see that it generated foo_exec.cc and bar_exec.cc and put that into foo.isa.dep (as actual SConscript type code, or flat data, or...). When scons ran the second time, it would read in foo.isa.dep and extract the dependencies from it and build that into the graph. It wouldn't construct foo.isa.dep again since all its inputs were the same, and it would still capture all those dependencies. This time around, the larger binary would see that it depended on foo_exec.cc and bar_exec.cc and that those depend on foo.isa.dep (as a convenient aggregation point of all *.isa files involved). If foo.isa changed later, foo.isa.dep would be out of date and have to be regenerated, and then foo_exec.cc and bar_exec.cc, and then the main binary. Running SCons twice in the way you suggest would be FAR slower than just running SLICC and the ISA parser twice the way we do. Also, notice that when SLICC is run twice, the modes are very different. The first time it is run, it does parse the files, but only to figure out what the dependencies are. The second time it runs, it is run in the mode to actually generate the files. We parse all files twice anyway to get the dependency information (scanners parse things like .cc and .hh files to figure out dependencies with #includes, though they basically just use regexes to do it). I think you should do this with the ISA parser. I'm not sure how SLICC is structured, but it might be hard to get out file information without running the whole description, depending on how the multi-file output thing is implemented. I wrote an email about this a while ago, but basically it depends on if the output files are more like a list or a program. #include files are like a list where you can scan for them and know what they are without caring about any of the other text (or at least are sort of like that). The mulit-file output may be a lot more programmable and would be like determining what files a program is going to generate at run time. You could build in a dry run sort of mode that would just figure that out, but ISA descriptions tend to be pretty complicated and that would be a maintenance headache. It might still work out, though, since the descriptions tend to be relatively quick. That's compared to building the ginormous files they produce which can be S.L.O.W., especially if you start swapping. Gabe ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
A mechanism where I can say binary foo needs file bar, and then when I tell it to build foo it build's it with bar, and bar doesn't get mixed in to other things. I think you're new widget that was like mercurial patch guards would help with that, right? You could have a guard for each extra target. Ah, I see. I personally wouldn't use any of that. Don't use Source or PySource or that sort of thing. Just set up normal looking SConscript files with their own targets and such. The Source/PySource/etc. stuff is really about building the m5 binary and dealing with all of the variations that we have. I'm not sure how SLICC is structured, but it might be hard to get out file information without running the whole description, depending on how the multi-file output thing is implemented. In SLICC, we parse the entire file using the ply grammar and generate the AST in both phases. In one phase, you walk the AST simply to figure out which files will be generated. In the other, you actually do the generation. I'm suggesting that you do that as well, the files aren't that long and I bet that parsing the ISA desc files and generating the AST takes a second or so. Nate ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Quoting nathan binkert n...@binkert.org: A mechanism where I can say binary foo needs file bar, and then when I tell it to build foo it build's it with bar, and bar doesn't get mixed in to other things. I think you're new widget that was like mercurial patch guards would help with that, right? You could have a guard for each extra target. Ah, I see. I personally wouldn't use any of that. Don't use Source or PySource or that sort of thing. Just set up normal looking SConscript files with their own targets and such. The Source/PySource/etc. stuff is really about building the m5 binary and dealing with all of the variations that we have. Ok. I'm not sure how SLICC is structured, but it might be hard to get out file information without running the whole description, depending on how the multi-file output thing is implemented. In SLICC, we parse the entire file using the ply grammar and generate the AST in both phases. In one phase, you walk the AST simply to figure out which files will be generated. In the other, you actually do the generation. I'm suggesting that you do that as well, the files aren't that long and I bet that parsing the ISA desc files and generating the AST takes a second or so. That doesn't really fit with how the ISA files work. They get broken into an AST, but that gets consumed as it goes, and it has a lot of anonymous python in it that just gets executed somehow. I want to move more into the python, so the AST will be less and less useful. I'm not sure whether the new support will be with explicit language support or part of the embedded python. I would prefer the later, but if it's harder to use that way language support may be best. Ultimately I want to make the ISA stuff a part of regular python scripts instead of the python being embedded in the ISA stuff, so in the longer term it will probably be best to have that mechanism in regular python. Gabe ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
That doesn't really fit with how the ISA files work. They get broken into an AST, but that gets consumed as it goes, Does it have to be? and it has a lot of anonymous python in it that just gets executed somehow. I want to move more into the python, so the AST will be less and less useful. Does the AST not contain enough information to know what files are being generated? The anonymous python itself creates files? That sounds crazy. Nate ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
On Thu, Apr 21, 2011 at 2:11 PM, nathan binkert n...@binkert.org wrote: Running SCons twice in the way you suggest would be FAR slower than just running SLICC and the ISA parser twice the way we do. Are you sure? Running SLICC every time just to get a (typically unchanging) list of files is not exactly instantaneous, and the names of the SLICC output files hardly ever change. What about this approach: 1. store the list of output files 2. scons builds a dependence graph based on that stored list 3. when the SLICC files change and actually need to be re-parsed, we compare the output file list generated as a side effect with the stored list 4. if those lists differ, scons blows itself away and starts over 5. if not, we can get by without parsing the SLICC files at all in the common case where you're rebuilding the binary and none of the SLICC inputs have changed Yes, step 4 is ugly and expensive, but the key is that it should hardly ever happen. Make the common case fast, and all that. If we commit the output file lists to hg, it wouldn't even need to happen on a clean build. Just brainstorming... Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Are you sure? Running SLICC every time just to get a (typically unchanging) list of files is not exactly instantaneous, and the names of the SLICC output files hardly ever change. What about this approach: It is pretty fast (and this includes starting the python interpreter) 0.48s user 0.03s system 99% cpu 0.514 total 1. store the list of output files 2. scons builds a dependence graph based on that stored list 3. when the SLICC files change and actually need to be re-parsed, we compare the output file list generated as a side effect with the stored list 4. if those lists differ, scons blows itself away and starts over 5. if not, we can get by without parsing the SLICC files at all in the common case where you're rebuilding the binary and none of the SLICC inputs have changed Yes, step 4 is ugly and expensive, but the key is that it should hardly ever happen. Make the common case fast, and all that. If we commit the output file lists to hg, it wouldn't even need to happen on a clean build. Just brainstorming... I think you're pre-optimizing. How long does the ISA parser actually take to parse the files? Nate ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Quoting nathan binkert n...@binkert.org: That doesn't really fit with how the ISA files work. They get broken into an AST, but that gets consumed as it goes, Does it have to be? Making it not work that way would likely be very painful. The parser part is finicky (like they all are in any language) and we have lots and lots of very intricate code built on top of it in the form of the descriptions themselves. and it has a lot of anonymous python in it that just gets executed somehow. I want to move more into the python, so the AST will be less and less useful. Does the AST not contain enough information to know what files are being generated? The anonymous python itself creates files? That sounds crazy. This may not be quite right, but off hand here is a summary of the isa parser's inputs and outputs. Going in, the parser starts with main.isa. There are ##includes (there are two #s on purpose) which bring in other .isa files. The parser reads in all those files by following the ##includes, stitches them together into one huge string, and then crunches through it all. As that's being processed, the description can read in other files that have, for instance, microcode in them. This already hits basically the problem we're talking about since the involved are determined by execution, not static landmarks like ##include. I solve this problem by manually listing all microcode files in the SConscript. It's a nasty hack, but it avoids not rebuilding when microcode changes which is even more annoying. On the output side, the parser generates two files which implement the decoder, decoder.hh and decoder.cc. It also outputs one file for each CPU model involved that implements the exec (and related) functions. These are called something_something_exec.cc I think. The problem is that for x86 for sure, but also now for ARM and likely for any other ISA with a lot of complexity and/or fidelity, those output files get to be very, very large. It's easy to run out or RAM, especially if scons tries to build more than one at a time or if you're on a smaller machine. Then the build grinds to a halt, as does everything else. Often the only solutions are to wait until it finishes or you die (whichever comes first) or rebooting the machine and trying again with more conservative settings. What this mechanism would do would be to allow you to put different portions of the output into different files which would be compiled independently. Then scons compiling three things at once is equivalent to three normal files at once, not a million lines of code all at once. To do that, you have to decide how to split things up so they still build. You could try hacking things up in an automated way, but that would likely either be overly restrictive, ineffective, incorrect, or all three. My plan is to expose the idea of different files to the ISA description author so that they can choose to put all the, say, floating point loads and stores together along with their utility functions. These may not all be defined in the same place or even in the same directory since there are ordering constraints in python as well as in the resulting C++. It might be that you define output files in a fixed place (def output floatMem, for instance) and then refer to them later when it comes time to put C++ someplace. That might make the most sense. It could also be that you have batches of similarly named output clusters (these will likely involve more than one file at a time, like a .cc and a .hh) and you'd want to generate them all programatically. I'm not sure exactly what it would look like to select an output file either. You might want to just put down markers that say, essentially, henceforth output goes in floatMem. Or pass an output cluster name into the outputting function (whatever that looks like). It might work best in the near to mid term to put in static, ISA language defined declarations of output files which would be feasible to scan. In the mid to long term, though, I want to move away from having a custom language and move towards having the same machinery (extended and parameterized more) exposed as a module or something inside regular python scripts. Maybe something ala scons's SConscripts which are regular python that run in an armature, sort of. I would be hesitant to make the ISA descriptions open and write to files themselves directly, but primarily because that would be cumbersome and error prone. I don't think we should design it out, though, unless it's just too evil to support. Gabe ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
[m5-dev] what scons can do
I was looking at some of the stuff in util, and it occurred to me that the m5 utility program is cross compiled using different Makefiles for different architectures. Statetrace used to be like that (sort of), but recently I converted it over to scons and set up some configuration variables that made it easier to work with. It would be nice to be able to share some of that with the m5 utility program, although I don't remember it being all that complicated. Anyway, it seems like it would be useful to be able to have multiple binaries that can be built by scons, specifically the utility stuff and unit tests. That way we could avoid having a hodge podge of small build systems which are either isolated or not in not quite the right ways. I know some of Nate's recent changes suggested this was going to get easier. Could you quickly summarize what that's all about, Nate? Speaking of which, the regressions are still broken. Since that's taking a little while, would you mind please backing out the problem change? Also, I was thinking about how to handle the dependencies/generated files/custom language issue a little while back, and what I kept coming back to were schemes where scons would use a cache of dependency information which it would regenerate if any of the input files which determined outputs and/or dependencies changed. The problem is that scons would need to run once and possibly regenerate its cache, and then run again to actually run. Is this sort of multi-pass setup possible somehow without major hacks? To explain more what I'm getting, lets say you have input file foo.isa which, when processed, generates the files foo_exec.cc and bar_exec.cc. What would happen is that you'd have a file like foo.isa.dep which would describe what would happen and make that depend on foo.isa. When you run for the first time, scons would see that foo.isa.dep doesn't exist. During it's build phase, it would run foo.isa through the system and see that it generated foo_exec.cc and bar_exec.cc and put that into foo.isa.dep (as actual SConscript type code, or flat data, or...). When scons ran the second time, it would read in foo.isa.dep and extract the dependencies from it and build that into the graph. It wouldn't construct foo.isa.dep again since all its inputs were the same, and it would still capture all those dependencies. This time around, the larger binary would see that it depended on foo_exec.cc and bar_exec.cc and that those depend on foo.isa.dep (as a convenient aggregation point of all *.isa files involved). If foo.isa changed later, foo.isa.dep would be out of date and have to be regenerated, and then foo_exec.cc and bar_exec.cc, and then the main binary. The net effect of this is that the thing that processed the .isa would only be run when necessary. In our current setup, that would mean SLICC wouldn't have to be run for every build, only ones where the SLICC input files changed. The problem here is that scons would need to basically call a nested instance of itself on foo.isa.dep, let that build a dep tree and run the build phase, then process foo.isa.dep in the parent dep phase, and then run the parent build phase. It could literally just call scons from scons (though that seems like a major hack) or it could, if scons has a facility for it, do some sort of fancy multi-pass thing. This is sort of related to the first thing (additional targets) because the dependency cache files are sort of like independent targets with their own invocations of scons. Also related to scons are those .pyc files that end up scattered around the source tree. I know I asked about those a long, long time ago, but why are they there? Why don't they end up in the build directories? Gabe ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
On Tue, Apr 19, 2011 at 3:00 AM, Gabe Black gbl...@eecs.umich.edu wrote: I was looking at some of the stuff in util, and it occurred to me that the m5 utility program is cross compiled using different Makefiles for different architectures. Statetrace used to be like that (sort of), but recently I converted it over to scons and set up some configuration variables that made it easier to work with. It would be nice to be able to share some of that with the m5 utility program, although I don't remember it being all that complicated. Anyway, it seems like it would be useful to be able to have multiple binaries that can be built by scons, specifically the utility stuff and unit tests. That way we could avoid having a hodge podge of small build systems which are either isolated or not in not quite the right ways. If you're suggesting that the stuff in util get built by scons and put the binaries somewhere like build/util, I think that's a great idea. I don't know of the pros and cons of making it an indepedent invocation of scons vs integrating it into our global scons configuration, but whatever way we do it should avoid redundant scons code for things like detecting your compiler version, which makes me think the integrated approach might be better. Either way might require some refactoring between the SConstruct file and src/SConscript, since it's not under src, but we should be able to work that out. Also, I was thinking about how to handle the dependencies/generated files/custom language issue a little while back, and what I kept coming back to were schemes where scons would use a cache of dependency information which it would regenerate if any of the input files which determined outputs and/or dependencies changed. The problem is that scons would need to run once and possibly regenerate its cache, and then run again to actually run. Is this sort of multi-pass setup possible somehow without major hacks? To explain more what I'm getting, lets say you have input file foo.isa which, when processed, generates the files foo_exec.cc and bar_exec.cc. What would happen is that you'd have a file like foo.isa.dep which would describe what would happen and make that depend on foo.isa. When you run for the first time, scons would see that foo.isa.dep doesn't exist. During it's build phase, it would run foo.isa through the system and see that it generated foo_exec.cc and bar_exec.cc and put that into foo.isa.dep (as actual SConscript type code, or flat data, or...). When scons ran the second time, it would read in foo.isa.dep and extract the dependencies from it and build that into the graph. It wouldn't construct foo.isa.dep again since all its inputs were the same, and it would still capture all those dependencies. This time around, the larger binary would see that it depended on foo_exec.cc and bar_exec.cc and that those depend on foo.isa.dep (as a convenient aggregation point of all *.isa files involved). If foo.isa changed later, foo.isa.dep would be out of date and have to be regenerated, and then foo_exec.cc and bar_exec.cc, and then the main binary. The net effect of this is that the thing that processed the .isa would only be run when necessary. In our current setup, that would mean SLICC wouldn't have to be run for every build, only ones where the SLICC input files changed. The problem here is that scons would need to basically call a nested instance of itself on foo.isa.dep, let that build a dep tree and run the build phase, then process foo.isa.dep in the parent dep phase, and then run the parent build phase. It could literally just call scons from scons (though that seems like a major hack) or it could, if scons has a facility for it, do some sort of fancy multi-pass thing. This is sort of related to the first thing (additional targets) because the dependency cache files are sort of like independent targets with their own invocations of scons. Caching the list of generated SLICC files sounds like a good idea to me. I'm not sure this would require recursive scons invocations, since we manage to build the list dynamically already without that. I wouldn't call it a cache of dependency information though, since scons already has one of those; this is really just a cache of generated filenames, right? Also related to scons are those .pyc files that end up scattered around the source tree. I know I asked about those a long, long time ago, but why are they there? Why don't they end up in the build directories? That's an artifact of including them in scons, where they're run in place and not by m5. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
On Tue, Apr 19, 2011 at 3:13 PM, Gabriel Michael Black gbl...@eecs.umich.edu wrote: Caching the list of generated SLICC files sounds like a good idea to me. I'm not sure this would require recursive scons invocations, since we manage to build the list dynamically already without that. I wouldn't call it a cache of dependency information though, since scons already has one of those; this is really just a cache of generated filenames, right? How would you be able to do it all in one shot? The tricky part is that you actually have to build the .dep in the build phase, but you need it in the dependency tree generating phase. Since you can't go backwards like that in one invocation (as far as I know or can imagine) then you'd need to rounds. OK, I see, maybe it is inherently another level beyond what we currently do. I'm not the scons expert... As far as what it would be caching I think it's largely a semantic difference. You could consider it a cache of generated files which are used to set up the dependencies. It's just terminology... not that it's wildly inaccurate to call it dependency info, since it is info that is eventually used to determine dependencies, just that there's already a dependency caching feature in scons that's totally different (http://www.scons.org/doc/2.0.1/HTML/scons-user.html#AEN1148), and in general when the scons docs talk about dependencies it's what are the files that this file depends on not what are the files that depend on this file. Thus it would be less confusing if you avoided referring to the info that you're discussing here as dependency info and used a different term like generated file info. That's all. Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] what scons can do
Quoting Steve Reinhardt ste...@gmail.com: On Tue, Apr 19, 2011 at 3:13 PM, Gabriel Michael Black gbl...@eecs.umich.edu wrote: Caching the list of generated SLICC files sounds like a good idea to me. I'm not sure this would require recursive scons invocations, since we manage to build the list dynamically already without that. I wouldn't call it a cache of dependency information though, since scons already has one of those; this is really just a cache of generated filenames, right? How would you be able to do it all in one shot? The tricky part is that you actually have to build the .dep in the build phase, but you need it in the dependency tree generating phase. Since you can't go backwards like that in one invocation (as far as I know or can imagine) then you'd need to rounds. OK, I see, maybe it is inherently another level beyond what we currently do. I'm not the scons expert... Me neither... Anybody else? As far as what it would be caching I think it's largely a semantic difference. You could consider it a cache of generated files which are used to set up the dependencies. It's just terminology... not that it's wildly inaccurate to call it dependency info, since it is info that is eventually used to determine dependencies, just that there's already a dependency caching feature in scons that's totally different (http://www.scons.org/doc/2.0.1/HTML/scons-user.html#AEN1148), and in general when the scons docs talk about dependencies it's what are the files that this file depends on not what are the files that depend on this file. Thus it would be less confusing if you avoided referring to the info that you're discussing here as dependency info and used a different term like generated file info. That's all. Yes, I see why that could be confusing. I won't call it that any more (unless I forget). Gabe ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
