Re: [m5-dev] bogus rebuild clue
Some more hints about what is going on from the SCons mailing list. Definitely could be our problem Ali Hi! I am currently tracking down a weird error that causes scons to rebuild files in my project when I run scons a 2nd time, even though the files were successfully built the first time. I have been tracking this in winpdb quite far, and I was hoping that someone would be able to accelerate my search. This is how far I have gotten: My project is a multi env project where I build a generator tool (exe) in one build environment and then I use that exe in a different environment (I use multiple envs as the exe is for win32 and the target is for arm) The exe is using precompiled headers and the PCH file is regenerated the 2nd time I run a clean build. The PCH has two target files, the .pch file and the .obj file. The reason why the PCH command is rerun is that that the dependency comparison retursn up_to_date False for the .obj file (whats strage is that the .pch file dependency is up to date!) When I looked closer _why_ the dependencies are not up to date, I noticed that the first entry of self.implicit for the .obj file was ..\cl.exe. This did not match the sig of the stored dependency. No wonder as the stored sig was the sig of a completely different file! The second time the build is run, the first implicit dependency has changed to match the stored sig and self.implicit now contained ..\cl.exe at the end of the list. As the list of implicit dependencies look different between the two runs, I am not surprised that the target looks out of date and 'fortunately' the implicit list will eventually 'stabilize' with .. \cl.exe at the end so this only causes rebuild once. However, as my project is rather large, I'd rather avoid the rebuild ;) Whats strange here is: Why is the implicit list different for the .pch and .obj? They are generated by the same emitter! Why is the implicit list for .obj changing beteween runs? If this sparks any ideas, let me know, in the meantime, I will keep looking and hopefully be able to write a test case that can trigger this. regards, Stefan Burstrom I think I have been able to narrow this down quite a bit. Here is the flow and what I think goes wrong: env.PCH() sets up a node with two targets: .pch and .obj In my env.PCH call I supply a file name that does not exist, it is generated during the build. When the build tree is considered and the .pch node is encountered, this triggers the generation of my file. This causes the implicit list in the .pch node to be cleared and recreated (in the call to Node.scan()) this will eventually trickle down to add_to_implicit which is going to add the new implicit dependencies. However, as the .obj node is also a part of the target, add_to_implicit is also called on the .obj node. But this time, the existing implicit list in the .obj node is not empty. The .pch and .obj node now has 2 lists of implicit dependencies that are differen (only the order though) even though they are targets depending on the same source file! These two lists are stored in the .sconsign file. Next time the build is run, the generated file exists, so the generation of the implicit deps lists will now result in two lists that are identical for .pch and for .obj. But as the previous stored info in the .sconsign file had the dependencies for .obj in a different order, .obj will now of course be out of date. This causes the .PCH command to be rerun. When the result from this build is stored in the .sconsign file, the new order of the deps is used which will now be consistent accross the runs. I guess I could fix this making the input file to the .PCH call be a static file. But at the end of the day, this really seem to be pointing out a flaw in the way scons deals with generated include files that are used as sources for builders with multiple targets. I dont yet have a test case for this, but my guess is that it should be fairly trivial to set one up that does not depend on using .PCH for triggering the problem (A simple custom builder with 2 target nodes should do it) A further investigation revealed the following: My build has a dependency in the .PCH call to a generated file. During the build process, the built() function is called when my generated file is ready. This causes scons to clear the implicit dependencies in the parent node to the generated file. However, the way my build executes, only the .pch file has been added as a waiting parent. The .obj file is not yet considered as a parent by the taskmaster. This causes the implicit list in the .pch file to be cleared (and subsequently updated), however, the implicit list for the .obj file will now contain the same entries but in a different order. It seems natural to clear the implicit list in all targets of a node when the one of the sources for the node has been built. (Instead of just clearing the implicit lists for the parents that have
Re: [m5-dev] bogus rebuild clue
I just did some searching and found this. http://comments.gmane.org/gmane.comp.programming.tools.scons.user/20816 Have you tried scons 1.3? Maybe it's fixed. Ali Sent from my ARM powered device On Mar 23, 2011, at 11:39 PM, Steve Reinhardt ste...@gmail.com wrote: I think many of us have noticed that scons sometimes rebuilds things when it's not necessary. Here's one clue: % scons build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] scons: done building targets. % scons --debug=explain build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] [SWIG] ALPHA_SE/python/swig/core.i - _wrap.cc, .py scons: rebuilding `build/ALPHA_SE/python/m5/internal/vptype_AddrRange.py' because the dependency order changed: old: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', '/usr/bin/swig', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i'] new: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i', '/usr/bin/swig'] [...] This is just one of many swig files that get regenerated... most of the dependency lists are much longer, but in all of the ones I looked at, /usr/bin/swig had moved from the second spot to the last spot on the list. Subsequent rebuilds say that `build/ALPHA_SE/m5.debug' is up to date, but if I delete build/ALPHA_SE/python/m5/internal then it takes *two* builds to get it back to that state again. Any ideas, Nate? Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
Re: [m5-dev] bogus rebuild clue
I just tried both 1.3.1 and 2.0.1 and the same thing happens in both. Steve On Wed, Mar 23, 2011 at 11:20 PM, Ali Saidi sa...@umich.edu wrote: I just did some searching and found this. http://comments.gmane.org/gmane.comp.programming.tools.scons.user/20816 Have you tried scons 1.3? Maybe it's fixed. Ali Sent from my ARM powered device On Mar 23, 2011, at 11:39 PM, Steve Reinhardt ste...@gmail.com wrote: I think many of us have noticed that scons sometimes rebuilds things when it's not necessary. Here's one clue: % scons build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] scons: done building targets. % scons --debug=explain build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] [ SWIG] ALPHA_SE/python/swig/core.i - _wrap.cc, .py scons: rebuilding `build/ALPHA_SE/python/m5/internal/vptype_AddrRange.py' because the dependency order changed: old: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', '/usr/bin/swig', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i'] new: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i', '/usr/bin/swig'] [...] This is just one of many swig files that get regenerated... most of the dependency lists are much longer, but in all of the ones I looked at, /usr/bin/swig had moved from the second spot to the last spot on the list. Subsequent rebuilds say that `build/ALPHA_SE/m5.debug' is up to date, but if I delete build/ALPHA_SE/python/m5/internal then it takes *two* builds to get it back to that state again. Any ideas, Nate? Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
[m5-dev] bogus rebuild clue
I think many of us have noticed that scons sometimes rebuilds things when it's not necessary. Here's one clue: % scons build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] scons: done building targets. % scons --debug=explain build/ALPHA_SE/m5.debug scons: Reading SConscript files ... [...] [SWIG] ALPHA_SE/python/swig/core.i - _wrap.cc, .py scons: rebuilding `build/ALPHA_SE/python/m5/internal/vptype_AddrRange.py' because the dependency order changed: old: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', '/usr/bin/swig', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i'] new: ['build/ALPHA_SE/python/m5/internal/vptype_AddrRange.i', 'build/ALPHA_SE/base/range.hh', 'build/ALPHA_SE/base/types.hh', 'build/ALPHA_SE/python/swig/range.i', '/usr/bin/swig'] [...] This is just one of many swig files that get regenerated... most of the dependency lists are much longer, but in all of the ones I looked at, /usr/bin/swig had moved from the second spot to the last spot on the list. Subsequent rebuilds say that `build/ALPHA_SE/m5.debug' is up to date, but if I delete build/ALPHA_SE/python/m5/internal then it takes *two* builds to get it back to that state again. Any ideas, Nate? Steve ___ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
