Thomas, There are no additional nodes only additional edges, so this becomes a bit tricky. The problem is that not all edges are equal. A new edge at the bottom of a tree makes almost no difference, but a single edge near the root (like your manifest file), can double the size of the tree.
V/R, William On Tue, Apr 12, 2016 at 1:26 PM, Thomas Berg <[email protected]> wrote: > William, > > Thank you, I think you are onto something here. I was about to send > the following observations when your last e-mail arrived: > > --- > > I did some diffing of the tree output, and found the following: > - part of the growth of the tree is indeed for Qt moc files. All the > correct header dependencies are listed here now. > - a big part of the grows is actually from Visual Studio manifest > files. We have been using env['WINDOWS_INSERT_MANIFEST'] = 1. Now, > with scons-2.5.0, the entire dependency-subtree for a dll is > duplicated for the corresponding manifest file. While this is correct, > I'm able to cut the tree size in half by turning this off. > > The only reason we us this flag is for the Program and SharedLibrary > builders then return the file in the target list, so it is easy to > copy the file to our target folder. While the new dependencies are > correct, it is of limited use for us, since the manifest file will > always be rebuilt if the dll/executable is rebuilt. > > So the growth of the dependency tree looks correct. It seems to me > that there is a huge potential for optimization of scons here though, > assuming that my understanding is correct: if scons could recognize > identical sub-trees in the dependency graph, and not process them > multiple times, speedup would be substantial. This would also bring us > far below our original 5 minute build time, because I see a lot of > duplicated subtrees in there too. > > --- > > After your mail I realize I didn't understand what --prune does. > Perhaps the dependency subtrees are not 100% identical between the > manifest files and the corresponding dlls, and this is the problem, > rather than identical sub-trees being processed multiple times? > > I will look more closely at what you suggested. > > > > On Tue, Apr 12, 2016 at 2:22 PM, William Blevins <[email protected]> > wrote: > > Thomas, > > > > The prune option keeps a group of dependencies from being printed more > than > > once. For example, you see the dependency tree for a c++ library L, but > > executable that files linking against L will only show that it depends > on L > > directly and give a line stating that the dependencies of L itself have > been > > pruned since they have already been listed previously. Considering this > > reduction of output, it is more likely that the 4-fold increase is > > exponential rather than linear. > > > > It might be wise to evaluate some of the new subtrees by diffing the two > > outputs. Chances are what's happening is that you have a N * M dependency > > relationship which you may or may not be able to untangle. > > > > To give you an idea of this change please see the original pull request. > Pay > > attention to the test updates to understand what dependencies were > > previously missing and pay attention to src/engine/SCons/executor.py and > > src/engine/SCons/Node/__init__.py for information on the functional > changes. > > > > I'm not sure that I can give you a 1-liner to disable the functionality > > sadly. Take a look first and see if you have any questions about why the > new > > dependencies exist. > > > > V/R, > > William > > > > > > > > On Tue, Apr 12, 2016 at 11:57 AM, Thomas Berg <[email protected]> > wrote: > >> > >> William, > >> > >> I ran scons --tree=prune on a small subset of our build, and the text > >> output goes up from 7.5MB to 35MB for that subset. I guess that means > >> the dependency graph is four times bigger now. > >> > >> Cheers, > >> Thomas > >> > >> On Tue, Apr 12, 2016 at 12:43 PM, Thomas Berg <[email protected]> > wrote: > >> > Bill, > >> > > >> > Here is the output of --debug=objects: > >> > > >> > > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_4_1.txt > >> > > >> > > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_2_5_0.txt > >> > > >> > Both are around 22MB, so here's a 3MB zip containing the two files: > >> > > >> > > https://dl.dropboxusercontent.com/u/40097540/misc/scons/scons_objects_output.zip > >> > > >> > The 2.5.0 output is around 2000 lines longer than the 2.4.1 output, > >> > both are around 430 000 lines. > >> > > >> > As a test, is it possible to disable the cross language scanner > >> > somehow, either by passing something to the environment or patching > >> > the scons code? > >> > > >> > Dirk: thanks for the link. I have tested some but not all the tricks > >> > used there, and I will definitely look closer at it after we > >> > (hopefully) get to the bottom of this. > >> > > >> > I would at least like to understand what's going on in our case, so > >> > thanks for all the help. Could it be that just adding a small number > >> > of dependencies in the wrong place can, due to recursive effects, > >> > cause such a significant increase in the processing time? > >> > > >> > Thanks, > >> > Thomas > >> > > >> > > >> > > >> > > >> > On Mon, Apr 11, 2016 at 6:02 PM, Dirk Bächle <[email protected]> wrote: > >> >> Hi Thomas, > >> >> > >> >> On 11.04.2016 16:59, Thomas Berg wrote: > >> >>> > >> >>> Bill, below is the output of --debug=count, it is identical with > >> >>> scons-2.4.1 and scons-2.5.0. > >> >>> > >> >>> Since my case was about the no-op build (nothing is built), less > >> >>> parallelization should not be an issue. ... > >> >> > >> >> > >> >> if you haven't already done so, you might want to try out my > "fastcpp" > >> >> extension at: > >> >> > >> >> https://bitbucket.org/dirkbaechle/scons_fastcpp > >> >> > >> >> . It may give you some speedup, but please regard its disclaimer. > >> >> > >> >> Best regards, > >> >> > >> >> Dirk > >> >> > >> >> > >> >> _______________________________________________ > >> >> Scons-dev mailing list > >> >> [email protected] > >> >> https://pairlist2.pair.net/mailman/listinfo/scons-dev > >> _______________________________________________ > >> Scons-dev mailing list > >> [email protected] > >> https://pairlist2.pair.net/mailman/listinfo/scons-dev > > > > > > > > _______________________________________________ > > Scons-dev mailing list > > [email protected] > > https://pairlist2.pair.net/mailman/listinfo/scons-dev > > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
