Re: [Qbs] QBS script parsing speed
Nice! It brings a ~44% speed improvement in the problematic case according to your numbers: > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s Thanks, Richard! So for "medium" projects in terms of depth of the dependency tree, it's sensible to use transitive dependencies now: withLeafExports/4-8-implicit: real0m18.180s vs. withAdditionalDepends/4-8-explicit: real0m19.319s However, for "large" projects, the workaround still seems to be 25% faster: withLeafExports/9-2-implicit: real0m24.494s vs. withAdditionalDepends/9-2-explicit: real0m18.149s Best Jochen Am 30.06.20, 15:09 schrieb "Richard Weickelt" : Jochen, >> Well [the time] goes into "Handling Products". So it is ModuleLoader::handleProduct(). >> And there, it could be the ModuleMerger. But speculation is dangerous when discussing about performance. > See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html > > You can find my test results together with the scripts to generate the projects here: > https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Then you might be happy to hear that the ModuleMerger was completely rewritten for Qbs 1.16.0 (https://codereview.qt-project.org/c/qbs/qbs/+/284737). Running your benchmark with Qbs 1.16.0 (Qt Creator 4.12) gives me significantly better results in case of transitive dependencies. See below. I am not sure whether this has any noticable effect on real-world projects. But if ModuleMerger was ever a bottleneck, then it should not longer be. - i5-2520M CPU @ 2.50GHz Xubuntu 16.04) First line: Qbs 1.15.1 Second line: Qbs 1.16.0 > Resolving 4-8-direct real0m3.089s real0m2.669s > Resolving 4-8-implicit real0m4.808s real0m3.409s > Resolving 4-8-explicit real0m3.898s real0m3.085s > Resolving withLeafExports/4-8-direct real0m16.508s real0m13.793s > Resolving withLeafExports/4-8-implicit real0m25.907s real0m18.180s > Resolving withLeafExports/4-8-explicit real0m21.393s real0m15.844s > Resolving withLeafExports/9-2-direct real0m13.992s real0m10.750s > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s > Resolving withLeafExports/9-2-explicit real0m26.019s real0m17.589s > Resolving withAdditionalDepends/4-8-direct real0m25.096s real0m18.892s > Resolving withAdditionalDepends/4-8-implicit real0m28.369s real0m19.868s > Resolving withAdditionalDepends/4-8-explicit real0m29.164s real0m19.319s > Resolving withAdditionalDepends/9-2-direct real0m19.110s real0m17.264s > Resolving withAdditionalDepends/9-2-implicit real0m20.887s real0m19.269s > Resolving withAdditionalDepends/9-2-explicit real0m18.359s real0m18.149s > Resolving 9-2-direct real0m2.500s > Resolving 9-2-implicit real0m6.393s > Resolving 9-2-explicit real0m3.833s real0m2.326s ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
I already use the second variant. The maximum that I export this: Export { Depends { name: "cpp" } cpp.includePaths: product.exportIncludePaths } But to connect files, I use wildcard mask almost everywhere. In topic "Improving qbs resolve performance" (https://lists.qt-project.org/pipermail/qbs/2019-July/002481.html) Christian Kandeler does not recommend doing this. But now I can’t do it differently - there are a lots of files, and I can’t connect them all individually. -- BR, Pavel Karelin 30.06.2020 11:24, Jochen Ulrich пишет: We faced similar problems but our resolve times were even higher (30 s to 2 mins). We were able to work around this and significantly reduce the resolve times by avoiding transitive dependencies (meaning exporting dependencies). So instead of this: Product { name: “A” Depends { name: "B" } } Product { name: “B” Depends { name: "C" } Export { Depends { name: "C" } } } Product { name: “C” Depends { name: "SomeLibNeededEverywhere" } Export { Depends { name: "SomeLibNeededEverywhere" } } } <<< You do it like this Product { name: “A” Depends { name: "B" } Depends { name: "C" } Depends { name: "SomeLibNeededEverywhere" } } Product { name: “B” Depends { name: "C" } Depends { name: "SomeLibNeededEverywhere" } } Product { name: “C” Depends { name: "SomeLibNeededEverywhere" } } <<< Of course it is ugly because you have to maintain the transitive dependencies manually. However, if you have cases where a dependency is needed in many places, you can create custom products that include it. For example: // qbs/imports/MyBaseApplication.qbs CppApplication { Depends { name: "SomeLibNeededEverywhere" } } // in other .qbs files use MyBaseApplication instead of CppApplication MyBaseApplication { } <<< See also this thread on the mailing list: https://lists.qt-project.org/pipermail/qbs/2019-July/002477.html and the follow up: https://lists.qt-project.org/pipermail/qbs/2019-August/002544.html Kind regards Jochen Ulrich Von: Qbs im Auftrag von Карелин Павел Datum: Dienstag, 30. Juni 2020 um 09:45 An: "qbs@qt-project.org" Betreff: [Qbs] QBS script parsing speed Hello. Now I'm trying to write a QBS build script for a fairly large project (QGIS). After making every change to the build script - the project tree is rebuilt and QtCreator freezes for 10-15 seconds. It seems that 10-15 is not so much, but since I have to make a lot of changes in the script, in the aggregate the slowdown turns out to be quite significant. I suppose I was not the only one who faced a similar problem on large projects. I would like to know if any measures are being planned to increase the speed of re-parsing a project? I am using QtCreator 4.11.2 (QBS 1.15.1) -- BR, Pavel Karelin ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
Jochen, >> Well [the time] goes into "Handling Products". So it is >> ModuleLoader::handleProduct(). >> And there, it could be the ModuleMerger. But speculation is dangerous when >> discussing about performance. > See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html > > You can find my test results together with the scripts to generate the > projects here: > https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Then you might be happy to hear that the ModuleMerger was completely rewritten for Qbs 1.16.0 (https://codereview.qt-project.org/c/qbs/qbs/+/284737). Running your benchmark with Qbs 1.16.0 (Qt Creator 4.12) gives me significantly better results in case of transitive dependencies. See below. I am not sure whether this has any noticable effect on real-world projects. But if ModuleMerger was ever a bottleneck, then it should not longer be. - i5-2520M CPU @ 2.50GHz Xubuntu 16.04) First line: Qbs 1.15.1 Second line: Qbs 1.16.0 > Resolving 4-8-direct real0m3.089s real0m2.669s > Resolving 4-8-implicit real0m4.808s real0m3.409s > Resolving 4-8-explicit real0m3.898s real0m3.085s > Resolving withLeafExports/4-8-direct real0m16.508s real0m13.793s > Resolving withLeafExports/4-8-implicit real0m25.907s real0m18.180s > Resolving withLeafExports/4-8-explicit real0m21.393s real0m15.844s > Resolving withLeafExports/9-2-direct real0m13.992s real0m10.750s > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s > Resolving withLeafExports/9-2-explicit real0m26.019s real0m17.589s > Resolving withAdditionalDepends/4-8-direct real0m25.096s real0m18.892s > Resolving withAdditionalDepends/4-8-implicit real0m28.369s real0m19.868s > Resolving withAdditionalDepends/4-8-explicit real0m29.164s real0m19.319s > Resolving withAdditionalDepends/9-2-direct real0m19.110s real0m17.264s > Resolving withAdditionalDepends/9-2-implicit real0m20.887s real0m19.269s > Resolving withAdditionalDepends/9-2-explicit real0m18.359s real0m18.149s > Resolving 9-2-direct real0m2.500s > Resolving 9-2-implicit real0m6.393s > Resolving 9-2-explicit real0m3.833s real0m2.326s ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
My analysis back then was: > Well [the time] goes into "Handling Products". So it is > ModuleLoader::handleProduct(). > And there, it could be the ModuleMerger. But speculation is dangerous when > discussing about performance. See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html You can find my test results together with the scripts to generate the projects here: https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Best Jochen ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
> I would like to know if any measures are being planned to > increase the speed of re-parsing a project? I am using QtCreator 4.11.2 > (QBS 1.15.1) Some ideas: 1. Incremental project resolving: Take the dependency graph into account when re-resolving a project so that unchanged products are not evaluated again. 2. Parallel resolving: Evaluate unrelated products using multiple threads. Both would require a lot of refactoring work in the projectresolver and moduleloader modules. I would really like to do something about it or see somebody else doing something about it. But as a spare time job, this looks undoable. Nevertheless, we can try our best to make small steps forward. Richard ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
> We faced similar problems but our resolve times were even higher (30 s to > 2 mins). > > We were able to work around this and significantly reduce the resolve > times by avoiding transitive dependencies (meaning exporting > dependencies). I would like to understand why exactly this pattern makes the evaluation time explode. ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
We faced similar problems but our resolve times were even higher (30 s to 2 mins). We were able to work around this and significantly reduce the resolve times by avoiding transitive dependencies (meaning exporting dependencies). So instead of this: >>> Product { name: “A” Depends { name: "B" } } Product { name: “B” Depends { name: "C" } Export { Depends { name: "C" } } } Product { name: “C” Depends { name: "SomeLibNeededEverywhere" } Export { Depends { name: "SomeLibNeededEverywhere" } } } <<< You do it like this >>> Product { name: “A” Depends { name: "B" } Depends { name: "C" } Depends { name: "SomeLibNeededEverywhere" } } Product { name: “B” Depends { name: "C" } Depends { name: "SomeLibNeededEverywhere" } } Product { name: “C” Depends { name: "SomeLibNeededEverywhere" } } <<< Of course it is ugly because you have to maintain the transitive dependencies manually. However, if you have cases where a dependency is needed in many places, you can create custom products that include it. For example: >>> // qbs/imports/MyBaseApplication.qbs CppApplication { Depends { name: "SomeLibNeededEverywhere" } } // in other .qbs files use MyBaseApplication instead of CppApplication MyBaseApplication { } <<< See also this thread on the mailing list: https://lists.qt-project.org/pipermail/qbs/2019-July/002477.html and the follow up: https://lists.qt-project.org/pipermail/qbs/2019-August/002544.html Kind regards Jochen Ulrich > Von: Qbs im Auftrag von Карелин Павел > > Datum: Dienstag, 30. Juni 2020 um 09:45 > An: "qbs@qt-project.org" > Betreff: [Qbs] QBS script parsing speed > > Hello. > > Now I'm trying to write a QBS build script for a fairly large project (QGIS). > After making every change to the build script - the project tree is rebuilt > and QtCreator freezes for 10-15 seconds. > It seems that 10-15 is not so much, but since I have to make a lot of changes > in the script, in the aggregate the slowdown turns out to be quite > significant. > I suppose I was not the only one who faced a similar problem on large > projects. I would like to know if any measures are being planned to increase > the speed of re-parsing a project? > I am using QtCreator 4.11.2 (QBS 1.15.1) > > > -- > BR, Pavel Karelin ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
[Qbs] QBS script parsing speed
Hello. Now I'm trying to write a QBS build script for a fairly large project (QGIS). After making every change to the build script - the project tree is rebuilt and QtCreator freezes for 10-15 seconds. It seems that 10-15 is not so much, but since I have to make a lot of changes in the script, in the aggregate the slowdown turns out to be quite significant. I suppose I was not the only one who faced a similar problem on large projects. I would like to know if any measures are being planned to increase the speed of re-parsing a project? I am using QtCreator 4.11.2 (QBS 1.15.1) -- BR, Pavel Karelin ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs