Re: [Qbs] QBS script parsing speed
Nice! It brings a ~44% speed improvement in the problematic case according to your numbers: > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s Thanks, Richard! So for "medium" projects in terms of depth of the dependency tree, it's sensible to use transitive dependencies now: withLeafExports/4-8-implicit: real0m18.180s vs. withAdditionalDepends/4-8-explicit: real0m19.319s However, for "large" projects, the workaround still seems to be 25% faster: withLeafExports/9-2-implicit: real0m24.494s vs. withAdditionalDepends/9-2-explicit: real0m18.149s Best Jochen Am 30.06.20, 15:09 schrieb "Richard Weickelt" : Jochen, >> Well [the time] goes into "Handling Products". So it is ModuleLoader::handleProduct(). >> And there, it could be the ModuleMerger. But speculation is dangerous when discussing about performance. > See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html > > You can find my test results together with the scripts to generate the projects here: > https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Then you might be happy to hear that the ModuleMerger was completely rewritten for Qbs 1.16.0 (https://codereview.qt-project.org/c/qbs/qbs/+/284737). Running your benchmark with Qbs 1.16.0 (Qt Creator 4.12) gives me significantly better results in case of transitive dependencies. See below. I am not sure whether this has any noticable effect on real-world projects. But if ModuleMerger was ever a bottleneck, then it should not longer be. - i5-2520M CPU @ 2.50GHz Xubuntu 16.04) First line: Qbs 1.15.1 Second line: Qbs 1.16.0 > Resolving 4-8-direct real0m3.089s real0m2.669s > Resolving 4-8-implicit real0m4.808s real0m3.409s > Resolving 4-8-explicit real0m3.898s real0m3.085s > Resolving withLeafExports/4-8-direct real0m16.508s real0m13.793s > Resolving withLeafExports/4-8-implicit real0m25.907s real0m18.180s > Resolving withLeafExports/4-8-explicit real0m21.393s real0m15.844s > Resolving withLeafExports/9-2-direct real0m13.992s real0m10.750s > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s > Resolving withLeafExports/9-2-explicit real0m26.019s real0m17.589s > Resolving withAdditionalDepends/4-8-direct real0m25.096s real0m18.892s > Resolving withAdditionalDepends/4-8-implicit real0m28.369s real0m19.868s > Resolving withAdditionalDepends/4-8-explicit real0m29.164s real0m19.319s > Resolving withAdditionalDepends/9-2-direct real0m19.110s real0m17.264s > Resolving withAdditionalDepends/9-2-implicit real0m20.887s real0m19.269s > Resolving withAdditionalDepends/9-2-explicit real0m18.359s real0m18.149s > Resolving 9-2-direct real0m2.500s > Resolving 9-2-implicit real0m6.393s > Resolving 9-2-explicit real0m3.833s real0m2.326s ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
I already use the second variant.
The maximum that I export this:
Export {
Depends { name: "cpp" }
cpp.includePaths: product.exportIncludePaths
}
But to connect files, I use wildcard mask almost everywhere. In topic
"Improving qbs resolve performance"
(https://lists.qt-project.org/pipermail/qbs/2019-July/002481.html)
Christian Kandeler does not recommend doing this. But now I can’t do it
differently - there are a lots of files, and I can’t connect them all
individually.
--
BR, Pavel Karelin
30.06.2020 11:24, Jochen Ulrich пишет:
We faced similar problems but our resolve times were even higher (30 s to 2
mins).
We were able to work around this and significantly reduce the resolve times by
avoiding transitive dependencies (meaning exporting dependencies).
So instead of this:
Product {
name: “A”
Depends { name: "B" }
}
Product {
name: “B”
Depends { name: "C" }
Export {
Depends { name: "C" }
}
}
Product {
name: “C”
Depends { name: "SomeLibNeededEverywhere" }
Export {
Depends { name: "SomeLibNeededEverywhere" }
}
}
<<<
You do it like this
Product {
name: “A”
Depends { name: "B" }
Depends { name: "C" }
Depends { name: "SomeLibNeededEverywhere" }
}
Product {
name: “B”
Depends { name: "C" }
Depends { name: "SomeLibNeededEverywhere" }
}
Product {
name: “C”
Depends { name: "SomeLibNeededEverywhere" }
}
<<<
Of course it is ugly because you have to maintain the transitive dependencies
manually.
However, if you have cases where a dependency is needed in many places, you can
create custom products that include it.
For example:
// qbs/imports/MyBaseApplication.qbs
CppApplication {
Depends { name: "SomeLibNeededEverywhere" }
}
// in other .qbs files use MyBaseApplication instead of CppApplication
MyBaseApplication {
}
<<<
See also this thread on the mailing list:
https://lists.qt-project.org/pipermail/qbs/2019-July/002477.html
and the follow up:
https://lists.qt-project.org/pipermail/qbs/2019-August/002544.html
Kind regards
Jochen Ulrich
Von: Qbs im Auftrag von Карелин Павел
Datum: Dienstag, 30. Juni 2020 um 09:45
An: "qbs@qt-project.org"
Betreff: [Qbs] QBS script parsing speed
Hello.
Now I'm trying to write a QBS build script for a fairly large project (QGIS).
After making every change to the build script - the project tree is rebuilt and
QtCreator freezes for 10-15 seconds.
It seems that 10-15 is not so much, but since I have to make a lot of changes
in the script, in the aggregate the slowdown turns out to be quite significant.
I suppose I was not the only one who faced a similar problem on large projects.
I would like to know if any measures are being planned to increase the speed of
re-parsing a project?
I am using QtCreator 4.11.2 (QBS 1.15.1)
--
BR, Pavel Karelin
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
Jochen, >> Well [the time] goes into "Handling Products". So it is >> ModuleLoader::handleProduct(). >> And there, it could be the ModuleMerger. But speculation is dangerous when >> discussing about performance. > See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html > > You can find my test results together with the scripts to generate the > projects here: > https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Then you might be happy to hear that the ModuleMerger was completely rewritten for Qbs 1.16.0 (https://codereview.qt-project.org/c/qbs/qbs/+/284737). Running your benchmark with Qbs 1.16.0 (Qt Creator 4.12) gives me significantly better results in case of transitive dependencies. See below. I am not sure whether this has any noticable effect on real-world projects. But if ModuleMerger was ever a bottleneck, then it should not longer be. - i5-2520M CPU @ 2.50GHz Xubuntu 16.04) First line: Qbs 1.15.1 Second line: Qbs 1.16.0 > Resolving 4-8-direct real0m3.089s real0m2.669s > Resolving 4-8-implicit real0m4.808s real0m3.409s > Resolving 4-8-explicit real0m3.898s real0m3.085s > Resolving withLeafExports/4-8-direct real0m16.508s real0m13.793s > Resolving withLeafExports/4-8-implicit real0m25.907s real0m18.180s > Resolving withLeafExports/4-8-explicit real0m21.393s real0m15.844s > Resolving withLeafExports/9-2-direct real0m13.992s real0m10.750s > Resolving withLeafExports/9-2-implicit real0m43.592s real0m24.494s > Resolving withLeafExports/9-2-explicit real0m26.019s real0m17.589s > Resolving withAdditionalDepends/4-8-direct real0m25.096s real0m18.892s > Resolving withAdditionalDepends/4-8-implicit real0m28.369s real0m19.868s > Resolving withAdditionalDepends/4-8-explicit real0m29.164s real0m19.319s > Resolving withAdditionalDepends/9-2-direct real0m19.110s real0m17.264s > Resolving withAdditionalDepends/9-2-implicit real0m20.887s real0m19.269s > Resolving withAdditionalDepends/9-2-explicit real0m18.359s real0m18.149s > Resolving 9-2-direct real0m2.500s > Resolving 9-2-implicit real0m6.393s > Resolving 9-2-explicit real0m3.833s real0m2.326s ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
My analysis back then was: > Well [the time] goes into "Handling Products". So it is > ModuleLoader::handleProduct(). > And there, it could be the ModuleMerger. But speculation is dangerous when > discussing about performance. See https://lists.qt-project.org/pipermail/qbs/2019-August/002546.html You can find my test results together with the scripts to generate the projects here: https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11#file-results-md Best Jochen ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
> I would like to know if any measures are being planned to > increase the speed of re-parsing a project? I am using QtCreator 4.11.2 > (QBS 1.15.1) Some ideas: 1. Incremental project resolving: Take the dependency graph into account when re-resolving a project so that unchanged products are not evaluated again. 2. Parallel resolving: Evaluate unrelated products using multiple threads. Both would require a lot of refactoring work in the projectresolver and moduleloader modules. I would really like to do something about it or see somebody else doing something about it. But as a spare time job, this looks undoable. Nevertheless, we can try our best to make small steps forward. Richard ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
> We faced similar problems but our resolve times were even higher (30 s to > 2 mins). > > We were able to work around this and significantly reduce the resolve > times by avoiding transitive dependencies (meaning exporting > dependencies). I would like to understand why exactly this pattern makes the evaluation time explode. ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] QBS script parsing speed
We faced similar problems but our resolve times were even higher (30 s to 2
mins).
We were able to work around this and significantly reduce the resolve times by
avoiding transitive dependencies (meaning exporting dependencies).
So instead of this:
>>>
Product {
name: “A”
Depends { name: "B" }
}
Product {
name: “B”
Depends { name: "C" }
Export {
Depends { name: "C" }
}
}
Product {
name: “C”
Depends { name: "SomeLibNeededEverywhere" }
Export {
Depends { name: "SomeLibNeededEverywhere" }
}
}
<<<
You do it like this
>>>
Product {
name: “A”
Depends { name: "B" }
Depends { name: "C" }
Depends { name: "SomeLibNeededEverywhere" }
}
Product {
name: “B”
Depends { name: "C" }
Depends { name: "SomeLibNeededEverywhere" }
}
Product {
name: “C”
Depends { name: "SomeLibNeededEverywhere" }
}
<<<
Of course it is ugly because you have to maintain the transitive dependencies
manually.
However, if you have cases where a dependency is needed in many places, you can
create custom products that include it.
For example:
>>>
// qbs/imports/MyBaseApplication.qbs
CppApplication {
Depends { name: "SomeLibNeededEverywhere" }
}
// in other .qbs files use MyBaseApplication instead of CppApplication
MyBaseApplication {
}
<<<
See also this thread on the mailing list:
https://lists.qt-project.org/pipermail/qbs/2019-July/002477.html
and the follow up:
https://lists.qt-project.org/pipermail/qbs/2019-August/002544.html
Kind regards
Jochen Ulrich
> Von: Qbs im Auftrag von Карелин Павел
>
> Datum: Dienstag, 30. Juni 2020 um 09:45
> An: "qbs@qt-project.org"
> Betreff: [Qbs] QBS script parsing speed
>
> Hello.
>
> Now I'm trying to write a QBS build script for a fairly large project (QGIS).
> After making every change to the build script - the project tree is rebuilt
> and QtCreator freezes for 10-15 seconds.
> It seems that 10-15 is not so much, but since I have to make a lot of changes
> in the script, in the aggregate the slowdown turns out to be quite
> significant.
> I suppose I was not the only one who faced a similar problem on large
> projects. I would like to know if any measures are being planned to increase
> the speed of re-parsing a project?
> I am using QtCreator 4.11.2 (QBS 1.15.1)
>
>
> --
> BR, Pavel Karelin
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
[Qbs] QBS script parsing speed
Hello. Now I'm trying to write a QBS build script for a fairly large project (QGIS). After making every change to the build script - the project tree is rebuilt and QtCreator freezes for 10-15 seconds. It seems that 10-15 is not so much, but since I have to make a lot of changes in the script, in the aggregate the slowdown turns out to be quite significant. I suppose I was not the only one who faced a similar problem on large projects. I would like to know if any measures are being planned to increase the speed of re-parsing a project? I am using QtCreator 4.11.2 (QBS 1.15.1) -- BR, Pavel Karelin ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
