Re: [Qbs] Improving qbs resolve performance

[Jochen Ulrich]

I want to follow up on this topic:

I created scripts to generate a Qbs project of a specified size.
Then, I created a few projects and measured the resolve time.

Bottom line:
Exporting dependencies significantly slows down the resolving.
It is better to explicitly depend on the needed modules instead of having 
another module export the dependency.
It’s a trade-off between having to write more “Depends” items vs. slower 
resolving.

The scripts and the results can be found here: 
https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11

Best
Jochen
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Christian Gagneraud]

On Sun, 14 Jul 2019 at 17:30, Christian Gagneraud  wrote:
> Looking at valgrind/callgrind data, it seems that the hot path is a
> loop, here is a simplified tree with the final loop
> - resolveProjectFromScratch()
> - loadProject()
> - handleTopLevelProject()
> - setupProductDependencies()
> - 0 - resolveDependencies()
> - 1 - resolveDependsItem()
> - 2 - loadModule()
> - 3 - instanciateModule()
> - 4 - goto 0

There is indeed a cache system, but it is only enable in very few
location (Using EvalCacheEnabler).
And that's one of the hot spots: Evaluator::property() - 26% of the
fetched instructions while resolving the qbs project.
I need to test with resolving QtCreator.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Christian Kandeler]

On Sat, 13 Jul 2019 17:37:04 +1200
Christian Gagneraud  wrote:

> On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
>  wrote:
> >
> > On Fri, 12 Jul 2019 12:05:48 +
> > Maximilian Hrabowski  wrote:
> >  
> > > >> I have a fairly large project with a root projects that pulls in 
> > > >> several SubProjects, altogether with unit tests its about 313 qbs 
> > > >> files that are pulled in.  
> > > >
> > > > The number of project files is not all that important by itself. Qt 
> > > > Creator, for instance, has a comparable number, and resolves much 
> > > > faster than your project.  
> > >
> > > Can you give any guidance what to avoid, i.e. what makes resolving slow 
> > > if QtC is much faster resolved?  
> >
> > Nothing beyond the obvious, such as calling external processes on the 
> > right-hand side of properties (but you should get warnings for that 
> > anyway). Related: If you have probes, be sure not to trigger them 
> > inadvertently. Excessive use of wildcards is also discouraged.
> > This is what comes to mind spontaneously.  
> 
> What about complex inter-dependencies? I tried once to convert a
> legacy project. This was quite big (bigger than QtC) and complex (in
> term of public/private inter-dependencies), and i do remember running
> into "performance" issues.

The number of modules pulled in per product is surely relevant (note that 
product dependencies are modules in this sense).

> What about the cpp and the moc scanner? if you have a lot of
> "moc'able" source files, with complex/crazy include hierarchy, could
> this seriously impact the resolve work?

No, that's done when building and has nothing to do with resolving.

> Pure speculation here, randomly picked:
> I'm not familiar with Qbs internals, but just stumble on the "Opaq"
> struct in cppscanner.cpp, which has a "QList
> includedFiles; " member. Given how lists are not CPU cache friendly,
> could this sort of "minor details" greatly impact Qbs performance when
> trying to resolve "poorly" organized giant projects? With modern CPU,
> cache locality is paramount.

Again, that's a build time thing.

> > (I certainly don't want to to dissuade anyone from trying, it's just that 
> > you need to be prepared for a disappointing outcome. For a much 
> > lower-hanging fruit in the area of performance improvement, take a look at 
> > https://bugreports.qt.io/browse/QBS-1448.)  
> (pure speculation here).
> 
> Is the build graph first built/resolve in-memory, or is it
> built/visited on-disk?
> If the former, then it is unlikely the bottle neck here, if the later,
> that might indeed be a good lead.

That task is about the time is takes to load a stored build graph.


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Christian Gagneraud]

On Fri, 12 Jul 2019 at 00:45, Christian Kandeler
 wrote:
>
> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
>
> > I have a fairly large project with a root projects that pulls in several 
> > SubProjects, altogether with unit tests its about 313 qbs files that are 
> > pulled in.
>
> The number of project files is not all that important by itself. Qt Creator, 
> for instance, has a comparable number, and resolves much faster than your 
> project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
> > that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following duration 
> > for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build 
> > graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.

I checked out my old experimental branch and compared resolving qtc
and this project:

QtC:
Qt was set up successfully.
Project file loading and parsing took 161ms.
Preparing products took 1ms.
Setting up product dependencies took 4s, 826ms.
Setting up transitive product dependencies took 930ms.
Handling products took 5s, 160ms.
Running Probes took 2s, 190ms.
1916 probes encountered, 8 configure scripts executed,
1901 re-used from current run, 0 re-used from earlier run.
Property checking took 689ms.
Activity 'ModuleLoader' took 11s, 433ms.

My project:
Qt was set up successfully.
Project file loading and parsing took 100ms.
Preparing products took 0ms.
Setting up product dependencies took 1m, 37s, 54ms.
Setting up transitive product dependencies took 1s, 74ms.
Handling products took 17s, 925ms.
Running Probes took 1s, 799ms.
804 probes encountered, 5 configure scripts executed,
797 re-used from current run, 0 re-used from earlier run.
Property checking took 194ms.
Activity 'ModuleLoader' took 1m, 55s, 496ms.

So the obvious difference is "Setting up product dependencies".

Looking at valgrind/callgrind data, it seems that the hot path is a
loop, here is a simplified tree with the final loop
- resolveProjectFromScratch()
- loadProject()
- handleTopLevelProject()
- setupProductDependencies()
- 0 - resolveDependencies()
- 1 - resolveDependsItem()
- 2 - loadModule()
- 3 - instanciateModule()
- 4 - goto 0

This match with "my gut feeling" expressed previously, projects with
complex dependency tree are penalised.
In this particular case, a cache system would greatly help, or maybe
there is one in place, and the project files do something that
invalidate this cache...

Chris



>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for improvement. 
> > To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build 
> > graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this and 
> > what the right approach would be. Maybe looking at “Depends {}” would be 
> > enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible 
> interdependencies for a statement like "if file x has changed, we know what 
> only product y can be affected". At the very least, much more now-temporary 
> data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
> > project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it 
> thinks that's necessary, i.e. a project file or a property or the environment 
> has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of concurrency 
> when resolving, e.g. have one thread per product. It would not be trivial, 
> though, due to inter-product dependencies etc; also, you'd need a dedicated 
> script engine per thread. I don't remember the details, but whenever I though 
> about this topic, I quickly stopped again after examining what we do in the 
> ModuleLoader, which is by far the most expensive stage of project resolving 
> these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> ___

Re: [Qbs] Improving qbs resolve performance

[Christian Gagneraud]

On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
 wrote:
>
> On Fri, 12 Jul 2019 12:05:48 +
> Maximilian Hrabowski  wrote:
>
> > >> I have a fairly large project with a root projects that pulls in several 
> > >> SubProjects, altogether with unit tests its about 313 qbs files that are 
> > >> pulled in.
> > >
> > > The number of project files is not all that important by itself. Qt 
> > > Creator, for instance, has a comparable number, and resolves much faster 
> > > than your project.
> >
> > Can you give any guidance what to avoid, i.e. what makes resolving slow if 
> > QtC is much faster resolved?
>
> Nothing beyond the obvious, such as calling external processes on the 
> right-hand side of properties (but you should get warnings for that anyway). 
> Related: If you have probes, be sure not to trigger them inadvertently. 
> Excessive use of wildcards is also discouraged.
> This is what comes to mind spontaneously.

What about complex inter-dependencies? I tried once to convert a
legacy project. This was quite big (bigger than QtC) and complex (in
term of public/private inter-dependencies), and i do remember running
into "performance" issues.

I do agree that QtC is faster to resolve, but it is quite "clean", and
so facilitate the job of the build sytem in that respect.

Just imagine a chain of dependencies, let say an app has direct
dependency to say 20 static libs, in turn these libs have a "meander"
of inter-dependencies, with say the longest chain being 10 or more
item longs, and the final number of libs to (directly or indirectly)
link is something around 100. On top of that let's add "lots" of
system dependencies (eg, 10+ Qt modules, 10's of lower level libs,
...)

Could that cause serious slow down of the resolving?

The project i'm talking about is so big and complex, that we hit all
the walls. For example we manage to get a 32bits linker run out of
address space, we solved the issue by tweaking/massaging the linker
flags, but still it can take one to 2 minutes just to link the final
executable.
So if ld (bfd or gold) fight with linking such an executable, could it
be that Qbs fight with resolving it's build graph?

What about the cpp and the moc scanner? if you have a lot of
"moc'able" source files, with complex/crazy include hierarchy, could
this seriously impact the resolve work?

Pure speculation here, randomly picked:
I'm not familiar with Qbs internals, but just stumble on the "Opaq"
struct in cppscanner.cpp, which has a "QList
includedFiles; " member. Given how lists are not CPU cache friendly,
could this sort of "minor details" greatly impact Qbs performance when
trying to resolve "poorly" organized giant projects? With modern CPU,
cache locality is paramount.


> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)
(pure speculation here).

Is the build graph first built/resolve in-memory, or is it
built/visited on-disk?
If the former, then it is unlikely the bottle neck here, if the later,
that might indeed be a good lead.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Christian Kandeler]

On Fri, 12 Jul 2019 12:05:48 +
Maximilian Hrabowski  wrote:

> >> I have a fairly large project with a root projects that pulls in several 
> >> SubProjects, altogether with unit tests its about 313 qbs files that are 
> >> pulled in.  
> > 
> > The number of project files is not all that important by itself. Qt 
> > Creator, for instance, has a comparable number, and resolves much faster 
> > than your project.  
> 
> Can you give any guidance what to avoid, i.e. what makes resolving slow if 
> QtC is much faster resolved?

Nothing beyond the obvious, such as calling external processes on the 
right-hand side of properties (but you should get warnings for that anyway). 
Related: If you have probes, be sure not to trigger them inadvertantly. 
Excessive use of wildcards is also discouraged.
This is what comes to mind spontaneously.


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Maximilian Hrabowski]

On 11. Jul 2019, at 14:45, Christian Kandeler  wrote:
> 
> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
> 
>> I have a fairly large project with a root projects that pulls in several 
>> SubProjects, altogether with unit tests its about 313 qbs files that are 
>> pulled in.
> 
> The number of project files is not all that important by itself. Qt Creator, 
> for instance, has a comparable number, and resolves much faster than your 
> project.

Can you give any guidance what to avoid, i.e. what makes resolving slow if QtC 
is much faster resolved?

> 
>> Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
>> that many other subprojects depend on.
>> 
>> With a single profile and single config I encounter the following duration 
>> for qbs resolve on my macbook pro with 12 logical CPUs:
>> 
>> 1) qbs resolve (clean, new build directory, so includes creating a build 
>> graph):  ~1m 25s
> 
> qbs --log-time might give some hints as to where that time is spent.

Adding —log-time revealed that indeed the ModuleLoader uses up nearly about 70% 
of the total time. 
> 
>> 2) qbs resolve (no changes): ~0.5 s
>> 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
>> 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
>> 
>> From the durations i would expect that there is some room for improvement. 
>> To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build 
>> graph if it detects any change to any qbs file in the project. 
> 
> Yes, there is no partial re-resolve.
> 
>> If this is the case i wonder how difficult it would be to improve this and 
>> what the right approach would be. Maybe looking at “Depends {}” would be 
>> enough to determine the “dirty path”.
> 
> I suspect this is hopeless. In general, there are too many possible 
> interdependencies for a statement like "if file x has changed, we know what 
> only product y can be affected". At the very least, much more now-temporary 
> data from the resolve stage would have to be stored for such logic to work.
> 
>> Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
>> project will no longer cause a full resolve.
> 
> I don't think anything has changed there recently. qbs re-resolved when it 
> thinks that's necessary, i.e. a project file or a property or the environment 
> has changed.

Well, before some event (I consider QtC update to 4.9.x as this event) opening 
the root project always resulted in about 2 minutes wait time. since then 
opening is always quite fast (~5sec). 

> 
>> Any ideas or maybe even someone looking at that already?
> 
> The only remote possibility that I see is somehow making use of concurrency 
> when resolving, e.g. have one thread per product. It would not be trivial, 
> though, due to inter-product dependencies etc; also, you'd need a dedicated 
> script engine per thread. I don't remember the details, but whenever I though 
> about this topic, I quickly stopped again after examining what we do in the 
> ModuleLoader, which is by far the most expensive stage of project resolving 
> these days, and thus the only one worth optimizing.

Indeed, a quick profiling using Instruments does not show any obvious culprit 
to optimize in ModuleLoader but show 70% of time goes there. 

> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)


___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Vincent Hui]

Perhaps QThreadPooler
 can
help accelerating building products with inter-product dependencies.


On Thu, 11 Jul 2019 at 20:46, Christian Kandeler 
wrote:

> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
>
> > I have a fairly large project with a root projects that pulls in several
> SubProjects, altogether with unit tests its about 313 qbs files that are
> pulled in.
>
> The number of project files is not all that important by itself. Qt
> Creator, for instance, has a comparable number, and resolves much faster
> than your project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a
> subproject that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following
> duration for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build
> graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.
>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for
> improvement. To me it seems (by looking at 3 and 4) that qbs rebuilds the
> whole build graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this
> and what the right approach would be. Maybe looking at “Depends {}” would
> be enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible
> interdependencies for a statement like "if file x has changed, we know what
> only product y can be affected". At the very least, much more now-temporary
> data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs
> project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it
> thinks that's necessary, i.e. a project file or a property or the
> environment has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of
> concurrency when resolving, e.g. have one thread per product. It would not
> be trivial, though, due to inter-product dependencies etc; also, you'd need
> a dedicated script engine per thread. I don't remember the details, but
> whenever I though about this topic, I quickly stopped again after examining
> what we do in the ModuleLoader, which is by far the most expensive stage of
> project resolving these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that
> you need to be prepared for a disappointing outcome. For a much
> lower-hanging fruit in the area of performance improvement, take a look at
> https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Improving qbs resolve performance

[Christian Kandeler]

On Thu, 11 Jul 2019 12:18:30 +
Maximilian Hrabowski  wrote:

> I have a fairly large project with a root projects that pulls in several 
> SubProjects, altogether with unit tests its about 313 qbs files that are 
> pulled in.

The number of project files is not all that important by itself. Qt Creator, 
for instance, has a comparable number, and resolves much faster than your 
project.

> Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
> that many other subprojects depend on.
> 
> With a single profile and single config I encounter the following duration 
> for qbs resolve on my macbook pro with 12 logical CPUs:
> 
> 1) qbs resolve (clean, new build directory, so includes creating a build 
> graph):  ~1m 25s

qbs --log-time might give some hints as to where that time is spent.

> 2) qbs resolve (no changes): ~0.5 s
> 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> 
> From the durations i would expect that there is some room for improvement. To 
> me it seems (by looking at 3 and 4) that qbs rebuilds the whole build graph 
> if it detects any change to any qbs file in the project. 

Yes, there is no partial re-resolve.

> If this is the case i wonder how difficult it would be to improve this and 
> what the right approach would be. Maybe looking at “Depends {}” would be 
> enough to determine the “dirty path”.

I suspect this is hopeless. In general, there are too many possible 
interdependencies for a statement like "if file x has changed, we know what 
only product y can be affected". At the very least, much more now-temporary 
data from the resolve stage would have to be stored for such logic to work.

> Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
> project will no longer cause a full resolve.

I don't think anything has changed there recently. qbs re-resolved when it 
thinks that's necessary, i.e. a project file or a property or the environment 
has changed.

> Any ideas or maybe even someone looking at that already?

The only remote possibility that I see is somehow making use of concurrency 
when resolving, e.g. have one thread per product. It would not be trivial, 
though, due to inter-product dependencies etc; also, you'd need a dedicated 
script engine per thread. I don't remember the details, but whenever I though 
about this topic, I quickly stopped again after examining what we do in the 
ModuleLoader, which is by far the most expensive stage of project resolving 
these days, and thus the only one worth optimizing.
(I certainly don't want to to dissuade anyone from trying, it's just that you 
need to be prepared for a disappointing outcome. For a much lower-hanging fruit 
in the area of performance improvement, take a look at 
https://bugreports.qt.io/browse/QBS-1448.)


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs