Re: [Qbs] Improving qbs resolve performance

2019-08-30 Thread Jochen Ulrich
> Is it realistic a depth of 9?

I don't know. In our project, we are facing resolve times of 50s and more. But 
I didn't check our dependency tree depth yet. :P
But anyway, it shows the resolve behavior depending on the tree depth.

>  Looking at your timings, it is still not clear where the bunch of the time 
> is spent for the "withLeafExports/9-2-implicit" use case.

Well it goes into "Handling Products". So it is ModuleLoader::handleProduct().
And there, it could be the ModuleMerger. But speculation is dangerous when 
discussing about performance.

I also did another test: re-resolving. That is: just touch the "root.qbs" and 
then resolve again.
And there is one interesting thing: The "Running Probes" step takes nearly the 
same amount of time:

Initial Resolve:
Handling products took 33s, 333ms.
Running Probes took 18s, 67ms.
5355 probes encountered, 5 configure scripts executed, 5348 re-used 
from current run, 0 re-used from earlier run.
Property checking took 412ms.

Re-Resolve:
Handling products took 32s, 798ms.
Running Probes took 17s, 543ms.
5355 probes encountered, 0 configure scripts executed, 0 re-used from 
current run, 5355 re-used from earlier run.
Property checking took 409ms.

I would have expected that "Running Probes" is way faster in the re-resolve 
case.

Best
Jochen
 

Am 30.08.19, 02:07 schrieb "Christian Gagneraud" :

On Fri, 30 Aug 2019 at 11:01, Jochen Ulrich  
wrote:
> I want to follow up on this topic:

I had to check the archives and re-read the whole thread!

> I created scripts to generate a Qbs project of a specified size.
> Then, I created a few projects and measured the resolve time.

Nice one. Maybe it would be nice to automate the data collection, this
way it might be interesting to plot the various measurement versus
tree depth and/or child count and/or product count. Just to see if we
have something linear or logarithmic.
Or maybe that's a job for a C++ benchmark...

> Bottom line:
> Exporting dependencies significantly slows down the resolving.
> It is better to explicitly depend on the needed modules instead of having 
another module export the dependency.
> It’s a trade-off between having to write more “Depends” items vs. slower 
resolving.

Good to know, But the Export/Depends has definitely it's use, you
don't need to track nested dependencies.

Random thoughts:
Is it realistic a depth of 9?
Looking at your timings, it is still not clear where the bunch of the
time is spent for the "withLeafExports/9-2-implicit" use case.

Thanks,
Chris


___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-08-29 Thread Christian Gagneraud
On Fri, 30 Aug 2019 at 11:01, Jochen Ulrich  wrote:
> I want to follow up on this topic:

I had to check the archives and re-read the whole thread!

> I created scripts to generate a Qbs project of a specified size.
> Then, I created a few projects and measured the resolve time.

Nice one. Maybe it would be nice to automate the data collection, this
way it might be interesting to plot the various measurement versus
tree depth and/or child count and/or product count. Just to see if we
have something linear or logarithmic.
Or maybe that's a job for a C++ benchmark...

> Bottom line:
> Exporting dependencies significantly slows down the resolving.
> It is better to explicitly depend on the needed modules instead of having 
> another module export the dependency.
> It’s a trade-off between having to write more “Depends” items vs. slower 
> resolving.

Good to know, But the Export/Depends has definitely it's use, you
don't need to track nested dependencies.

Random thoughts:
Is it realistic a depth of 9?
Looking at your timings, it is still not clear where the bunch of the
time is spent for the "withLeafExports/9-2-implicit" use case.

Thanks,
Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-08-29 Thread Jochen Ulrich
I want to follow up on this topic:

I created scripts to generate a Qbs project of a specified size.
Then, I created a few projects and measured the resolve time.

Bottom line:
Exporting dependencies significantly slows down the resolving.
It is better to explicitly depend on the needed modules instead of having 
another module export the dependency.
It’s a trade-off between having to write more “Depends” items vs. slower 
resolving.

The scripts and the results can be found here: 
https://gist.github.com/j-ulrich/35f345007809c77403a8ee88f4d9db11

Best
Jochen
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-17 Thread Christian Gagneraud
On Sun, 14 Jul 2019 at 17:30, Christian Gagneraud  wrote:
> Looking at valgrind/callgrind data, it seems that the hot path is a
> loop, here is a simplified tree with the final loop
> - resolveProjectFromScratch()
> - loadProject()
> - handleTopLevelProject()
> - setupProductDependencies()
> - 0 - resolveDependencies()
> - 1 - resolveDependsItem()
> - 2 - loadModule()
> - 3 - instanciateModule()
> - 4 - goto 0

There is indeed a cache system, but it is only enable in very few
location (Using EvalCacheEnabler).
And that's one of the hot spots: Evaluator::property() - 26% of the
fetched instructions while resolving the qbs project.
I need to test with resolving QtCreator.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-15 Thread Christian Kandeler
On Sat, 13 Jul 2019 17:37:04 +1200
Christian Gagneraud  wrote:

> On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
>  wrote:
> >
> > On Fri, 12 Jul 2019 12:05:48 +
> > Maximilian Hrabowski  wrote:
> >  
> > > >> I have a fairly large project with a root projects that pulls in 
> > > >> several SubProjects, altogether with unit tests its about 313 qbs 
> > > >> files that are pulled in.  
> > > >
> > > > The number of project files is not all that important by itself. Qt 
> > > > Creator, for instance, has a comparable number, and resolves much 
> > > > faster than your project.  
> > >
> > > Can you give any guidance what to avoid, i.e. what makes resolving slow 
> > > if QtC is much faster resolved?  
> >
> > Nothing beyond the obvious, such as calling external processes on the 
> > right-hand side of properties (but you should get warnings for that 
> > anyway). Related: If you have probes, be sure not to trigger them 
> > inadvertently. Excessive use of wildcards is also discouraged.
> > This is what comes to mind spontaneously.  
> 
> What about complex inter-dependencies? I tried once to convert a
> legacy project. This was quite big (bigger than QtC) and complex (in
> term of public/private inter-dependencies), and i do remember running
> into "performance" issues.

The number of modules pulled in per product is surely relevant (note that 
product dependencies are modules in this sense).

> What about the cpp and the moc scanner? if you have a lot of
> "moc'able" source files, with complex/crazy include hierarchy, could
> this seriously impact the resolve work?

No, that's done when building and has nothing to do with resolving.

> Pure speculation here, randomly picked:
> I'm not familiar with Qbs internals, but just stumble on the "Opaq"
> struct in cppscanner.cpp, which has a "QList
> includedFiles; " member. Given how lists are not CPU cache friendly,
> could this sort of "minor details" greatly impact Qbs performance when
> trying to resolve "poorly" organized giant projects? With modern CPU,
> cache locality is paramount.

Again, that's a build time thing.

> > (I certainly don't want to to dissuade anyone from trying, it's just that 
> > you need to be prepared for a disappointing outcome. For a much 
> > lower-hanging fruit in the area of performance improvement, take a look at 
> > https://bugreports.qt.io/browse/QBS-1448.)  
> (pure speculation here).
> 
> Is the build graph first built/resolve in-memory, or is it
> built/visited on-disk?
> If the former, then it is unlikely the bottle neck here, if the later,
> that might indeed be a good lead.

That task is about the time is takes to load a stored build graph.


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-13 Thread Christian Gagneraud
On Fri, 12 Jul 2019 at 00:45, Christian Kandeler
 wrote:
>
> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
>
> > I have a fairly large project with a root projects that pulls in several 
> > SubProjects, altogether with unit tests its about 313 qbs files that are 
> > pulled in.
>
> The number of project files is not all that important by itself. Qt Creator, 
> for instance, has a comparable number, and resolves much faster than your 
> project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
> > that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following duration 
> > for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build 
> > graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.

I checked out my old experimental branch and compared resolving qtc
and this project:

QtC:
Qt was set up successfully.
Project file loading and parsing took 161ms.
Preparing products took 1ms.
Setting up product dependencies took 4s, 826ms.
Setting up transitive product dependencies took 930ms.
Handling products took 5s, 160ms.
Running Probes took 2s, 190ms.
1916 probes encountered, 8 configure scripts executed,
1901 re-used from current run, 0 re-used from earlier run.
Property checking took 689ms.
Activity 'ModuleLoader' took 11s, 433ms.

My project:
Qt was set up successfully.
Project file loading and parsing took 100ms.
Preparing products took 0ms.
Setting up product dependencies took 1m, 37s, 54ms.
Setting up transitive product dependencies took 1s, 74ms.
Handling products took 17s, 925ms.
Running Probes took 1s, 799ms.
804 probes encountered, 5 configure scripts executed,
797 re-used from current run, 0 re-used from earlier run.
Property checking took 194ms.
Activity 'ModuleLoader' took 1m, 55s, 496ms.

So the obvious difference is "Setting up product dependencies".

Looking at valgrind/callgrind data, it seems that the hot path is a
loop, here is a simplified tree with the final loop
- resolveProjectFromScratch()
- loadProject()
- handleTopLevelProject()
- setupProductDependencies()
- 0 - resolveDependencies()
- 1 - resolveDependsItem()
- 2 - loadModule()
- 3 - instanciateModule()
- 4 - goto 0

This match with "my gut feeling" expressed previously, projects with
complex dependency tree are penalised.
In this particular case, a cache system would greatly help, or maybe
there is one in place, and the project files do something that
invalidate this cache...

Chris



>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for improvement. 
> > To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build 
> > graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this and 
> > what the right approach would be. Maybe looking at “Depends {}” would be 
> > enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible 
> interdependencies for a statement like "if file x has changed, we know what 
> only product y can be affected". At the very least, much more now-temporary 
> data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
> > project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it 
> thinks that's necessary, i.e. a project file or a property or the environment 
> has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of concurrency 
> when resolving, e.g. have one thread per product. It would not be trivial, 
> though, due to inter-product dependencies etc; also, you'd need a dedicated 
> script engine per thread. I don't remember the details, but whenever I though 
> about this topic, I quickly stopped again after examining what we do in the 
> ModuleLoader, which is by far the most expensive stage of project resolving 
> these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> ___

Re: [Qbs] Improving qbs resolve performance

2019-07-12 Thread Christian Gagneraud
On Sat, 13 Jul 2019 at 00:48, Christian Kandeler
 wrote:
>
> On Fri, 12 Jul 2019 12:05:48 +
> Maximilian Hrabowski  wrote:
>
> > >> I have a fairly large project with a root projects that pulls in several 
> > >> SubProjects, altogether with unit tests its about 313 qbs files that are 
> > >> pulled in.
> > >
> > > The number of project files is not all that important by itself. Qt 
> > > Creator, for instance, has a comparable number, and resolves much faster 
> > > than your project.
> >
> > Can you give any guidance what to avoid, i.e. what makes resolving slow if 
> > QtC is much faster resolved?
>
> Nothing beyond the obvious, such as calling external processes on the 
> right-hand side of properties (but you should get warnings for that anyway). 
> Related: If you have probes, be sure not to trigger them inadvertently. 
> Excessive use of wildcards is also discouraged.
> This is what comes to mind spontaneously.

What about complex inter-dependencies? I tried once to convert a
legacy project. This was quite big (bigger than QtC) and complex (in
term of public/private inter-dependencies), and i do remember running
into "performance" issues.

I do agree that QtC is faster to resolve, but it is quite "clean", and
so facilitate the job of the build sytem in that respect.

Just imagine a chain of dependencies, let say an app has direct
dependency to say 20 static libs, in turn these libs have a "meander"
of inter-dependencies, with say the longest chain being 10 or more
item longs, and the final number of libs to (directly or indirectly)
link is something around 100. On top of that let's add "lots" of
system dependencies (eg, 10+ Qt modules, 10's of lower level libs,
...)

Could that cause serious slow down of the resolving?

The project i'm talking about is so big and complex, that we hit all
the walls. For example we manage to get a 32bits linker run out of
address space, we solved the issue by tweaking/massaging the linker
flags, but still it can take one to 2 minutes just to link the final
executable.
So if ld (bfd or gold) fight with linking such an executable, could it
be that Qbs fight with resolving it's build graph?

What about the cpp and the moc scanner? if you have a lot of
"moc'able" source files, with complex/crazy include hierarchy, could
this seriously impact the resolve work?

Pure speculation here, randomly picked:
I'm not familiar with Qbs internals, but just stumble on the "Opaq"
struct in cppscanner.cpp, which has a "QList
includedFiles; " member. Given how lists are not CPU cache friendly,
could this sort of "minor details" greatly impact Qbs performance when
trying to resolve "poorly" organized giant projects? With modern CPU,
cache locality is paramount.


> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)
(pure speculation here).

Is the build graph first built/resolve in-memory, or is it
built/visited on-disk?
If the former, then it is unlikely the bottle neck here, if the later,
that might indeed be a good lead.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-12 Thread Christian Kandeler
On Fri, 12 Jul 2019 12:05:48 +
Maximilian Hrabowski  wrote:

> >> I have a fairly large project with a root projects that pulls in several 
> >> SubProjects, altogether with unit tests its about 313 qbs files that are 
> >> pulled in.  
> > 
> > The number of project files is not all that important by itself. Qt 
> > Creator, for instance, has a comparable number, and resolves much faster 
> > than your project.  
> 
> Can you give any guidance what to avoid, i.e. what makes resolving slow if 
> QtC is much faster resolved?

Nothing beyond the obvious, such as calling external processes on the 
right-hand side of properties (but you should get warnings for that anyway). 
Related: If you have probes, be sure not to trigger them inadvertantly. 
Excessive use of wildcards is also discouraged.
This is what comes to mind spontaneously.


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-12 Thread Maximilian Hrabowski
On 11. Jul 2019, at 14:45, Christian Kandeler  wrote:
> 
> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
> 
>> I have a fairly large project with a root projects that pulls in several 
>> SubProjects, altogether with unit tests its about 313 qbs files that are 
>> pulled in.
> 
> The number of project files is not all that important by itself. Qt Creator, 
> for instance, has a comparable number, and resolves much faster than your 
> project.

Can you give any guidance what to avoid, i.e. what makes resolving slow if QtC 
is much faster resolved?

> 
>> Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
>> that many other subprojects depend on.
>> 
>> With a single profile and single config I encounter the following duration 
>> for qbs resolve on my macbook pro with 12 logical CPUs:
>> 
>> 1) qbs resolve (clean, new build directory, so includes creating a build 
>> graph):  ~1m 25s
> 
> qbs --log-time might give some hints as to where that time is spent.

Adding —log-time revealed that indeed the ModuleLoader uses up nearly about 70% 
of the total time. 
> 
>> 2) qbs resolve (no changes): ~0.5 s
>> 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
>> 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
>> 
>> From the durations i would expect that there is some room for improvement. 
>> To me it seems (by looking at 3 and 4) that qbs rebuilds the whole build 
>> graph if it detects any change to any qbs file in the project. 
> 
> Yes, there is no partial re-resolve.
> 
>> If this is the case i wonder how difficult it would be to improve this and 
>> what the right approach would be. Maybe looking at “Depends {}” would be 
>> enough to determine the “dirty path”.
> 
> I suspect this is hopeless. In general, there are too many possible 
> interdependencies for a statement like "if file x has changed, we know what 
> only product y can be affected". At the very least, much more now-temporary 
> data from the resolve stage would have to be stored for such logic to work.
> 
>> Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
>> project will no longer cause a full resolve.
> 
> I don't think anything has changed there recently. qbs re-resolved when it 
> thinks that's necessary, i.e. a project file or a property or the environment 
> has changed.

Well, before some event (I consider QtC update to 4.9.x as this event) opening 
the root project always resulted in about 2 minutes wait time. since then 
opening is always quite fast (~5sec). 

> 
>> Any ideas or maybe even someone looking at that already?
> 
> The only remote possibility that I see is somehow making use of concurrency 
> when resolving, e.g. have one thread per product. It would not be trivial, 
> though, due to inter-product dependencies etc; also, you'd need a dedicated 
> script engine per thread. I don't remember the details, but whenever I though 
> about this topic, I quickly stopped again after examining what we do in the 
> ModuleLoader, which is by far the most expensive stage of project resolving 
> these days, and thus the only one worth optimizing.

Indeed, a quick profiling using Instruments does not show any obvious culprit 
to optimize in ModuleLoader but show 70% of time goes there. 

> (I certainly don't want to to dissuade anyone from trying, it's just that you 
> need to be prepared for a disappointing outcome. For a much lower-hanging 
> fruit in the area of performance improvement, take a look at 
> https://bugreports.qt.io/browse/QBS-1448.)


___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-11 Thread Vincent Hui
Perhaps QThreadPooler
 can
help accelerating building products with inter-product dependencies.


On Thu, 11 Jul 2019 at 20:46, Christian Kandeler 
wrote:

> On Thu, 11 Jul 2019 12:18:30 +
> Maximilian Hrabowski  wrote:
>
> > I have a fairly large project with a root projects that pulls in several
> SubProjects, altogether with unit tests its about 313 qbs files that are
> pulled in.
>
> The number of project files is not all that important by itself. Qt
> Creator, for instance, has a comparable number, and resolves much faster
> than your project.
>
> > Let root.qbs be the root qbs file of my project and depend.qbs a
> subproject that many other subprojects depend on.
> >
> > With a single profile and single config I encounter the following
> duration for qbs resolve on my macbook pro with 12 logical CPUs:
> >
> > 1) qbs resolve (clean, new build directory, so includes creating a build
> graph):  ~1m 25s
>
> qbs --log-time might give some hints as to where that time is spent.
>
> > 2) qbs resolve (no changes): ~0.5 s
> > 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> > 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> >
> > From the durations i would expect that there is some room for
> improvement. To me it seems (by looking at 3 and 4) that qbs rebuilds the
> whole build graph if it detects any change to any qbs file in the project.
>
> Yes, there is no partial re-resolve.
>
> > If this is the case i wonder how difficult it would be to improve this
> and what the right approach would be. Maybe looking at “Depends {}” would
> be enough to determine the “dirty path”.
>
> I suspect this is hopeless. In general, there are too many possible
> interdependencies for a statement like "if file x has changed, we know what
> only product y can be affected". At the very least, much more now-temporary
> data from the resolve stage would have to be stored for such logic to work.
>
> > Fortunately QtC 4.9.x seems to cache something now since opening a qbs
> project will no longer cause a full resolve.
>
> I don't think anything has changed there recently. qbs re-resolved when it
> thinks that's necessary, i.e. a project file or a property or the
> environment has changed.
>
> > Any ideas or maybe even someone looking at that already?
>
> The only remote possibility that I see is somehow making use of
> concurrency when resolving, e.g. have one thread per product. It would not
> be trivial, though, due to inter-product dependencies etc; also, you'd need
> a dedicated script engine per thread. I don't remember the details, but
> whenever I though about this topic, I quickly stopped again after examining
> what we do in the ModuleLoader, which is by far the most expensive stage of
> project resolving these days, and thus the only one worth optimizing.
> (I certainly don't want to to dissuade anyone from trying, it's just that
> you need to be prepared for a disappointing outcome. For a much
> lower-hanging fruit in the area of performance improvement, take a look at
> https://bugreports.qt.io/browse/QBS-1448.)
>
>
> Christian
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


Re: [Qbs] Improving qbs resolve performance

2019-07-11 Thread Christian Kandeler
On Thu, 11 Jul 2019 12:18:30 +
Maximilian Hrabowski  wrote:

> I have a fairly large project with a root projects that pulls in several 
> SubProjects, altogether with unit tests its about 313 qbs files that are 
> pulled in.

The number of project files is not all that important by itself. Qt Creator, 
for instance, has a comparable number, and resolves much faster than your 
project.

> Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
> that many other subprojects depend on.
> 
> With a single profile and single config I encounter the following duration 
> for qbs resolve on my macbook pro with 12 logical CPUs:
> 
> 1) qbs resolve (clean, new build directory, so includes creating a build 
> graph):  ~1m 25s

qbs --log-time might give some hints as to where that time is spent.

> 2) qbs resolve (no changes): ~0.5 s
> 3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
> 4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s
> 
> From the durations i would expect that there is some room for improvement. To 
> me it seems (by looking at 3 and 4) that qbs rebuilds the whole build graph 
> if it detects any change to any qbs file in the project. 

Yes, there is no partial re-resolve.

> If this is the case i wonder how difficult it would be to improve this and 
> what the right approach would be. Maybe looking at “Depends {}” would be 
> enough to determine the “dirty path”.

I suspect this is hopeless. In general, there are too many possible 
interdependencies for a statement like "if file x has changed, we know what 
only product y can be affected". At the very least, much more now-temporary 
data from the resolve stage would have to be stored for such logic to work.

> Fortunately QtC 4.9.x seems to cache something now since opening a qbs 
> project will no longer cause a full resolve.

I don't think anything has changed there recently. qbs re-resolved when it 
thinks that's necessary, i.e. a project file or a property or the environment 
has changed.

> Any ideas or maybe even someone looking at that already?

The only remote possibility that I see is somehow making use of concurrency 
when resolving, e.g. have one thread per product. It would not be trivial, 
though, due to inter-product dependencies etc; also, you'd need a dedicated 
script engine per thread. I don't remember the details, but whenever I though 
about this topic, I quickly stopped again after examining what we do in the 
ModuleLoader, which is by far the most expensive stage of project resolving 
these days, and thus the only one worth optimizing.
(I certainly don't want to to dissuade anyone from trying, it's just that you 
need to be prepared for a disappointing outcome. For a much lower-hanging fruit 
in the area of performance improvement, take a look at 
https://bugreports.qt.io/browse/QBS-1448.)


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs


[Qbs] Improving qbs resolve performance

2019-07-11 Thread Maximilian Hrabowski
Hi all,

I have a fairly large project with a root projects that pulls in several 
SubProjects, altogether with unit tests its about 313 qbs files that are pulled 
in.

Let root.qbs be the root qbs file of my project and depend.qbs a subproject 
that many other subprojects depend on.

With a single profile and single config I encounter the following duration for 
qbs resolve on my macbook pro with 12 logical CPUs:

1) qbs resolve (clean, new build directory, so includes creating a build 
graph):  ~1m 25s
2) qbs resolve (no changes): ~0.5 s
3) qbs resolve after "touch root.qbs" (restores build graph): ~1m 15 s
4) qbs resolve after “touch depend.qbs” (restores build graph): ~1m 15s

From the durations i would expect that there is some room for improvement. To 
me it seems (by looking at 3 and 4) that qbs rebuilds the whole build graph if 
it detects any change to any qbs file in the project. If this is the case i 
wonder how difficult it would be to improve this and what the right approach 
would be. Maybe looking at “Depends {}” would be enough to determine the “dirty 
path”.

Fortunately QtC 4.9.x seems to cache something now since opening a qbs project 
will no longer cause a full resolve.

Any ideas or maybe even someone looking at that already?

Best regards,
Maximilian


___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs