Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Epting, Thomas]

> I didn’t do any measurements, but quick search for QList in Qbs
> source code shows a lot of places  where it’s used incorrectly –
 > it stores «big» structures, std::shared_pointers, even all
 > QSharedDataPointer classes are not marked as Q_MOVABLE_TYPE
 > and thus result in extra allocations in QLists…
>
 > My point is - current usage of Qt containers in Qbs should be
> carefully revisited, switching to more suitable containers
> (QVector or std::vector/std::deque).

I think we should know the performance hot spots first, before going to spend 
effort on doing broad code changes where it’s not affecting performance at all 
(which is typically true for 95% of the code). We should use tools like VTune 
to identify these hot spots and possibly segregate them, then focus on 
optimizing the sensible parts only.

Thomas

Von: Qbs  Im Auftrag von  ??
Gesendet: Donnerstag, 16. Mai 2019 01:25
An: Christian Gagneraud 
Cc: qbs 
Betreff: Re: [Qbs] Donation to QBS developers/maintainers/contributes

In many cases, you don’t need to copy them. For trivial getters, you can return 
const-ref/span to the internal vector instead of a copy because in many places 
all we do is iterate over that vector.

The only argument for copies is that it’s impossible to change the 
implementation from returning a member to some «transformed member» without 
breaking BC. Well, so far Qbs didn’t do any promises about BC so I don’t think 
it’s relevant.

This will save us from creating a temp variables or usages of qAsCont to avoid 
detaching in range-for loops (see this commit 
https://codereview.qt-project.org/#/c/253792/ to estimate the amount of 
incorrect usages of Qt containers). Note, that those clutter API as well, 
especially the need of a temp variable.

I didn’t do any measurements, but quick search for QList in Qbs source code 
shows a lot of places  where it’s used incorrectly - it stores «big» 
structures, std::shared_pointers, even all QSharedDataPointer classes are not 
marked as Q_MOVABLE_TYPE and thus result in extra allocations in QLists…

My point is - current usage of Qt containers in Qbs should be carefully 
revisited, switching to more suitable containers (QVector or 
std::vector/std::deque).



16 мая 2019 г., в 0:00, Christian Gagneraud 
mailto:chg...@gmail.com>> написал(а):

On Thu, 16 May 2019 at 09:32, Иван Комиссаров 
mailto:abba...@gmail.com>> wrote:

15 мая 2019 г., в 19:50, André Pönitz 
mailto:apoen...@t-online.de>> написал(а):

Getting rid of implicitly shared containers should be a rather obvious move
when "performance" is part of any "final picture", coherent or not.

+1 from me, hidden detaches are evil =)

How do you avoid copy then? Without cluttering an API.

Chris

___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Christian Gagneraud]

On Thu, 16 May 2019 at 11:24, Иван Комиссаров  wrote:
>
> In many cases, you don’t need to copy them. For trivial getters, you can 
> return const-ref/span to the internal vector instead of a copy because in 
> many places all we do is iterate over that vector.
>
> The only argument for copies is that it’s impossible to change the 
> implementation from returning a member to some «transformed member» without 
> breaking BC. Well, so far Qbs didn’t do any promises about BC so I don’t 
> think it’s relevant.
>
> This will save us from creating a temp variables or usages of qAsCont to 
> avoid detaching in range-for loops (see this commit 
> https://codereview.qt-project.org/#/c/253792/ to estimate the amount of 
> incorrect usages of Qt containers). Note, that those clutter API as well, 
> especially the need of a temp variable.
>
> I didn’t do any measurements, but quick search for QList in Qbs source code 
> shows a lot of places  where it’s used incorrectly - it stores «big» 
> structures, std::shared_pointers, even all QSharedDataPointer classes are not 
> marked as Q_MOVABLE_TYPE and thus result in extra allocations in QLists…

can QList -> QVector be automated? Or does it has to be done on a case by case?
I have a python script to do "mass but targeted" clang/clazy auto-fixit.

There is 
https://github.com/KDE/clazy/blob/master/docs/checks/README-inefficient-qlist.md
But it doesn't offer a fixit :(

Concerning the QSharedDataPointer classes are not marked as
Q_MOVABLE_TYPE, could clazy detect/fix these as well?

> My point is - current usage of Qt containers in Qbs should be carefully 
> revisited, switching to more suitable containers (QVector or 
> std::vector/std::deque).

Moving to QVector seems easier, at least as a first test. But we would
need performace regression testing to see if it really speed up
things.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Иван Комиссаров]

In many cases, you don’t need to copy them. For trivial getters, you can return 
const-ref/span to the internal vector instead of a copy because in many places 
all we do is iterate over that vector.

The only argument for copies is that it’s impossible to change the 
implementation from returning a member to some «transformed member» without 
breaking BC. Well, so far Qbs didn’t do any promises about BC so I don’t think 
it’s relevant.

This will save us from creating a temp variables or usages of qAsCont to avoid 
detaching in range-for loops (see this commit 
https://codereview.qt-project.org/#/c/253792/ 
 to estimate the amount of 
incorrect usages of Qt containers). Note, that those clutter API as well, 
especially the need of a temp variable.

I didn’t do any measurements, but quick search for QList in Qbs source code 
shows a lot of places  where it’s used incorrectly - it stores «big» 
structures, std::shared_pointers, even all QSharedDataPointer classes are not 
marked as Q_MOVABLE_TYPE and thus result in extra allocations in QLists…

My point is - current usage of Qt containers in Qbs should be carefully 
revisited, switching to more suitable containers (QVector or 
std::vector/std::deque).


> 16 мая 2019 г., в 0:00, Christian Gagneraud  написал(а):
> 
> On Thu, 16 May 2019 at 09:32, Иван Комиссаров  wrote:
>>> 15 мая 2019 г., в 19:50, André Pönitz  написал(а):
>>> 
>>> Getting rid of implicitly shared containers should be a rather obvious move
>>> when "performance" is part of any "final picture", coherent or not.
>> 
>> +1 from me, hidden detaches are evil =)
> 
> How do you avoid copy then? Without cluttering an API.
> 
> Chris

___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Christian Gagneraud]

On Thu, 16 May 2019 at 09:32, Иван Комиссаров  wrote:
> > 15 мая 2019 г., в 19:50, André Pönitz  написал(а):
> >
> > Getting rid of implicitly shared containers should be a rather obvious move
> > when "performance" is part of any "final picture", coherent or not.
>
> +1 from me, hidden detaches are evil =)

How do you avoid copy then? Without cluttering an API.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Иван Комиссаров]

+1 from me, hidden detaches are evil =)

> 15 мая 2019 г., в 19:50, André Pönitz  написал(а):
> 
> Getting rid of implicitly shared containers should be a rather obvious move
> when "performance" is part of any "final picture", coherent or not.
> 
> Andre'
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs

___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[André Pönitz]

On Wed, May 15, 2019 at 11:38:25AM +0200, Oswald Buddenhagen wrote:
> [...]
> jake had started that "qt-free qbs" project (he got as far as
> eliminating (most) use of qt containers), but it was snuffed out because
> it was unrealistic and counterproductive at that time. even now, no
> patches in that direction will be accepted

Won't they?

> unless a coherent final
> picture and a credible committment to further maintenance is presented.

Getting rid of implicitly shared containers should be a rather obvious move
when "performance" is part of any "final picture", coherent or not.

Andre'
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Иван Комиссаров]

Interesting. Any idea why Qbs is so slow with ccache?

Иван Комиссаров

> 15 мая 2019 г., в 17:29, Orgad Shaneh  написал(а):
> 
> du -shc build-qtc-qbs
> 15G build-qtc-qbs
> 15G total
> du -shc build-qtc-cmake
> 4.0Gbuild-qtc-cmake
> 4.0Gtotal
> 
> The number of jobs is default. For qbs it's 40, ninja uses 42.
> 
> I tried again without autotests, as Christian suggested:
> 
> time qbs -f ../qt-creator/qtcreator.qbs profile:qt-5-11-1 
> modules.cpp.compilerWrapper:ccache project.withAutotests:false
> cold ccache:
> real4m58.383s
> user114m45.623s
> sys 15m38.866s
>  
> warm cache:
> real1m20.800s
> user16m20.307s
> sys 3m57.194s
>  
> time cmake ../qt-creator -DCMAKE_CXX_COMPILER_LAUNCHER=ccache 
> -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Debug -G Ninja
> real0m8.256s
> user0m7.425s
> sys 0m0.827s
>  
> time ninja
> cold cache:
> real5m23.413s
> user128m25.457s
> sys 16m11.551s
>  
> time ninja:
> warm cache:
> real0m45.964s
> user7m12.563s
> sys 2m23.616s
> 
> 
>> On Wed, May 15, 2019 at 5:50 PM Vincent Hui  wrote:
>> How many number of threads were used? Is the number of threads used by Qbs 
>> equal to the number of threads used by CMake? 
>> 
>> 
>>> On Wed, 15 May 2019 at 22:11, Christian Kandeler  
>>> wrote:
>>> On Thu, 16 May 2019 02:01:21 +1200
>>> Christian Gagneraud  wrote:
>>> 
>>> > As well, please note that the current CMake doesn't build as much
>>> > stuff as the qbs one. Not sure the difference is huge, but you need to
>>> > compare "equal jobs".
>>> 
>>> For instance, all autotests are missing from the cmake build, so this
>>> alone accounts for a difference of ~100 products. You need to set
>>> "project.withAutotests:false" when building with qbs.
>>> 
>>> 
>>> Christian
>>> ___
>>> Qbs mailing list
>>> Qbs@qt-project.org
>>> https://lists.qt-project.org/listinfo/qbs
>> ___
>> Qbs mailing list
>> Qbs@qt-project.org
>> https://lists.qt-project.org/listinfo/qbs
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Orgad Shaneh]

du -shc build-qtc-qbs
15G build-qtc-qbs
15G total
du -shc build-qtc-cmake
4.0Gbuild-qtc-cmake
4.0Gtotal

The number of jobs is default. For qbs it's 40, ninja uses 42.

I tried again without autotests, as Christian suggested:

time qbs -f ../qt-creator/qtcreator.qbs profile:qt-5-11-1
modules.cpp.compilerWrapper:ccache project.withAutotests:false
cold ccache:
real4m58.383s
user114m45.623s
sys 15m38.866s

warm cache:
real1m20.800s
user16m20.307s
sys 3m57.194s

time cmake ../qt-creator -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Debug -G Ninja
real0m8.256s
user0m7.425s
sys 0m0.827s

time ninja
cold cache:
real5m23.413s
user128m25.457s
sys 16m11.551s

time ninja:
warm cache:
real0m45.964s
user7m12.563s
sys 2m23.616s


On Wed, May 15, 2019 at 5:50 PM Vincent Hui  wrote:

> How many number of threads were used? Is the number of threads used by Qbs
> equal to the number of threads used by CMake?
>
>
> On Wed, 15 May 2019 at 22:11, Christian Kandeler 
> wrote:
>
>> On Thu, 16 May 2019 02:01:21 +1200
>> Christian Gagneraud  wrote:
>>
>> > As well, please note that the current CMake doesn't build as much
>> > stuff as the qbs one. Not sure the difference is huge, but you need to
>> > compare "equal jobs".
>>
>> For instance, all autotests are missing from the cmake build, so this
>> alone accounts for a difference of ~100 products. You need to set
>> "project.withAutotests:false" when building with qbs.
>>
>>
>> Christian
>> ___
>> Qbs mailing list
>> Qbs@qt-project.org
>> https://lists.qt-project.org/listinfo/qbs
>>
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Vincent Hui]

How many number of threads were used? Is the number of threads used by Qbs
equal to the number of threads used by CMake?


On Wed, 15 May 2019 at 22:11, Christian Kandeler 
wrote:

> On Thu, 16 May 2019 02:01:21 +1200
> Christian Gagneraud  wrote:
>
> > As well, please note that the current CMake doesn't build as much
> > stuff as the qbs one. Not sure the difference is huge, but you need to
> > compare "equal jobs".
>
> For instance, all autotests are missing from the cmake build, so this
> alone accounts for a difference of ~100 products. You need to set
> "project.withAutotests:false" when building with qbs.
>
>
> Christian
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Christian Kandeler]

On Thu, 16 May 2019 02:01:21 +1200
Christian Gagneraud  wrote:

> As well, please note that the current CMake doesn't build as much
> stuff as the qbs one. Not sure the difference is huge, but you need to
> compare "equal jobs".

For instance, all autotests are missing from the cmake build, so this
alone accounts for a difference of ~100 products. You need to set
"project.withAutotests:false" when building with qbs.


Christian
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Christian Gagneraud]

On Thu, 16 May 2019 at 01:45, Orgad Shaneh  wrote:
>
> I compared qbs and cmake/ninja, and got these results (Debug build, no QbsPM 
> and no Clang):
>
> Notice that qbs builds unit tests, which cmake doesn't.
>
> time qbs -f ../qt-creator/qtcreator.qbs profile:qt-5-11-1 
> modules.cpp.compilerWrapper:ccache
> cold ccache:
> real5m26.149s
> user125m3.443s
> sys 16m55.134s
>
> warm cache:
> real1m40.335s
> user17m20.419s
> sys 4m12.409s
>
> time cmake ../qt-creator -DCMAKE_CXX_COMPILER_LAUNCHER=ccache 
> -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Debug -G Ninja
> real0m8.256s
> user0m7.425s
> sys 0m0.827s
>
> time ninja
> cold cache:
> real5m23.413s
> user128m25.457s
> sys 16m11.551s
>
> time ninja:
> warm cache:
> real0m45.964s
> user7m12.563s
> sys 2m23.616s
>

Thanks for reporting, i'm planning to run my own benchmark as well,
not to state who's the winner, just for curiosity.
One important metric IMHO is the ratio sys/user, as it shows I/O
bottlenecks (esp. relevant when doing debug builds with hot ccache).
In my setup i will use ram disks for ccache and build dir (source dir
has nearly no effects)
It used to be my typical config (had to fight for memory tho), until i
got a new machine with NVRam as mass storage. It's almost as fast as
ramdisks.

As well, please note that the current CMake doesn't build as much
stuff as the qbs one. Not sure the difference is huge, but you need to
compare "equal jobs".

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Orgad Shaneh]

I compared qbs and cmake/ninja, and got these results (Debug build, no
QbsPM and no Clang):

Notice that qbs builds unit tests, which cmake doesn't.

time qbs -f ../qt-creator/qtcreator.qbs profile:qt-5-11-1
modules.cpp.compilerWrapper:ccache
cold ccache:
real5m26.149s
user125m3.443s
sys 16m55.134s

warm cache:
real1m40.335s
user17m20.419s
sys 4m12.409s

time cmake ../qt-creator -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_BUILD_TYPE=Debug -G Ninja
real0m8.256s
user0m7.425s
sys 0m0.827s

time ninja
cold cache:
real5m23.413s
user128m25.457s
sys 16m11.551s

time ninja:
warm cache:
real0m45.964s
user7m12.563s
sys 2m23.616s

- Orgad

On Wed, May 15, 2019 at 2:24 PM Epting, Thomas 
wrote:

>  > I would like to see qbs as a supported build system in Creator for as
> long as it does not
>  > cause too much hassle to do so. That is no different to any of the
> other plugins Creator
>  > has.
>
> Side note: I plan to take care of the qbsprojectmanager plugin (not Qbs
> itself) in the future. I also plan to do Qt6 adaptions to that plugin, if
> needed, and to keep the cmake build up to date. I've already spent some
> substantial effort in improving the plugin, it's just not yet pushed
> upstream because I don't have any time for doing so at the moment.
>
> Thomas
>
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Epting, Thomas]

 > I would like to see qbs as a supported build system in Creator for as long 
 > as it does not
 > cause too much hassle to do so. That is no different to any of the other 
 > plugins Creator
 > has.

Side note: I plan to take care of the qbsprojectmanager plugin (not Qbs itself) 
in the future. I also plan to do Qt6 adaptions to that plugin, if needed, and 
to keep the cmake build up to date. I've already spent some substantial effort 
in improving the plugin, it's just not yet pushed upstream because I don't have 
any time for doing so at the moment.

Thomas

___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Donation to QBS developers/maintainers/contributes

[Oswald Buddenhagen]

On Tue, May 14, 2019 at 04:17:41PM +0200, Иван Комиссаров wrote:
> Qbs itself has a huge dependency on a QtCore, but some stuff is
> present in standard library and can be easily changed to that (we
> still have enourmous amount of places where qlist is used instead of a
> qvector/std::vector and used inefficient);
>
yes, getting rid of qt containers is easy. only the qtcreator plugin
would still need them (and it's a good question where to draw the lines
to not introduce unnecessary conversion inefficiencies).

qstring is a lot harder ... 

> others are trickier but can be bootstrapped (like qfile) if needed.
>
bootstrapping qt classes is counterproductive (you notice latest when
you link libqbs into qtcreator) and laborious (you notice latest when
you get to qprocess or another class that *needs* full qobject/moc and
the qt event loop to operate sensibly).

as of now, there is no reasonable alternative to pulling in random
external dependencies, possibly boost (fwiw, the relevant c++ stdlib
projects are apparently dormant).

> But what about QtScript's dependency on the QtCore/other libs? My
> point was that it may depend on the qtlibs more than Qbs does and the
> biggest job is there, not in Qbs core.
> 
for the js engine, a bare JSC was considered.
hmm, come to think of it, the js engine might not be the only part from
webkit (or webengine) that makes sense to build upon ...

jake had started that "qt-free qbs" project (he got as far as
eliminating (most) use of qt containers), but it was snuffed out because
it was unrealistic and counterproductive at that time. even now, no
patches in that direction will be accepted unless a coherent final
picture and a credible committment to further maintenance is presented.
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Who compared build times of Qt Creator with CMake and qbs?

[Tobias Hunger]

Hi Richard,

I can only provide an opinion, it is not my place to make decisions.

On Tue, 2019-05-14 at 20:42 +0200, Richard Weickelt wrote:
> Tobias,
> 
> I think it is of utmost importance for the survival of Qbs that it is fully
> supported by an IDE like QtCreator.

I would like to see qbs as a supported build system in Creator for as long as it
does not cause too much hassle to do so. That is no different to any of the
other plugins Creator has.

> What would yo define as the minimum criteria for keeping Qbs supported in Qt
> Creator?

Keep it supported: Qbs needs to build with a Qt version supported by Qt Creator.

Keep it enabled in the binary packages: Qbs needs to build with the Qt version
used by the binary packages.

> > The challenge will be Qt 6: Creator will use Qt 6 ASAP and since qbs is
> > linked
> > into Creator, that will also need to be ported to Qt 6 to stay feasible.
> 
> Do you already have an idea when QtCreator will switch to Qt 6? Given the
> current release plan https://wiki.qt.io/Qt5Releasing, I expect the first
> release of Qt6 to happen around November 2020.

I do not know more about the timeline for Qt 6 than you do.

We will build Qt Creator with Qt 6 ASAP. It is a testbed and bug-detector for Qt
after all.

But as Andre already stated: We will also keep Qt 5 compatibility for as long as
that is practical.

Best Regards,
Tobias

-- 
Tobias Hunger, Senior Software Engineer | The Qt Company
The Qt Company GmbH, Rudower Chaussee 13, D-12489 Berlin
Geschäftsführer: Mika Pälsi, Juha Varelius, Mika Harjuaho. Sitz der
Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs