Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Friedemann Kleint via Development]

Hi,

quick status update: I think the major part of source code changes is 
done, few changes are still in the works or waiting for review. Thanks 
for the reviews and proposals. I think most changes were actually for 
the better and quite some cruft and duplication was unearthed. We are 
hoping to get it all in + a change adding an option to configure so that 
it can be used.


I am now able to build Qt (qtbase/qtdeclarative and additional modules) 
in 10min (home-office), whereas before, qtdeclarative alone would have 
taken longer than that. This shows the potential; also for the projects 
depending on us in terms of server load, time and energy consumption.


Regards, Friedemann

--
Friedemann Kleint
The Qt Company GmbH
--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Cristian Adam via Development]

On 01/16/2023 13:49, Friedemann Kleint via Development wrote:

Hi,

It is basically what webengine has been using for a long time: Source 
files are concatenated to jumbo files which are then compiled.


Back in August 2019 Chromium announced the removal of the jumbo build 
 
with the following rationale:


   /We have decided to remove support for the jumbo (aka unity) build
   configuration from the project.//
   //
   //We are very conscious of the fact that Chromium is very expensive
   and slow to compile, and that that represents a real barrier to
   contributing.//
   //
   //However, based on the experience we've gained with this
   configuration over the past many months it's been supported, we've
   reached the conclusion that the way jumbo is currently implemented,
   it interferes too much with the way people frequently write C++
   code; you have to work around jumbo-specific conflicts in a way that
   can be awkward, unnatural, or otherwise unnecessary. As a result, we
   feel that the cost to the project of maintaining jumbo is not worth
   the benefit it is providing./

Did they brought it back or is webengine maintaining a patch to support it?

Cheers,
Cristian.
-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Sérgio Martins via Development]

On 2023-01-23 14:25, Jörg Bornemann via Development wrote:
(...)

One ugliness I've noticed: CMAKE_UNITY_BUILD is documented to not play
well with CMAKE_EXPORT_COMPILE_COMMANDS. This might hurt users of
clangd (using clangd with Qt's sources).


Yes, but probably you won't be developing with CMAKE_UNITY_BUILD, since 
it ruins incremental build times anyway.


It's great for compiling 3rdparty and any module you're *not* 
developing.



Regards,
--
Sérgio Martins | sergio.mart...@kdab.com | Senior Software Engineer
Klarälvdalens Datakonsult AB, a KDAB Group company
Tel: Sweden (HQ) +46-563-540090, USA +1-866-777-KDAB(5322)
KDAB - The Qt, C++ and OpenGL Experts
--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Jörg Bornemann via Development]

On 1/23/23 10:20, Friedemann Kleint via Development wrote:


As for Shawn asking whether the concatenation is deterministic; in my 
experience, it is.


The concatenation is determinic and depends on the file order in the 
CMake files and the batch size.


One ugliness I've noticed: CMAKE_UNITY_BUILD is documented to not play 
well with CMAKE_EXPORT_COMPILE_COMMANDS. This might hurt users of clangd 
(using clangd with Qt's sources).


--
Jörg Bornemann | The Qt Company

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Friedemann Kleint via Development]

Hi,

thank you for the reponses; so we conclude that we should support it and use it 
partially in COIN, ideally for slow platforms. I will try to focus on qtbase 
and qttools. qtdeclarative would also be a worthy goal; would someone be up for 
it? Basically the changes linked to 
https://bugreports.qt.io/browse/QTBUG-109394 and 
https://bugreports.qt.io/browse/PYSIDE-2155  give you an idea what to do.

As for Shawn asking whether the concatenation is deterministic; in my 
experience, it is.

Regards, Friedemann
--

Friedemann Kleint
The Qt Company GmbH
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Thiago Macieira]

On Tuesday, 17 January 2023 01:44:09 PST Shawn Rutledge via Development wrote:
> Any trouble with icecream after this?

I think you need to measure to see if it's worth it. My gut feeling is that it 
isn't for a large cluster. The amount of preprocessing work will drop, but it 
might not be meaningful. However, the data transfer will increase, both in the 
sending of preprocessed sources (which compress well) and the resulting binary 
objects (which don't). 

Then there's cluster utilisation: if the number of jumbo files is smaller than 
the number of job slots available, you're underutilising your cluster.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering


smime.p7s
Description: S/MIME cryptographic signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Christian Tismer-Sperling]

On 17.01.23 17:11, Thiago Macieira wrote:

On Tuesday, 17 January 2023 02:55:42 PST Christian Tismer-Sperling wrote:

What was the reason that you first considered to object to it?


Because it adds work for those of us who don't use that thing.

Unity builds are faster than non-unity if you're doing a full build and you
don't have access to a cluster. But if you're doing an incremental build while
developing, they're not. Adding it to COIN implies that everyone is
responsible for keeping them working.


Ah, sure! I still use it on and off. Probably I'm seeing less
effect because I almost always do an incremental build.
Did not think about it, thanks!

--
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Strandstraße 37  : https://github.com/PySide
24217 Schönberg  : GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Thiago Macieira]

On Tuesday, 17 January 2023 07:26:28 PST Volker Hilsheimer via Development 
wrote:
> No idea why windows-11_22h2-mingw11-developer-build is so slow

My experience is that the standard GNU binutils ld is mighty slow on Windows. 
That might be the case for all platforms other than ELF ones, but I don't have 
any non-Windows non-ELF to test on. It has a quadratic symbol search time, so 
it's slow even when cross-compiling and file system access is fast. On Windows, 
filesystem access is not fast.

An LLVM lld build might be faster. This need not be a Clang build, as GCC 
accepts -fuse-ld=lld. I've just tested:

$ gcc -fuse-ld=lld -xc /dev/null
ld.lld: error: undefined symbol: main
>>> referenced by start.S:103 (../sysdeps/x86_64/start.S:103)
>>>   /usr/lib64/gcc/x86_64-generic-linux/12/../../../../lib64/
Scrt1.o:(_start)
collect2: error: ld returned 1 exit status


-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering


smime.p7s
Description: S/MIME cryptographic signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Thiago Macieira]

On Tuesday, 17 January 2023 02:55:42 PST Christian Tismer-Sperling wrote:
> What was the reason that you first considered to object to it?

Because it adds work for those of us who don't use that thing.

Unity builds are faster than non-unity if you're doing a full build and you 
don't have access to a cluster. But if you're doing an incremental build while 
developing, they're not. Adding it to COIN implies that everyone is 
responsible for keeping them working.

But it's for a good reason.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering


smime.p7s
Description: S/MIME cryptographic signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Edward Welbourne via Development]

On Monday, 16 January 2023 04:49:23 PST Friedemann Kleint via Development wrote:
> Summmarising: we stand to gain a speed-up of compilation;
> particularly for clean builds like in COIN; but it requires some
> work. We might do a step-by step approach process enabling modules
> one by one. What do you think?

On 17.01.23 05:09, Thiago Macieira wrote:
 I was going to object to it being in COIN, but considering it may
 improve build times in the CI, especially for constrained platforms
 (like macOS), I withdraw my objection before I even make it. I
 think this effort is worth it.

Christian Tismer-Sperling (17 January 2023 11:55) asked:
>>> What was the reason that you first considered to object to it?

On 17 Jan 2023, at 13:35, Edward Welbourne via Development 
 wrote:
>> While I can't be sure what Thiago was thinking of, one obvious
>> objection to doing this in Coin would be that: if someone changes a
>> source file in a way that needs a #include addition to the changed
>> source file, but Coin's Unity build combines that file with one that
>> has the #include, Coin won't notice, the change will get integrated
>> and those doing a non-Unity builds, or for whom the Unity build
>> splits files up differently, may find themselves with broken builds.
>> I imagine there are other mistakes that can likewise be concealed by
>> a Unity build.

Volker Hilsheimer (17 January 2023 16:26) replied:
> I would actually turn Thiago’s objection on its head and ask that we
> have at least one configuration in COIN that verifies that
> UNITY_BUILDs work, if we agree that this is a good idea.

That's more or less the conclusion I understood Thaigo to have reached,
too.

> In the short term I don’t expect that this will improve effective CI
> performance considerably, at least not until every CI configuration
> uses UNITY_BUILDs. The spread of build times in
> e.g. https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1681530493
> is wild (from a few minutes to almost 1.5hours). No idea why
> windows-11_22h2-mingw11-developer-build is so slow, but in the end,
> the slowest platform for building and testing defines the time we have
> to wait until we know that our changes were good (failing build or
> test shortcuts that, of course).

That rather suggests making the slower builders use Unity while keeping
the fastest build of each platform a traditional build.  Then we test
that both types work while improving Coin's build speeds.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Volker Hilsheimer via Development]

On 17 Jan 2023, at 13:35, Edward Welbourne via Development 
 wrote:

On Monday, 16 January 2023 04:49:23 PST Friedemann Kleint via Development wrote:
Summmarising: we stand to gain a speed-up of compilation; particularly
for clean builds like in COIN; but it requires some work. We might do a
step-by step approach process enabling modules one by one. What do you
think?

On 17.01.23 05:09, Thiago Macieira wrote:
I was going to object to it being in COIN, but considering it may improve
build times in the CI, especially for constrained platforms (like macOS), I
withdraw my objection before I even make it. I think this effort is worth it.

Christian Tismer-Sperling (17 January 2023 11:55) asked:
What was the reason that you first considered to object to it?

While I can't be sure what Thiago was thinking of, one obvious objection
to doing this in Coin would be that: if someone changes a source file in
a way that needs a #include addition to the changed source file, but
Coin's Unity build combines that file with one that has the #include,
Coin won't notice, the change will get integrated and those doing a
non-Unity builds, or for whom the Unity build splits files up
differently, may find themselves with broken builds.  I imagine there
are other mistakes that can likewise be concealed by a Unity build.

Eddy.


I would actually turn Thiago’s objection on its head and ask that we have at 
least one configuration in COIN that verifies that UNITY_BUILDs work, if we 
agree that this is a good idea.

Better usage of our CI capacity is good for the project, so anything that helps 
speeding up the build time with reasonable, sustainable effort is good. Since 
making UNITY_BUILDs work require unintuitive changes (such as having to 
disambiguate names that look like they are local to a translation unit), we 
should make sure that we don’t break such builds with the next commit adding a 
static function or a symbol in the anonymous namespace.

In the short term I don’t expect that this will improve effective CI 
performance considerably, at least not until every CI configuration uses 
UNITY_BUILDs. The spread of build times in e.g. 
https://testresults.qt.io/coin/integration/qt/qtbase/tasks/1681530493 is wild 
(from a few minutes to almost 1.5hours). No idea why 
windows-11_22h2-mingw11-developer-build is so slow, but in the end, the slowest 
platform for building and testing defines the time we have to wait until we 
know that our changes were good (failing build or test shortcuts that, of 
course).

But small steps. And as long as we have only a few UNITY_BUILD-configurations, 
it seems unlikely that we don’t catch changes that break non-UNITY_BUILDs in 
other configurations.

So, I like the idea. The amount of code we would have to touch seems 
comparatively small, so we shouldn't end up with a ton of code churn that 
breaks cherry-picks to older branches. However, I’d like to understand the pros 
and cons of the different mechanism cmake gives us to prevent issues. If the 
downside of excluding files with e.g. ambiguous statics from UNITY_BUILDs is 
small, then would that be a better approach than putting some _qt_xyx_ prefix 
in front of every static, at least as a first step?

Volker

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Edward Welbourne via Development]

On Monday, 16 January 2023 04:49:23 PST Friedemann Kleint via Development wrote:
>>> Summmarising: we stand to gain a speed-up of compilation; particularly
>>> for clean builds like in COIN; but it requires some work. We might do a
>>> step-by step approach process enabling modules one by one. What do you
>>> think?

On 17.01.23 05:09, Thiago Macieira wrote:
>> I was going to object to it being in COIN, but considering it may improve
>> build times in the CI, especially for constrained platforms (like macOS), I
>> withdraw my objection before I even make it. I think this effort is worth it.

Christian Tismer-Sperling (17 January 2023 11:55) asked:
> What was the reason that you first considered to object to it?

While I can't be sure what Thiago was thinking of, one obvious objection
to doing this in Coin would be that: if someone changes a source file in
a way that needs a #include addition to the changed source file, but
Coin's Unity build combines that file with one that has the #include,
Coin won't notice, the change will get integrated and those doing a
non-Unity builds, or for whom the Unity build splits files up
differently, may find themselves with broken builds.  I imagine there
are other mistakes that can likewise be concealed by a Unity build.

Eddy.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Christian Tismer-Sperling]

On 17.01.23 05:09, Thiago Macieira wrote:

On Monday, 16 January 2023 04:49:23 PST Friedemann Kleint via Development
wrote:

Summmarising: we stand to gain a speed-up of compilation; particularly
for clean builds like in COIN; but it requires some work. We might do a
step-by step approach process enabling modules one by one. What do you
think?


I was going to object to it being in COIN, but considering it may improve
build times in the CI, especially for constrained platforms (like macOS), I
withdraw my objection before I even make it. I think this effort is worth it.



What was the reason that you first considered to object to it?

--
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Strandstraße 37  : https://github.com/PySide
24217 Schönberg  : GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Shawn Rutledge via Development]

> On 16 Jan 2023, at 13:49, Friedemann Kleint via Development 
>  wrote:
> 
> Hi,
> 
> I would like to propose considering making Qt suitable for CMake UNITY_BUILDs 
> ( https://bugreports.qt.io/browse/QTBUG-109394 , 
> https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html ) .
> 
> This has originally been proposed for Qt for Python  ( 
> https://bugreports.qt.io/browse/PYSIDE-2155 ). It has greatly benefited from 
> it; builds are much faster; particuarly for us in home-office where no 
> distributed build environment exists.
> 
> It is basically what webengine has been using for a long time: Source files 
> are concatenated to jumbo files which are then compiled.
> 
> For Qt for Python with its lots of generated files, the speed-up is drastic 
> (~10x); for an average module (qttools), less so, but still:
> 
> Initial build of qttools:
> 
> Standard (675 files): real 3m26,016s / user 19m4,352s / sys 1m48,346s
> Unity (203 files)   : real 2m2,557s  / user 9m40,147s / sys 0m47,144s 
> (CMAKE_UNITY_BUILD_BATCH_SIZE=16)
> 
> with CCACHE/warmed up:
> 
> Standard (675 files): real 1m50,605s / user 8m9,512s  / sys 1m9,187s
> Unity (203 files)   : real 1m30,990s / user 5m11,773s / sys 0m39,687s

Speaking of ccache, how stable is the selection of files that are concatenated? 
 I suppose small changes will tend to invalidate more of the cache? Where 
before you could make a change in one cpp, and only one smaller cache object 
was invalidated, now it will invalidate the whole jumbo file’s cache, and after 
all rebuilding the jumbo file should take longer and produce a larger object 
than the smaller file that was changed.  But I guess that’s the way it goes, 
since we are rebuilding Qt to test changes, not just rebuilding the same 
version over and over.  I wonder if the ccache hit rate will actually go down 
though, so incremental rebuilding when you are just changing a line at a time 
could be slower.

Any trouble with icecream after this?

Otherwise it sounds like a win, mostly.

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Thiago Macieira]

On Monday, 16 January 2023 04:49:23 PST Friedemann Kleint via Development 
wrote:
> Summmarising: we stand to gain a speed-up of compilation; particularly
> for clean builds like in COIN; but it requires some work. We might do a
> step-by step approach process enabling modules one by one. What do you
> think?

I was going to object to it being in COIN, but considering it may improve 
build times in the CI, especially for constrained platforms (like macOS), I 
withdraw my objection before I even make it. I think this effort is worth it.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Cloud Software Architect - Intel DCAI Cloud Engineering


smime.p7s
Description: S/MIME cryptographic signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] CMake UNITY_BUILD ( QTBUG-109394 )

[Friedemann Kleint via Development]

Hi,

I would like to propose considering making Qt suitable for CMake 
UNITY_BUILDs ( https://bugreports.qt.io/browse/QTBUG-109394 , 
https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html ) .


This has originally been proposed for Qt for Python  ( 
https://bugreports.qt.io/browse/PYSIDE-2155 ). It has greatly benefited 
from it; builds are much faster; particuarly for us in home-office where 
no distributed build environment exists.


It is basically what webengine has been using for a long time: Source 
files are concatenated to jumbo files which are then compiled.


For Qt for Python with its lots of generated files, the speed-up is 
drastic (~10x); for an average module (qttools), less so, but still:


Initial build of qttools:

Standard (675 files): real 3m26,016s / user 19m4,352s / sys 1m48,346s
Unity (203 files)   : real 2m2,557s  / user 9m40,147s / sys 0m47,144s 
(CMAKE_UNITY_BUILD_BATCH_SIZE=16)


with CCACHE/warmed up:

Standard (675 files): real 1m50,605s / user 8m9,512s  / sys 1m9,187s
Unity (203 files)   : real 1m30,990s / user 5m11,773s / sys 0m39,687s

Increasing CMAKE_UNITY_BUILD_BATCH_SIZE might also give better results.

Some modifications of the source code are required, though. The majority 
is actually improvements; things like missing header guards and macro 
re-definitions immediately show up; also Q_DECLARE_METATYPE() can be 
removed (see 
https://lists.qt-project.org/pipermail/development/2022-July/042744.html ).


But there is also a part that is more questionable; static functions 
need to be disambiguated; "using namespace" can cause clashes; class 
member fields from anonymous namespaces cause errors ( 
-Werror=subobject-linkage producing qerrormessage.cpp:38:7: error: 
‘QErrorMessagePrivate’ has a field ‘QErrorMessagePrivate::pending’ whose 
type uses the anonymous namespace ) and similar.


Some things are not fixable; exclusions can be done on a source file or 
project level. Alternatively, one can group suitable source files 
together; but this requires more work.


In development work, if COIN uses UNITY_BUILD, there is also a small 
risk of changes with missing header files being accepted.


Summmarising: we stand to gain a speed-up of compilation; particularly 
for clean builds like in COIN; but it requires some work. We might do a 
step-by step approach process enabling modules one by one. What do you 
think?


Regards, Friedemann

--

Friedemann Kleint
The Qt Company GmbH

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development