Re: [Development] Unity Build on CI for Qt Base for 3 Platforms

[Jörg Bornemann via Development]

On 8/14/23 12:47, Friedrich W. H. Kossebau wrote:


While at it, I am curious what purpose unity builds of Qt are supposed to
serve, i.e. in which cases would exclusive use of unity builds be recommended
to be used? Are there some docs/notes somewhere, when to use and when not?


Unity builds are mostly for improving build speed. If you're just 
building Qt modules without actively working on the files then you 
should see build speed improvements. If you're actively working on a Qt 
module you don't want to use unity builds, because you don't want big 
chunks of the module being rebuilt just because you're working on one 
file. One thing we're hoping for is improved CI times with unity builds.



BR,

Joerg

--
Jörg Bornemann | The Qt Company

--
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Unity Build on CI for Qt Base for 3 Platforms

[Friedrich W. H. Kossebau]

Hi,

Am Freitag, 11. August 2023, 22:14:13 CEST schrieb Marc Mutz via Development:
> All my -unity-build-batch-size 100'000 patches except for one are now
> merged: https://codereview.qt-project.org/q/topic:unity-build
> 
> I'm currently testing without the unmerged one, because it won't apply
> to 6.5, anyway, and the goal is to have unity-builds from 6.5 onwards.
> Maybe a few more TUs will have to be excluded, but you should be able to
> see the pattern. So far it looks like it's not needed.
> 
> Compile with -unity-build -unity-build-batch-size 10 and add all TUs
> that create problems with a small comment as to what's breaking to
> NO_UNITY_BUILD_SOURCES.

Just a quick note/question given 10 seems to be used to denote "all the 
sources!":

CMake actually supports "0" as value for that purpose of "all!", had that been 
considered in the design, to be consistent with CMake's approach?

>From https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD_BATCH_SIZE.html
--- 8< ---
Although strongly discouraged, the batch size may be set to a value of 0 to 
combine all the sources for the target into a single unity file, regardless of 
how many sources are involved. This runs the risk of creating an excessively 
large unity source file and negatively impacting the build performance, so a 
value of 0 is not generally recommended.
--- 8< ---

Curiousity:

While at it, I am curious what purpose unity builds of Qt are supposed to 
serve, i.e. in which cases would exclusive use of unity builds be recommended 
to be used? Are there some docs/notes somewhere, when to use and when not?
Personally had been very impressed by the speed up when I first learned about 
the concept decades ago (KDE's --enable-final flag with unsermake IIRC) and 
the compiler at that time having more options to optimize the generated code. 
But with LTO these days, by what I understood, that advantage/purpose is gone.
And while I made some own project recently work with cmake unity builds for 
curiousity (thus knew about that special value "0" for batch sizes :) ) I am 
still wondering when actually unity builds make sense to be used, especially 
in parallel with supporting non-unity builds.
So, for which use cases are Qt's unity builds done? Whom will the docs 
recommend to use that build option?

Cheers
Friedrich


-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Unity Build on CI for Qt Base for 3 Platforms

[Marc Mutz via Development]

Hi,

All my -unity-build-batch-size 100'000 patches except for one are now 
merged: https://codereview.qt-project.org/q/topic:unity-build

I'm currently testing without the unmerged one, because it won't apply 
to 6.5, anyway, and the goal is to have unity-builds from 6.5 onwards. 
Maybe a few more TUs will have to be excluded, but you should be able to 
see the pattern. So far it looks like it's not needed.

Compile with -unity-build -unity-build-batch-size 10 and add all TUs 
that create problems with a small comment as to what's breaking to 
NO_UNITY_BUILD_SOURCES.

I've done the work on Linux, we need someone that does the (considerably 
less involved) work on at least Windows and OSX (though the immediate 
problem that caused the CI to block was Integrity, of course).

Please use topic:unity-build, and link to the most fitting Jira tickets 
in the https://bugreports.qt.io/browse/QTBUG-109394 epic.

We should then be able to re-enable unity builds on the CI, and maintain 
a 10 build as a nightly.

Thanks,
Marc

On 21.07.23 10:25, Marc Mutz via Development wrote:
> Hi all,
> 
> I don't like the "what _you_ should expect". IMNSHO, _we_ should expect
> it to break only when _we_ add code that causes it to break. IOW: before
> we enable this feature on the production CI for a given configuration X,
> the submitter should have tried configuration X with a
> -unity-build-batch-size of 10, or whatever is necessary to ensure
> that each sub-project gets built with exactly one unity__cxx.cxx,
> and exclude all the .cpp files that cause problems in those builds with
> NO_UNITY_BUILD_SOURCES. Only then can we enable the feature for that
> configuration, being (reasonably) certain that a) the batch size
> configured doesn't matter and b) introducing new .cpp files into a
> sub-project's SOURCES causes unity-builds to blow up _only_ due to
> content in _that_ .cpp file.
> 
> I have done that work for qtbase and qtdeclarative and Linux yesterday
> instead of my normal work. I can continue with the rest for Linux
> (already on qttools), but someone else needs to do the same for Windows,
> OSX, Integrity, and whatever other configuration we want to enable
> unity-builds for, _before_ enabling them.
> 
> As it is now, with unity-builds only checked to some insufficient finite
> limit, any unrelated change, even Wednesday's _decrease_ (if it was that)
> of the batch size from 32 to 16, can break the build of totally unrelated
> changes.
> 
> Yesterday was therefore a lost day for the dev CI, and everyone that
> depends on it to integrate their patches. This should be enough of a
> shot across the bow to reconsider strategy.
> 
> I therefore propose to disable all unity-builds on the production CI
> until the proper tests have been done locally (preferably, with a
> nightly build to prove it to reviewers).
> 
> Proposed reverts end in
> https://codereview.qt-project.org/c/qt/qt5/+/492193 (already approved by
> Volker, thanks).
> 
> Thanks,
> Marc
> 
> On 26.06.23 09:39, EXT Amir Abdol via Development wrote:
>> Hi,
>>
>> Unity Build is now enabled on CI for *Qt Base*, on *dev* branch, for the
>> following platforms: *INTEGRITY, Windows 10 MinGW, Windows 11 MinGW*,
>> which are among the slowest platforms on our CI.
>>
>> Here is a brief note about the unity build, and what you should expect:
>>
>>* Unity build achieves a faster build by pooling several source files
>>  into one translation unit and compiling that instead of individually
>>  compiling every source file. This means the build is especially
>>  susceptible to duplicate symbol collision. If you notice such instances,
>>o Try resolving the issue, as it is often achievable in a
>>  reasonable timeframe. If not,
>>o You can exclude the file from the unity build, by adding it to
>>  |NO_UNITY_BUILD_SOURCES|, which is available in most of our
>>  |qt_internal_*| CMake APIs, e.g., |qt_internal_add_tool|. When
>>  doing so, please leave a brief comment and/or TODO on why/how to
>>  resolve the issue.
>>o If an entire |target| is troublesome, and resolving the issue
>>  needs major work, you can exclude it from the unity build by
>>  passing |NO_UNITY_BUILD|to the respective |qt_internal_*| call.
>>* You can enable the unity build locally by passing |-unity-build| and
>>  |-unity-build-batch-size| to |configure|, or use
>>  |-DQT_UNITY_BUILD|if you are using CMake. Default batch size is 32.
>>* We have tested the build with batch size of 32. Changing the batch
>>  size may result in undiscovered symbol collisions, or other issues.
>>* Tests and examples are not being built using unity build at the moment.
>>
>> While at the moment, we are only partially enabling the unity build for
>> Qt Base on CI, the build should be successful on other platforms as
>> well. Other Qt modules can be built with unity build as well, with
>> varying 

Re: [Development] Unity Build on CI for Qt Base for 3 Platforms

[Marc Mutz via Development]

Hi all,

I don't like the "what _you_ should expect". IMNSHO, _we_ should expect 
it to break only when _we_ add code that causes it to break. IOW: before 
we enable this feature on the production CI for a given configuration X, 
the submitter should have tried configuration X with a 
-unity-build-batch-size of 10, or whatever is necessary to ensure 
that each sub-project gets built with exactly one unity__cxx.cxx, 
and exclude all the .cpp files that cause problems in those builds with 
NO_UNITY_BUILD_SOURCES. Only then can we enable the feature for that 
configuration, being (reasonably) certain that a) the batch size 
configured doesn't matter and b) introducing new .cpp files into a 
sub-project's SOURCES causes unity-builds to blow up _only_ due to 
content in _that_ .cpp file.

I have done that work for qtbase and qtdeclarative and Linux yesterday 
instead of my normal work. I can continue with the rest for Linux 
(already on qttools), but someone else needs to do the same for Windows, 
OSX, Integrity, and whatever other configuration we want to enable 
unity-builds for, _before_ enabling them.

As it is now, with unity-builds only checked to some insufficient finite 
limit, any unrelated change, even Wednesday's _decrease_ (if it was that)
of the batch size from 32 to 16, can break the build of totally unrelated
changes. 

Yesterday was therefore a lost day for the dev CI, and everyone that 
depends on it to integrate their patches. This should be enough of a 
shot across the bow to reconsider strategy.

I therefore propose to disable all unity-builds on the production CI 
until the proper tests have been done locally (preferably, with a 
nightly build to prove it to reviewers).

Proposed reverts end in 
https://codereview.qt-project.org/c/qt/qt5/+/492193 (already approved by 
Volker, thanks).

Thanks,
Marc

On 26.06.23 09:39, EXT Amir Abdol via Development wrote:
> Hi,
> 
> Unity Build is now enabled on CI for *Qt Base*, on *dev* branch, for the 
> following platforms: *INTEGRITY, Windows 10 MinGW, Windows 11 MinGW*, 
> which are among the slowest platforms on our CI.
> 
> Here is a brief note about the unity build, and what you should expect:
> 
>   * Unity build achieves a faster build by pooling several source files
> into one translation unit and compiling that instead of individually
> compiling every source file. This means the build is especially
> susceptible to duplicate symbol collision. If you notice such instances,
>   o Try resolving the issue, as it is often achievable in a
> reasonable timeframe. If not,
>   o You can exclude the file from the unity build, by adding it to
> |NO_UNITY_BUILD_SOURCES|, which is available in most of our
> |qt_internal_*| CMake APIs, e.g., |qt_internal_add_tool|. When
> doing so, please leave a brief comment and/or TODO on why/how to
> resolve the issue.
>   o If an entire |target| is troublesome, and resolving the issue
> needs major work, you can exclude it from the unity build by
> passing |NO_UNITY_BUILD|to the respective |qt_internal_*| call.
>   * You can enable the unity build locally by passing |-unity-build| and
> |-unity-build-batch-size| to |configure|, or use
> |-DQT_UNITY_BUILD|if you are using CMake. Default batch size is 32.
>   * We have tested the build with batch size of 32. Changing the batch
> size may result in undiscovered symbol collisions, or other issues.
>   * Tests and examples are not being built using unity build at the moment.
> 
> While at the moment, we are only partially enabling the unity build for 
> Qt Base on CI, the build should be successful on other platforms as 
> well. Other Qt modules can be built with unity build as well, with 
> varying degrees of success. Please give it a try, and let us know if you 
> ran into issues, either by opening a bug report, or submitting a patch!
> 
> P.S. Unity build for Qt was initially discussed here: 
> https://lists.qt-project.org/pipermail/development/2023-January/043518.html 
> .
> 
> Best,
> 
> Amir.
> 
> 
-- 
Marc Mutz 
Principal Software Engineer

The Qt Company
Erich-Thilo-Str. 10 12489
Berlin, Germany
www.qt.io

Geschäftsführer: Mika Pälsi, Juha Varelius, Jouni Lintunen
Sitz der Gesellschaft: Berlin,
Registergericht: Amtsgericht Charlottenburg,
HRB 144331 B

-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Unity Build on CI for Qt Base for 3 Platforms

[Volker Hilsheimer via Development]

On 26 Jun 2023, at 09:39, EXT Amir Abdol via Development 
 wrote:


Hi,

Unity Build is now enabled on CI for Qt Base, on dev branch, for the following 
platforms: INTEGRITY, Windows 10 MinGW, Windows 11 MinGW, which are among the 
slowest platforms on our CI.

Here is a brief note about the unity build, and what you should expect:

  *   Unity build achieves a faster build by pooling several source files into 
one translation unit and compiling that instead of individually compiling every 
source file. This means the build is especially susceptible to duplicate symbol 
collision. If you notice such instances,
 *   Try resolving the issue, as it is often achievable in a reasonable 
timeframe. If not,
 *   You can exclude the file from the unity build, by adding it to 
NO_UNITY_BUILD_SOURCES, which is available in most of our qt_internal_* CMake 
APIs, e.g., qt_internal_add_tool. When doing so, please leave a brief comment 
and/or TODO on why/how to resolve the issue.
 *   If an entire target is troublesome, and resolving the issue needs 
major work, you can exclude it from the unity build by passing NO_UNITY_BUILDto 
the respective qt_internal_* call.
  *   You can enable the unity build locally by passing -unity-build and 
-unity-build-batch-size to configure, or use -DQT_UNITY_BUILDif you are using 
CMake. Default batch size is 32.
  *   We have tested the build with batch size of 32. Changing the batch size 
may result in undiscovered symbol collisions, or other issues.
  *   Tests and examples are not being built using unity build at the moment.

While at the moment, we are only partially enabling the unity build for Qt Base 
on CI, the build should be successful on other platforms as well. Other Qt 
modules can be built with unity build as well, with varying degrees of success. 
Please give it a try, and let us know if you ran into issues, either by opening 
a bug report, or submitting a patch!

P.S. Unity build for Qt was initially discussed here: 
https://lists.qt-project.org/pipermail/development/2023-January/043518.html.

Best,

Amir

Very nice, thanks a lot!

Cheers,
Volker


-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Unity Build on CI for Qt Base for 3 Platforms

[EXT Amir Abdol via Development]

Hi,

Unity Build is now enabled on CI for Qt Base, on dev branch, for the following 
platforms: INTEGRITY, Windows 10 MinGW, Windows 11 MinGW, which are among the 
slowest platforms on our CI.

Here is a brief note about the unity build, and what you should expect:

  *   Unity build achieves a faster build by pooling several source files into 
one translation unit and compiling that instead of individually compiling every 
source file. This means the build is especially susceptible to duplicate symbol 
collision. If you notice such instances,
 *   Try resolving the issue, as it is often achievable in a reasonable 
timeframe. If not,
 *   You can exclude the file from the unity build, by adding it to 
NO_UNITY_BUILD_SOURCES, which is available in most of our qt_internal_* CMake 
APIs, e.g., qt_internal_add_tool. When doing so, please leave a brief comment 
and/or TODO on why/how to resolve the issue.
 *   If an entire target is troublesome, and resolving the issue needs 
major work, you can exclude it from the unity build by passing NO_UNITY_BUILDto 
the respective qt_internal_* call.
  *   You can enable the unity build locally by passing -unity-build and 
-unity-build-batch-size to configure, or use -DQT_UNITY_BUILDif you are using 
CMake. Default batch size is 32.
  *   We have tested the build with batch size of 32. Changing the batch size 
may result in undiscovered symbol collisions, or other issues.
  *   Tests and examples are not being built using unity build at the moment.

While at the moment, we are only partially enabling the unity build for Qt Base 
on CI, the build should be successful on other platforms as well. Other Qt 
modules can be built with unity build as well, with varying degrees of success. 
Please give it a try, and let us know if you ran into issues, either by opening 
a bug report, or submitting a patch!

P.S. Unity build for Qt was initially discussed here: 
https://lists.qt-project.org/pipermail/development/2023-January/043518.html.

Best,

Amir.
-- 
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development