Re: [Development] Building additional components with Conan for Qt 6

[Konstantin Tokarev]

02.10.2020, 15:53, "Toni Saario" :
> As Iikka said that there has not been plans to use Conan when provisioning 
> the images for CI.

That's sad, I hoped plan was to use Conan for QtBase dependencies eventually to 
avoid custom bundling code and 3rd-party sources checked into the repo. And 
also eliminate potential for conflicts when user's project wants to use the 
same libraries as bundled into QtBase

>
> However there has been some discussions how to make the provisioning easier 
> to manage and have some documentability in them. Currently the provisioning 
> is just a pile of scripts which have zero value for documentation tools. 
> Easiest solution for this would be to list the dependencies on some other 
> file in common format and then install listed dependencies with simple 
> scripts. About using Conan there, I cannot really say anything about it with 
> my knowledge about it.
>
> Another thing is that there are many items built during provisioning which 
> become dependencies, can we reduce the provisioning workload by having these 
> cached in Conan storage?

Yes, I've done this for QtWebKit dependencies. Packages of dependencies are 
built on public CI services for all necessary compilers and uploaded to 
https://bintray.com/qtproject/, then during provisioning they are downloaded 
and verified.

> One thing to check is that how much the provisioning it is possible to 
> translate to Conan, I would believe that in the end it does not work for 
> everything.

Most of provisioning scripts fall into 2 categories:
1) Download installer (or other binary package) of 3rd party stuff, verify 
checksum, install. These scripts are quite simple.
2) Build stuff from sources in provisioning time, sometimes in a sophisticated 
way (e.g. 
https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/android-openssl.ps1).
 These can be (and IMO should be) replaced with Conan, to reduce provisioning 
time and complexity.

>
> As for provisioning and why the CI images are different from what are 
> documented, biggest problem there is that the images are used by other 
> components also e.g. RTA testing. Those require some additional stuff and 
> provisioning currently installs everything. AFAIK also all the optional 
> dependencies are installed, this makes it hard to distinguish between them 
> and required dependencies because both are always present. These make it easy 
> that new dependencies etc. slip through unnoticed.

Note that when Conan packages are used via conanfiles (i.e. as it's originally 
intended by Conan developers), there cannot be any dependency slips because 
Conan dependencies don't become visible to any project which doesn't require 
them via conanfile (directly or indirectly).

I can see another possible way to use Conan in CI - install packages into the 
system via "deploy" generator so that Qt build system can find them without any 
knowledge of Conan. I guess this way may be called "Ossi-friendly because Ossi 
was adamantly opposed to any explicit support of Conan in build system, see 
e.g. https://codereview.qt-project.org/c/qt/qtbase/+/184241, however this way 
indeed may cause dependency slips.

>
> -Toni
>
> 
> From: Konstantin Tokarev 
> Sent: Friday, October 2, 2020 3:20 PM
> To: Iikka Eklund ; Richard Weickelt 
> ; development@qt-project.org 
> ; Toni Saario 
> Subject: Re: [Development] Building additional components with Conan for Qt 6
>
> 02.10.2020, 14:47, "Iikka Eklund" :
>> Hi Konstantin, all,
>>
  thanks for the heads up. I have some further questions:

  1. Will Conan be used to manage dependencies of Qt as well?

 If this is the plan, note that there is infrastructure [1] for using Conan 
 in provisioning images,
 with verification of downloaded file checksums and build reproducibility 
 for each qt5 commit [2].
>>
>> Regarding Qt CI @Toni Saario is the best one to answer on this, though I 
>> haven't heard
>> if there are plans to use Conan more extensively by CI/provisioning at this 
>> point?
>
> From my experience, when the way project is built in CI is radically 
> different from a way suggested to
> the end user, it inevitably leads to troubles. Users hit unexpected errors 
> (in this case they may be
> caused by changes in 3rd party Conan recipes/packages, or by contributors to 
> Qt module which
> don't use Conan), and they come back to me asking for solution.
>
>>
>> What comes to add-ons 
>> (https://doc-snapshots.qt.io/qt6-dev/qtmodules.html#qt-add-ons) as mentioned
>> earlier the idea is to release add-ons as src packages accompanied with 
>> conanfile.py i.e. the build recipe.
>> If the add-on depends on 3rd party libraries which are available e.g. in 
>> conan-center the recipe could
>> naturally utilize packages from there (openssl, zlib, etc.).
>>
>>> 2. How will the recipes be managed and where on code.qt.io can I find 
>>>them?
>>
>> The recipes of the first 

Re: [Development] Building additional components with Conan for Qt 6

[Toni Saario]

>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>>
>>> If this is the plan, note that there is infrastructure [1] for using Conan 
>>> in provisioning images,
>>> with verification of downloaded file checksums and build reproducibility 
>>> for each qt5 commit [2].
>
> Regarding Qt CI @Toni Saario is the best one to answer on this, though I 
> haven't heard
> if there are plans to use Conan more extensively by CI/provisioning at this 
> point?

>From my experience, when the way project is built in CI is radically different 
>from a way suggested to
the end user, it inevitably leads to troubles. Users hit unexpected errors (in 
this case they may be
caused by changes in 3rd party Conan recipes/packages, or by contributors to Qt 
module which
don't use Conan), and they come back to me asking for solution.

As Iikka said that there has not been plans to use Conan when provisioning the 
images for CI.

However there has been some discussions how to make the provisioning easier to 
manage and have some documentability in them. Currently the provisioning is 
just a pile of scripts which have zero value for documentation tools. Easiest 
solution for this would be to list the dependencies on some other file in 
common format and then install listed dependencies with simple scripts. About 
using Conan there, I cannot really say anything about it with my knowledge 
about it.

Another thing is that there are many items built during provisioning which 
become dependencies, can we reduce the provisioning workload by having these 
cached in Conan storage? One thing to check is that how much the provisioning 
it is possible to translate to Conan, I would believe that in the end it does 
not work for everything.

As for provisioning and why the CI images are different from what are 
documented, biggest problem there is that the images are used by other 
components also e.g. RTA testing. Those require some additional stuff and 
provisioning currently installs everything. AFAIK also all the optional 
dependencies are installed, this makes it hard to distinguish between them and 
required dependencies because both are always present. These make it easy that 
new dependencies etc. slip through unnoticed.

-Toni

From: Konstantin Tokarev 
Sent: Friday, October 2, 2020 3:20 PM
To: Iikka Eklund ; Richard Weickelt ; 
development@qt-project.org ; Toni Saario 

Subject: Re: [Development] Building additional components with Conan for Qt 6



02.10.2020, 14:47, "Iikka Eklund" :
> Hi Konstantin, all,
>
>>>  thanks for the heads up. I have some further questions:
>>>
>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>>
>>> If this is the plan, note that there is infrastructure [1] for using Conan 
>>> in provisioning images,
>>> with verification of downloaded file checksums and build reproducibility 
>>> for each qt5 commit [2].
>
> Regarding Qt CI @Toni Saario is the best one to answer on this, though I 
> haven't heard
> if there are plans to use Conan more extensively by CI/provisioning at this 
> point?

>From my experience, when the way project is built in CI is radically different 
>from a way suggested to
the end user, it inevitably leads to troubles. Users hit unexpected errors (in 
this case they may be
caused by changes in 3rd party Conan recipes/packages, or by contributors to Qt 
module which
don't use Conan), and they come back to me asking for solution.

>
> What comes to add-ons 
> (https://doc-snapshots.qt.io/qt6-dev/qtmodules.html#qt-add-ons) as mentioned
> earlier the idea is to release add-ons as src packages accompanied with 
> conanfile.py i.e. the build recipe.
> If the add-on depends on 3rd party libraries which are available e.g. in 
> conan-center the recipe could
> naturally utilize packages from there (openssl, zlib, etc.).
>
>> 2. How will the recipes be managed and where on code.qt.io can I find 
>> them?
>
> The recipes of the first two add-ons are not part of the source repositories 
> yet, probably should be.
> Though there are some drawbacks in keeping the recipes and sources in the 
> same repo
> (some related discussion https://github.com/conan-io/conan/issues/4734).
>
> Currently the recipes are part of an internal packaging repository but the 
> current recipes can be found from the src
> installation directory: "6.0.0/Src/qtnetworkauth/conanfile.py" after you have 
> installed an add-on package.
>
> Iikka Eklund
> Senior Software Engineer
>
> iikka.ekl...@qt.io
> http://qt.io
>
> 
> From: Konstantin Tokarev 
> Sent: Friday, October 2, 2020 12:15 PM
> To: Richard Weickelt ; Iikka Eklund 
> ; development@qt-project.org 
> Subject: Spam: Re: [Development] Building additional components with Conan 
> for Qt 6
>
> 01.10.2020, 16:16, "Konstantin Tokarev" :
>> 01.10.2020, 16:07, "Richard Weickelt" :
>>>  Hello Ilkka,
>>>
>>>  thanks for the heads up. I have some further questions:
>>>
>>>  1. 

Re: [Development] http://download.qt.io/new_archive/qt/

[Kai Köhne]

Hi,

We noticed the same. It seems a sync'ing issue, I created 
https://bugreports.qt.io/browse/QTQAINFRA-3944.

Kai

> -Ursprüngliche Nachricht-
> Von: Development  Im Auftrag von
> Konstantin Tokarev
> Gesendet: Freitag, 2. Oktober 2020 14:41
> An: Qt Project Development Mailing-List 
> Betreff: [Development] http://download.qt.io/new_archive/qt/
> 
> Hello,
> 
> Seems like a lot of files were removed today from
> http://download.qt.io/new_archive/qt/
> Was it intentional? Are these archives available somewhere else?
> 
> --
> Regards,
> Konstantin
> 
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] http://download.qt.io/new_archive/qt/

[Konstantin Tokarev]

Hello,

Seems like a lot of files were removed today from 
http://download.qt.io/new_archive/qt/
Was it intentional? Are these archives available somewhere else?

-- 
Regards,
Konstantin

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Building additional components with Conan for Qt 6

[Konstantin Tokarev]

02.10.2020, 14:47, "Iikka Eklund" :
> Hi Konstantin, all,
>
>>>  thanks for the heads up. I have some further questions:
>>>
>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>>
>>> If this is the plan, note that there is infrastructure [1] for using Conan 
>>> in provisioning images,
>>> with verification of downloaded file checksums and build reproducibility 
>>> for each qt5 commit [2].
>
> Regarding Qt CI @Toni Saario is the best one to answer on this, though I 
> haven't heard
> if there are plans to use Conan more extensively by CI/provisioning at this 
> point?

From my experience, when the way project is built in CI is radically different 
from a way suggested to
the end user, it inevitably leads to troubles. Users hit unexpected errors (in 
this case they may be
caused by changes in 3rd party Conan recipes/packages, or by contributors to Qt 
module which
don't use Conan), and they come back to me asking for solution.

>
> What comes to add-ons 
> (https://doc-snapshots.qt.io/qt6-dev/qtmodules.html#qt-add-ons) as mentioned
> earlier the idea is to release add-ons as src packages accompanied with 
> conanfile.py i.e. the build recipe.
> If the add-on depends on 3rd party libraries which are available e.g. in 
> conan-center the recipe could
> naturally utilize packages from there (openssl, zlib, etc.).
>
>> 2. How will the recipes be managed and where on code.qt.io can I find 
>>them?
>
> The recipes of the first two add-ons are not part of the source repositories 
> yet, probably should be.
> Though there are some drawbacks in keeping the recipes and sources in the 
> same repo
> (some related discussion https://github.com/conan-io/conan/issues/4734).
>
> Currently the recipes are part of an internal packaging repository but the 
> current recipes can be found from the src
> installation directory: "6.0.0/Src/qtnetworkauth/conanfile.py" after you have 
> installed an add-on package.
>
> Iikka Eklund
> Senior Software Engineer
>
> iikka.ekl...@qt.io
> http://qt.io
>
> 
> From: Konstantin Tokarev 
> Sent: Friday, October 2, 2020 12:15 PM
> To: Richard Weickelt ; Iikka Eklund 
> ; development@qt-project.org 
> Subject: Spam: Re: [Development] Building additional components with Conan 
> for Qt 6
>
> 01.10.2020, 16:16, "Konstantin Tokarev" :
>> 01.10.2020, 16:07, "Richard Weickelt" :
>>>  Hello Ilkka,
>>>
>>>  thanks for the heads up. I have some further questions:
>>>
>>>  1. Will Conan be used to manage dependencies of Qt as well?
>>
>> If this is the plan, note that there is infrastructure [1] for using Conan 
>> in provisioning images,
>> with verification of downloaded file checksums and build reproducibility for 
>> each qt5 commit [2].
>>
>> If you are interested in details we could discuss them in depth in IRC.
>>
>> [1] 
>> https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/conan.ps1
>> [2] requires https://codereview.qt-project.org/c/qt/qt5/+/313930 to handle 
>> situations when the same package version is rebuilt without version bump
>
> Iikka, any comments?
>
> --
> Regards,
> Konstantin


-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] [Announce] Qt Creator 4.13.2 released

[List for announcements regarding Qt releases and development]

We are happy to announce the release of Qt Creator 4.13.2 !
https://www.qt.io/blog/qt-creator-4.13.2-released

-- 
Eike Ziller
Principal Software Engineer

The Qt Company GmbH
Erich-Thilo-Straße 10
D-12489 Berlin
eike.zil...@qt.io
http://qt.io
Geschäftsführer: Mika Pälsi,
Juha Varelius, Jouni Lintunen
Sitz der Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 
144331 B

___
Announce mailing list
annou...@qt-project.org
https://lists.qt-project.org/listinfo/announce
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Building additional components with Conan for Qt 6

[Iikka Eklund]

Hi Konstantin, all,

>>  thanks for the heads up. I have some further questions:
>>
>>  1. Will Conan be used to manage dependencies of Qt as well?
>>
>> If this is the plan, note that there is infrastructure [1] for using Conan 
>> in provisioning images,
>> with verification of downloaded file checksums and build reproducibility for 
>> each qt5 commit [2].

Regarding Qt CI @Toni Saario is the best one to 
answer on this, though I haven't heard
if there are plans to use Conan more extensively by CI/provisioning at this 
point?

What comes to add-ons 
(https://doc-snapshots.qt.io/qt6-dev/qtmodules.html#qt-add-ons) as mentioned
earlier the idea is to release add-ons as src packages accompanied with 
conanfile.py i.e. the build recipe.
If the add-on depends on 3rd party libraries which are available e.g. in 
conan-center the recipe could
naturally utilize packages from there (openssl, zlib, etc.).


> 2. How will the recipes be managed and where on code.qt.io can I find 
> them?

The recipes of the first two add-ons are not part of the source repositories 
yet, probably should be.
Though there are some drawbacks in keeping the recipes and sources in the same 
repo
(some related discussion https://github.com/conan-io/conan/issues/4734).

Currently the recipes are part of an internal packaging repository but the 
current recipes can be found from the src
installation directory: "6.0.0/Src/qtnetworkauth/conanfile.py" after you have 
installed an add-on package.





Iikka Eklund
Senior Software Engineer

iikka.ekl...@qt.io
http://qt.io



From: Konstantin Tokarev 
Sent: Friday, October 2, 2020 12:15 PM
To: Richard Weickelt ; Iikka Eklund ; 
development@qt-project.org 
Subject: Spam: Re: [Development] Building additional components with Conan for 
Qt 6



01.10.2020, 16:16, "Konstantin Tokarev" :
> 01.10.2020, 16:07, "Richard Weickelt" :
>>  Hello Ilkka,
>>
>>  thanks for the heads up. I have some further questions:
>>
>>  1. Will Conan be used to manage dependencies of Qt as well?
>
> If this is the plan, note that there is infrastructure [1] for using Conan in 
> provisioning images,
> with verification of downloaded file checksums and build reproducibility for 
> each qt5 commit [2].
>
> If you are interested in details we could discuss them in depth in IRC.
>
> [1] 
> https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/conan.ps1
> [2] requires https://codereview.qt-project.org/c/qt/qt5/+/313930 to handle 
> situations when the same package version is rebuilt without version bump

Iikka, any comments?

--
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Building additional components with Conan for Qt 6

[Konstantin Tokarev]

01.10.2020, 16:16, "Konstantin Tokarev" :
> 01.10.2020, 16:07, "Richard Weickelt" :
>>  Hello Ilkka,
>>
>>  thanks for the heads up. I have some further questions:
>>
>>  1. Will Conan be used to manage dependencies of Qt as well?
>
> If this is the plan, note that there is infrastructure [1] for using Conan in 
> provisioning images,
> with verification of downloaded file checksums and build reproducibility for 
> each qt5 commit [2].
>
> If you are interested in details we could discuss them in depth in IRC.
>
> [1] 
> https://code.qt.io/cgit/qt/qt5.git/tree/coin/provisioning/common/windows/conan.ps1
> [2] requires https://codereview.qt-project.org/c/qt/qt5/+/313930 to handle 
> situations when the same package version is rebuilt without version bump

Iikka, any comments?

-- 
Regards,
Konstantin
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

[Development] Qt Online Installer and Maintenance Tool 4.0 beta released - testers wanted

[Tino Pyssysalo]

Hi all,

We are happy to announce that Qt Online Installer and Maintenance Tool 4.0 beta 
have been released.

We are about one month away from the final release, so we invite all Qt users 
to start using and
testing the new installer.

It has completely renewed CLI experience, significant performance improvements 
in terms of meta-data
download speed. Also the UI has been modified here and there based on the user 
suggestions.

Please download the installer from 
http://download.qt.io/development_releases/online_installers/ and
start using it today. The installer uses the same package repos as Installer 
3.x, so it is possible to install standard
production packages with it.

We look forward to getting your feedback. You can find ready-made links and 
further information in a
related blog post in 
https://www.qt.io/blog/qt-online-installer-4.0-beta-released

Thanks for all contributors for this – especially the development team.
--
Tino Pyssysalo
Installer PO

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development