Re: [Development] New Coin feature: automatic provisioning
On Wednesday 27 of July 2016 13:58:44 Dominik Holland wrote: > Hi, > > Am 07/27/2016 um 01:47 PM schrieb Jędrzej Nowacki: > > Hi, > > > > Good news everyone! Last month we managed to deploy a new automatic > > > > provisioning system to Coin. What does it mean to you? Now you can modify > > existing VM templates used by Coin. Installing additional packages should > > be trivial and more or less safe from now. > > > > I. The goal: > > - Coin should use VMs with vanilla OS installations (aka base template) > > - Anything that needs to be installed in addition should be documented, > > > > reproducible in an automatic way (create a provisioned template) > > > > - templates may be only modified in such way that the _whole_ Qt still > > works> > > on a VM create from it > > > > - It should not be required to log in to a VM to modify it > > > > II. So how it works now: > > Qt5 is our product repository. The product repository contains directory > > > > called "coin/provisioning", which contains provisioning scripts used > > against base templates to produce provisioned templates. When an > > integration starts on a certain platform, it checks for the directory. If > > base platform template name matches one of the subdirectories then all > > scripts in the subdirectory are used to provision the new template. Of > > course the result is cached and re- used for all integrations that needs > > it. > > > > III. Qt Developer oriented example: > > We need to install FooBar on Ubuntu 14.04 > > > > 1. Prepare bash script that installs it: > > #!/bin/env bash > > # Package FooBar is need to ... and can be safely removed if ... > > apt install FooBar > > > > 2. Place it in the product repository (Qt5) in > > coin/provisioning/qtci-linux-> > > Ubuntu-14.04-x86_64/ under foobar.sh name. > > > > 3. Commit it to the right branch (5.6..dev) depending where the package > > is > > > > needed > > > > 4. Get through the review, by default we do __not__ want any additional > > > > software on VMs, so you need to have a good reason to install something > > > > 5. Stage the change together with a recent submodule update > > 6. If it pass then you can enjoy FooBar in your code > > 7. Follow-up on Qt5 merges, to ensure that FooBar is installed in right > > > > branches and only in them > > > > Be aware that provisioning scripts are tested against Qt5 state while > > newly > > > > provisioned templates are used for all integrations. So technically it is > > possible to break stuff if Qt5 is old or if a breakage was integrated > > during Qt5 testing. In reality I do not think it would affect anyone, but > > that is why in point 5 I recommend to staging together with a recent > > submodule update.> > > Currently only bash and powershell are supported. > > > > If you want to share a script between different platforms (for example > > > > Ubuntu 14.04, and Ubuntu 16.04) you place it in > > "coin/provisioning/common". > > Then you can call it from any other "coin/provisioning" subdirectory. > > > > The current naming convention for base templates is suboptimal and it > > will > > > > be changed in a reasonable future, but you should not worry about that, > > the > > process will be transparent (TM). > > > > Cheers, > > > > Jędrek > > > > ps. The feature consists of a few dozen patches and the probability says > > that there is at least one bug, so keep your eyes open. > > great news ! > > Does the provisioning only work for the qt5 repository or does it also > work for each of the modules ? > > E.g. Could there be a provisioning script in qtmultimedia, installing > some of the dependencies e.g. gstreamer1.0-dev ? > > Dominik > Qt5 repository contains definition for all modules. All modules shares the same set of machines(*). There is no direct way to install something just for one module. Technically you could install something to a custom place and export a environment variable, which would be used only by one module. I advice against that. The feature is useful for installing software that is already shipped by a distribution but just not installed by default. Regarding gstreamer1.0, the problem is that in RHEL 6 it doesn't exist at all therefore it would need to be compiled from sources and then we would need to ship gstreamer binaries... So, no the feature do not solve qstreamer case :( Cheers, Jędrek *there are tricks to opt-out and opt-in, they but should be avoided ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] New Coin feature: automatic provisioning
Hi, Am 07/27/2016 um 01:47 PM schrieb Jędrzej Nowacki: > Hi, > > Good news everyone! Last month we managed to deploy a new automatic > provisioning system to Coin. What does it mean to you? Now you can modify > existing VM templates used by Coin. Installing additional packages should be > trivial and more or less safe from now. > > I. The goal: > - Coin should use VMs with vanilla OS installations (aka base template) > - Anything that needs to be installed in addition should be documented, > reproducible in an automatic way (create a provisioned template) > - templates may be only modified in such way that the _whole_ Qt still > works > on a VM create from it > - It should not be required to log in to a VM to modify it > > II. So how it works now: > Qt5 is our product repository. The product repository contains directory > called "coin/provisioning", which contains provisioning scripts used against > base templates to produce provisioned templates. When an integration starts > on > a certain platform, it checks for the directory. If base platform template > name matches one of the subdirectories then all scripts in the subdirectory > are used to provision the new template. Of course the result is cached and re- > used for all integrations that needs it. > > III. Qt Developer oriented example: > We need to install FooBar on Ubuntu 14.04 > 1. Prepare bash script that installs it: > #!/bin/env bash > # Package FooBar is need to ... and can be safely removed if ... > apt install FooBar > 2. Place it in the product repository (Qt5) in coin/provisioning/qtci-linux- > Ubuntu-14.04-x86_64/ under foobar.sh name. > 3. Commit it to the right branch (5.6..dev) depending where the package is > needed > 4. Get through the review, by default we do __not__ want any additional > software on VMs, so you need to have a good reason to install something > 5. Stage the change together with a recent submodule update > 6. If it pass then you can enjoy FooBar in your code > 7. Follow-up on Qt5 merges, to ensure that FooBar is installed in right > branches and only in them > > Be aware that provisioning scripts are tested against Qt5 state while newly > provisioned templates are used for all integrations. So technically it is > possible to break stuff if Qt5 is old or if a breakage was integrated during > Qt5 testing. In reality I do not think it would affect anyone, but that is > why > in point 5 I recommend to staging together with a recent submodule update. > > Currently only bash and powershell are supported. > > If you want to share a script between different platforms (for example > Ubuntu 14.04, and Ubuntu 16.04) you place it in "coin/provisioning/common". > Then you can call it from any other "coin/provisioning" subdirectory. > > The current naming convention for base templates is suboptimal and it will > be changed in a reasonable future, but you should not worry about that, the > process will be transparent (TM). > > Cheers, > Jędrek > > ps. The feature consists of a few dozen patches and the probability says that > there is at least one bug, so keep your eyes open. great news ! Does the provisioning only work for the qt5 repository or does it also work for each of the modules ? E.g. Could there be a provisioning script in qtmultimedia, installing some of the dependencies e.g. gstreamer1.0-dev ? Dominik > > ___ > Development mailing list > Development@qt-project.org > http://lists.qt-project.org/mailman/listinfo/development > -- Dominik Holland SENIOR SOFTWARE ENGINEER Pelagicore AG Balanstr. 55, 81541 Munich, Germany +49 (0)171 760 25 96 dominik.holl...@pelagicore.com www.pelagicore.com ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] New Coin feature: automatic provisioning
Hi, Good news everyone! Last month we managed to deploy a new automatic provisioning system to Coin. What does it mean to you? Now you can modify existing VM templates used by Coin. Installing additional packages should be trivial and more or less safe from now. I. The goal: - Coin should use VMs with vanilla OS installations (aka base template) - Anything that needs to be installed in addition should be documented, reproducible in an automatic way (create a provisioned template) - templates may be only modified in such way that the _whole_ Qt still works on a VM create from it - It should not be required to log in to a VM to modify it II. So how it works now: Qt5 is our product repository. The product repository contains directory called "coin/provisioning", which contains provisioning scripts used against base templates to produce provisioned templates. When an integration starts on a certain platform, it checks for the directory. If base platform template name matches one of the subdirectories then all scripts in the subdirectory are used to provision the new template. Of course the result is cached and re- used for all integrations that needs it. III. Qt Developer oriented example: We need to install FooBar on Ubuntu 14.04 1. Prepare bash script that installs it: #!/bin/env bash # Package FooBar is need to ... and can be safely removed if ... apt install FooBar 2. Place it in the product repository (Qt5) in coin/provisioning/qtci-linux- Ubuntu-14.04-x86_64/ under foobar.sh name. 3. Commit it to the right branch (5.6..dev) depending where the package is needed 4. Get through the review, by default we do __not__ want any additional software on VMs, so you need to have a good reason to install something 5. Stage the change together with a recent submodule update 6. If it pass then you can enjoy FooBar in your code 7. Follow-up on Qt5 merges, to ensure that FooBar is installed in right branches and only in them Be aware that provisioning scripts are tested against Qt5 state while newly provisioned templates are used for all integrations. So technically it is possible to break stuff if Qt5 is old or if a breakage was integrated during Qt5 testing. In reality I do not think it would affect anyone, but that is why in point 5 I recommend to staging together with a recent submodule update. Currently only bash and powershell are supported. If you want to share a script between different platforms (for example Ubuntu 14.04, and Ubuntu 16.04) you place it in "coin/provisioning/common". Then you can call it from any other "coin/provisioning" subdirectory. The current naming convention for base templates is suboptimal and it will be changed in a reasonable future, but you should not worry about that, the process will be transparent (TM). Cheers, Jędrek ps. The feature consists of a few dozen patches and the probability says that there is at least one bug, so keep your eyes open. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
