Le 09/12/2016 à 12:39, Fabien Boucher a écrit : > > > Le 09/12/2016 à 12:05, Matthieu Huin a écrit : >> Some additional shower thoughts: >> >> I'm thinking we could add a "smoke" pipeline prior to the "check" one. It >> would launch one job running unit tests; >> if they pass then the package building occurs and a publisher job pushes it >> to swift or a repo with a guessable URI >> (for example computed from the commit sha). The review gets a >> "smoke-verified +1" or something, which will trigger >> the proper functional tests where the test env will be installed using the >> RPM(s) generated by the smoke pipeline. >> >> I believe this would make testing and getting feedback much faster. WDYT? > > Yes. Having the unittest run + packaging (with unittest run again during the > packaging) in a first step can > be a good idea. There is a way to do that via a pipeline or even via a parent > task in Zuul. >
Another workflow inspired by your proposition: Each software we package whatever it is internal to our Gerrit or not have a <name>-distgit as for the other proposal but we keep the job attached to their changes quite simple for instance (with a limited depends-on support): - rpm-build (include unittest run via the test rpm section) (scratch build) - rpm-lint - smoke tests (if relevant (not for libs for instance)) For sources we managed like pysflib, managesf, cauth [1] the job attached are for instance: - unittest (tox) - rpm-build (include unittest run via the test rpm section) (scratch build) - rpm-lint - smoke tests (but standalone (w/o the whole sf deployed) We keep a model of check and gate pipeline (scratch build in check, non-scratch build in gate). At this level we support the ZUUL_CHANGES between only: - [source-x, source-x-distgit,] source-x-ansible, source-x-ansible-distgit. In case of any change in the source-x that require changes in the packaging and configuration. And we have a at the top of that a sf repo with just two files: sf.spec and a release.yaml file: * sf.spec contains the instructions to install mandatory sf packages such as sf-ansible, gerrit-*, cauth-* ... * the release file list the full packages list of a SF release with packages pinned. packages such as storyboard-*, etherpad-*, repoxplorer-* and their deps will be listed here. At this sf repo level we support dependencies to every -distgit repo and we run the SF functional tests at this level only: - sf-functional-test - sf-upgrade-test - sf-backup-restore-test [1]: some other will be cauth-ansible, gerrit-ansible, storyboard-ansible ... and a bunch of others. Cheers, Fabien >> ----- Original Message ----- >>> From: "Matthieu Huin" <[email protected]> >>> To: "Fabien Boucher" <[email protected]> >>> Cc: [email protected] >>> Sent: Friday, December 9, 2016 11:19:08 AM >>> Subject: Re: [Softwarefactory-dev] CI/Release workflow for a RPM >>> packaged version of SF >>> >>> >>> >>> ----- Original Message ----- >>>> From: "Fabien Boucher" <[email protected]> >>>> To: [email protected] >>>> Sent: Friday, December 9, 2016 11:00:55 AM >>>> Subject: [Softwarefactory-dev] CI/Release workflow for a RPM packaged >>>> version of SF >>>> >>>> Hi, >>>> >>>> I'm thinking about it since yesterday so and I wanted to share my thoughts >>>> here. >>>> >>>> The idea is to have a sf-master target in Koji which is a "pot commun" >>>> where >>>> all >>>> packages built during the CI will land. We need to have a magic tool that >>>> know >>>> how to manage rpm(s) build/runtime dependencies according to the >>>> ZUUL_CHANGES >>>> variables passed by Zuul. >>>> >>>> Each build of a rpm need to have the Software commit hash (for the ones we >>>> host and dev >>>> on Gerrit) and the tag (for the other ones like Gerrit, Storyboard). Also >>>> I'm >>>> thinking of a meta package called sf-<version>.rpm that contains all the >>>> mandatories dependencies. >>>> >>>> For example: >>>> * sf-3.0.0-1.rpm - depends on: >>>> ** managesf-<hash>-1.rpm >>>> ** managesf-ansible-<hash>-1.rpm >>>> ** sf-ansible-<hash>-1.rpm >>>> ** gerrit-2.X.X-1.rpm >>>> ** gerrit-ansible-0.1-1.rpm >>>> ** sf-docs-3.0.0-1.rpm >>>> >>>> This meta package is then the entry point to install SF and its mandatory >>>> dependencies. >>>> (Maybe it should also include a file with the list of extra component (such >>>> repoxplorer, ...) >>>> at the exact versions supported by this release of SF). We can even imagine >>>> it contains >>>> our reno notes. In this version of "dreamland" the SF Git repository should >>>> only >>>> contains the "(template) ?" .spec of this meta package. >>>> >>>> In the CI the meta package .spec file is modified according the build >>>> context. For >>>> example managesf is in the ZUUL_CHANGES then this meta package will be >>>> rebuilt to pin >>>> the freshly built version of managesf. >>>> But doing that at this meta package level is not enough. For instance >>>> pysflib >>>> is >>>> modified then the managesf's build/runtime rpm deps need to changed to pin >>>> the >>>> pysflib version. >>>> >>>> Here could be the resulting workflow of the CI to test an incoming change >>>> in >>>> the SF CI: >>>> We bump Gerrit: >>>> 1 -> A change in the gerrit-distgit repo is proposed >>>> 2 -> First Gerrit is built on koji (non-scratch) build and land in the "pot >>>> commun" >>>> 3 -> The meta package is rebuild and pin the new version of Gerrit >>>> 4 -> The NVR of the meta package can maybe an epoch or the ZUUL_UUID ? >>>> 5 -> The SF image is built/or updated (if a previous on exist on the >>>> slave - Then having the epoch make sense) using the "pot commun" koji >>>> repo >>>> in /etc/yum.repo.d. >>>> 6 -> Test the SF image as usual >>>> 7 -> If a success (in the gate) the gerrit-distgit proposed changed lands >>>> in >>>> the >>>> Git repo. >>> >>> As you mentioned above, how do you deal with dependencies between components >>> ? >>> I'm thinking of a common use case whenever we add smth in the manageSF API, >>> we >>> usually have to change the manageSF config template in SF; both changes land >>> semi-simultaneously. Thus our packaging workflow should be aware of >>> depends-on >>> tags to reflect that dependency in the build. >>> >>>> Building a master SF for our test env could be then : only install the "pot >>>> commun" >>>> repo in yum.repo.d and yum install sf[-<epoch>]. Note that as the meta >>>> package sf use the epoch >>>> as NVR so theoretically you could just yum install sf from the "pot commum" >>>> but as >>>> in the flow I described the meta package sf is built for each patchset then >>>> we won't >>>> have the guarantie the latest sf-<epoch>.rpm is a working version of SF :(. >>>> >>>> Working on a dev env (let's say on managesf again) could be then as follow: >>>> -> Make you change locally in managesf source repo >>>> -> Build the SRPM >>>> -> Send it to koji (in scratch or non scratch) whatever >>> >>> Could the rpm building be done locally on the dev env, with a local dev >>> repo? >>> Might be faster and easier. >>> >>>> -> Fetch the artifacts >>>> -> and install them in the devel SF >>> >>> Wouldn't that be broken if dependencies versions are pinned ? ie you can't >>> install a newer version of this package because of dep pinning. >>> >>>> -> When your are satisfied with your changes -> propose them >>>> on Gerrit and the CI will re-test your changes as usual. >>> >>>> Additional note : We want to be sure at any time that all master branch >>>> of each repo[-distgit] that are part of the SF distribution are working >>>> together (pass the SF tests). >>>> >>>> What about the SF release ? Let's say now we want to release sf-3.0.0. >>>> Then we will tag (in the koji terminology) >>>> https://fedoraproject.org/wiki/Koji#Package_Organization >>>> a specific list of built packages. We tag them with the tag name sf-3.0.0. >>>> We >>>> do >>>> the same when we want to release sf-3.0.1 and so on. >>>> So each stable release of the SF (Distribution) will have its own RPM repo. >>>> -> I'm not sure about that and need to be experimented ... >>>> >>>> Let's discuss about that and raise questions and issues. >>>> I propose also to setup a semi-official Koji for our needs and we could >>>> start >>>> using it >>>> to experiment. >>>> >>>> Cheers, >>>> Fabien >>>> >>>> >>>> _______________________________________________ >>>> Softwarefactory-dev mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/softwarefactory-dev >>>> >>> >>> _______________________________________________ >>> Softwarefactory-dev mailing list >>> [email protected] >>> https://www.redhat.com/mailman/listinfo/softwarefactory-dev >>> > > > > _______________________________________________ > Softwarefactory-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/softwarefactory-dev >
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
