Some additional shower thoughts: I'm thinking we could add a "smoke" pipeline prior to the "check" one. It would launch one job running unit tests; if they pass then the package building occurs and a publisher job pushes it to swift or a repo with a guessable URI (for example computed from the commit sha). The review gets a "smoke-verified +1" or something, which will trigger the proper functional tests where the test env will be installed using the RPM(s) generated by the smoke pipeline.
I believe this would make testing and getting feedback much faster. WDYT? ----- Original Message ----- > From: "Matthieu Huin" <[email protected]> > To: "Fabien Boucher" <[email protected]> > Cc: [email protected] > Sent: Friday, December 9, 2016 11:19:08 AM > Subject: Re: [Softwarefactory-dev] CI/Release workflow for a RPM > packaged version of SF > > > > ----- Original Message ----- > > From: "Fabien Boucher" <[email protected]> > > To: [email protected] > > Sent: Friday, December 9, 2016 11:00:55 AM > > Subject: [Softwarefactory-dev] CI/Release workflow for a RPM packaged > > version of SF > > > > Hi, > > > > I'm thinking about it since yesterday so and I wanted to share my thoughts > > here. > > > > The idea is to have a sf-master target in Koji which is a "pot commun" > > where > > all > > packages built during the CI will land. We need to have a magic tool that > > know > > how to manage rpm(s) build/runtime dependencies according to the > > ZUUL_CHANGES > > variables passed by Zuul. > > > > Each build of a rpm need to have the Software commit hash (for the ones we > > host and dev > > on Gerrit) and the tag (for the other ones like Gerrit, Storyboard). Also > > I'm > > thinking of a meta package called sf-<version>.rpm that contains all the > > mandatories dependencies. > > > > For example: > > * sf-3.0.0-1.rpm - depends on: > > ** managesf-<hash>-1.rpm > > ** managesf-ansible-<hash>-1.rpm > > ** sf-ansible-<hash>-1.rpm > > ** gerrit-2.X.X-1.rpm > > ** gerrit-ansible-0.1-1.rpm > > ** sf-docs-3.0.0-1.rpm > > > > This meta package is then the entry point to install SF and its mandatory > > dependencies. > > (Maybe it should also include a file with the list of extra component (such > > repoxplorer, ...) > > at the exact versions supported by this release of SF). We can even imagine > > it contains > > our reno notes. In this version of "dreamland" the SF Git repository should > > only > > contains the "(template) ?" .spec of this meta package. > > > > In the CI the meta package .spec file is modified according the build > > context. For > > example managesf is in the ZUUL_CHANGES then this meta package will be > > rebuilt to pin > > the freshly built version of managesf. > > But doing that at this meta package level is not enough. For instance > > pysflib > > is > > modified then the managesf's build/runtime rpm deps need to changed to pin > > the > > pysflib version. > > > > Here could be the resulting workflow of the CI to test an incoming change > > in > > the SF CI: > > We bump Gerrit: > > 1 -> A change in the gerrit-distgit repo is proposed > > 2 -> First Gerrit is built on koji (non-scratch) build and land in the "pot > > commun" > > 3 -> The meta package is rebuild and pin the new version of Gerrit > > 4 -> The NVR of the meta package can maybe an epoch or the ZUUL_UUID ? > > 5 -> The SF image is built/or updated (if a previous on exist on the > > slave - Then having the epoch make sense) using the "pot commun" koji > > repo > > in /etc/yum.repo.d. > > 6 -> Test the SF image as usual > > 7 -> If a success (in the gate) the gerrit-distgit proposed changed lands > > in > > the > > Git repo. > > As you mentioned above, how do you deal with dependencies between components > ? > I'm thinking of a common use case whenever we add smth in the manageSF API, > we > usually have to change the manageSF config template in SF; both changes land > semi-simultaneously. Thus our packaging workflow should be aware of > depends-on > tags to reflect that dependency in the build. > > > Building a master SF for our test env could be then : only install the "pot > > commun" > > repo in yum.repo.d and yum install sf[-<epoch>]. Note that as the meta > > package sf use the epoch > > as NVR so theoretically you could just yum install sf from the "pot commum" > > but as > > in the flow I described the meta package sf is built for each patchset then > > we won't > > have the guarantie the latest sf-<epoch>.rpm is a working version of SF :(. > > > > Working on a dev env (let's say on managesf again) could be then as follow: > > -> Make you change locally in managesf source repo > > -> Build the SRPM > > -> Send it to koji (in scratch or non scratch) whatever > > Could the rpm building be done locally on the dev env, with a local dev repo? > Might be faster and easier. > > > -> Fetch the artifacts > > -> and install them in the devel SF > > Wouldn't that be broken if dependencies versions are pinned ? ie you can't > install a newer version of this package because of dep pinning. > > > -> When your are satisfied with your changes -> propose them > > on Gerrit and the CI will re-test your changes as usual. > > > Additional note : We want to be sure at any time that all master branch > > of each repo[-distgit] that are part of the SF distribution are working > > together (pass the SF tests). > > > > What about the SF release ? Let's say now we want to release sf-3.0.0. > > Then we will tag (in the koji terminology) > > https://fedoraproject.org/wiki/Koji#Package_Organization > > a specific list of built packages. We tag them with the tag name sf-3.0.0. > > We > > do > > the same when we want to release sf-3.0.1 and so on. > > So each stable release of the SF (Distribution) will have its own RPM repo. > > -> I'm not sure about that and need to be experimented ... > > > > Let's discuss about that and raise questions and issues. > > I propose also to setup a semi-official Koji for our needs and we could > > start > > using it > > to experiment. > > > > Cheers, > > Fabien > > > > > > _______________________________________________ > > Softwarefactory-dev mailing list > > [email protected] > > https://www.redhat.com/mailman/listinfo/softwarefactory-dev > > > > _______________________________________________ > Softwarefactory-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/softwarefactory-dev > _______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
