Hi, Thinking again about it. And trying to think generic enough to have a system that work for every RPM distribution of packages.
The base expectation for a RPM distribution is that all packages can be installed all together in the OS. This can be tested by installing all of them and check there is no error. But every changes proposed on a distgit repo need to be build and tested (-> installed with the packaged on the system) and if that pass we can approve and merge the change. Using this mechanic we know that the HEAD of each distgit repo are known to form a coherent and working distribution of RPM (pass the all together installation test). Then at a given time we want to want to release them in an identified (by a name/version) RPM distribution, the release. We know all HEAD works together so "snapshot" all the corresponding RPMs in a specific RPM release repo. In that case the RPM distribution is coherent because all packages install together (verified by a job) but we can have more complex job like in SF where we run complex scenario workflows. In a typical RPM distribution you have a bunch of distgit Git repositories and some source repositories (like the installer, the branding, the docs, ...) and we will want to test all that stuff together before merging any patches in the distgit or source repos. and the way to do that with Zuul is to host all of them in Gerrit and use the zuul-cloner to clone all of them (all distgit and source repos part of the RPM distro) in the job workspace. Basically you can build the RPM for the repo, install all of them and return a success of all install w/o issue. But it is smarter to only built those you never built. Having a master RPM repo make sense here "the pot commun" where previous built packages remain stored with the NVR such as: <name>-<sf-ver>-<source-(short-sha|tag)>-<distgit-short-sha>-<packaging-ver>.rpm can help us in the job by walking through all cloned repos which one has not been built previously. If some have not been built then build them and have them land in the master repo. Usually only the repo mentioned in the ZUUL_CHANGES will need a rebuild. Come the problem of dependencies that need to be resolved before we build each RPM. Dependencies will be between some packages part of the full RPM distribution we cloned via the zuul-cloner in the Job workspace. We can then walk through all repo's spec file and replace Require and BuildRequire section by the version we have in the workspace. Example: repo A spec: Require: B BuildRequire: B Source: D repo B spec: Require: C BuildRequire: C Source: E (external soft) repo C spec: Require: Nothing BuildRequire: Nothing repo D soft Repo A, B, C, D (part of our RPM distro have been cloned in the workspace) Require and BuildRequire sections can be changed for A, B, C and pin the real resulting package NVR. B has been checkouted with HEAD SHA such as "1234", A with "6767", D with "5678". The A spec file will be changed to target B-<sf-ver>-<source-E-tag>-1234-0.rpm for Build and BuildRequire and the Source section will need to target the tarball of D (the tarball need to be build and expose somewhere (managed by the job) and will we be name named A-<sf-ver>-5678-6767-0.rpm. This mechanics is applied across all repo cloned by zuul-cloner. So a script is needed here but I think it can be really generic. At this point we know how to prepare the spec files to build RPM by taking in account the Zuul context so then we can build them (via Koji for instance). But something need again to be handled. RPM builds need to occur in the right order to satisfied BuildRequire dependencies. But again I think we can deal with that in a generic way. When all RPM have been build we can do all tests we want \o/. And after if the tests succeed only changes mentioned in the ZUUL_CHANGES land in their respective Git repo. So we know that all our Distribution Git repos HEAD pass the CI :) When we want to release our distribution the RPM packages have been already built we just want to have a job to make the packages available in another RPM repo than master, a version named one. This job will go over all our Git distribution repo and detect HEAD for all of them then it compute the packages list that need to land in that new RPM repo, (maybe a last functional test may run (we never know)) and make them land. (The koji "add-pkg to tag" action if we use Koji). Cheers, Fabien Le 09/12/2016 à 11:00, Fabien Boucher a écrit : > Hi, > > I'm thinking about it since yesterday so and I wanted to share my thoughts > here. > > The idea is to have a sf-master target in Koji which is a "pot commun" where > all > packages built during the CI will land. We need to have a magic tool that know > how to manage rpm(s) build/runtime dependencies according to the ZUUL_CHANGES > variables passed by Zuul. > > Each build of a rpm need to have the Software commit hash (for the ones we > host and dev > on Gerrit) and the tag (for the other ones like Gerrit, Storyboard). Also I'm > thinking of a meta package called sf-<version>.rpm that contains all the > mandatories dependencies. > > For example: > * sf-3.0.0-1.rpm - depends on: > ** managesf-<hash>-1.rpm > ** managesf-ansible-<hash>-1.rpm > ** sf-ansible-<hash>-1.rpm > ** gerrit-2.X.X-1.rpm > ** gerrit-ansible-0.1-1.rpm > ** sf-docs-3.0.0-1.rpm > > This meta package is then the entry point to install SF and its mandatory > dependencies. > (Maybe it should also include a file with the list of extra component (such > repoxplorer, ...) > at the exact versions supported by this release of SF). We can even imagine > it contains > our reno notes. In this version of "dreamland" the SF Git repository should > only > contains the "(template) ?" .spec of this meta package. > > In the CI the meta package .spec file is modified according the build > context. For > example managesf is in the ZUUL_CHANGES then this meta package will be > rebuilt to pin > the freshly built version of managesf. > But doing that at this meta package level is not enough. For instance pysflib > is > modified then the managesf's build/runtime rpm deps need to changed to pin the > pysflib version. > > Here could be the resulting workflow of the CI to test an incoming change in > the SF CI: > We bump Gerrit: > 1 -> A change in the gerrit-distgit repo is proposed > 2 -> First Gerrit is built on koji (non-scratch) build and land in the "pot > commun" > 3 -> The meta package is rebuild and pin the new version of Gerrit > 4 -> The NVR of the meta package can maybe an epoch or the ZUUL_UUID ? > 5 -> The SF image is built/or updated (if a previous on exist on the > slave - Then having the epoch make sense) using the "pot commun" koji > repo > in /etc/yum.repo.d. > 6 -> Test the SF image as usual > 7 -> If a success (in the gate) the gerrit-distgit proposed changed lands in > the > Git repo. > > Building a master SF for our test env could be then : only install the "pot > commun" > repo in yum.repo.d and yum install sf[-<epoch>]. Note that as the meta > package sf use the epoch > as NVR so theoretically you could just yum install sf from the "pot commum" > but as > in the flow I described the meta package sf is built for each patchset then > we won't > have the guarantie the latest sf-<epoch>.rpm is a working version of SF :(. > > Working on a dev env (let's say on managesf again) could be then as follow: > -> Make you change locally in managesf source repo > -> Build the SRPM > -> Send it to koji (in scratch or non scratch) whatever > -> Fetch the artifacts > -> and install them in the devel SF > -> When your are satisfied with your changes -> propose them > on Gerrit and the CI will re-test your changes as usual. > > Additional note : We want to be sure at any time that all master branch > of each repo[-distgit] that are part of the SF distribution are working > together (pass the SF tests). > > What about the SF release ? Let's say now we want to release sf-3.0.0. > Then we will tag (in the koji terminology) > https://fedoraproject.org/wiki/Koji#Package_Organization > a specific list of built packages. We tag them with the tag name sf-3.0.0. We > do > the same when we want to release sf-3.0.1 and so on. > So each stable release of the SF (Distribution) will have its own RPM repo. > -> I'm not sure about that and need to be experimented ... > > Let's discuss about that and raise questions and issues. > I propose also to setup a semi-official Koji for our needs and we could start > using it > to experiment. > > Cheers, > Fabien > > > > _______________________________________________ > Softwarefactory-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/softwarefactory-dev >
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
