On Wed, Jul 27, 2016 at 02:54:00PM +0100, Derek Higgins wrote: > On 21 July 2016 at 23:04, Paul Belanger <[email protected]> wrote: > > Greetings, > > > > I write today to see how I can remove this server from > > tripleo-test-cloud-rh2. I > > have an open patch[1] currently to migrate tripleo-ci to use our AFS > > mirrors for > > centos and epel. However, I'm still struggling to see what else you are > > using > > the local mirror for. > > > > From what I see, there appears to be some puppet modules in the mirror? > > > > The reason I am doing this work, is to help bring tripleo inline with > > openstack-infra tooling. There shouldn't be the need for a project to > > maintain > > its own infrastructure outside of openstack-infra. If so, I see that as > > some > > sort of a failure between the project and openstack-infra. And with that > > in > > mind, I am here to help fix that. > > > > For the most part, I think we have everything currently in place to migrate > > away > > from your locally mirror. I just need some help figuring what else is left > > and > > then delete it. > > Hi Paul, > The mirror server hosts 3 sets of data used in CI long with a cron > a job aimed at promoting trunk repositories, > The first you've already mentioned, there is a list of puppet modules > hosted here, we soon hope to move to packaged puppet modules so the > need for this will go away. > Ya, I was looking at an open review to rework this. If we moved these puppet modules to tarballs over git repos, I think we could mirror them pretty easy into our AFS mirrors. Them being git repos requires more work because some policies around git repos.
> The second is a mirror of the centos cloud images, these are updated > hourly by the centos-cloud-images cronjob[1], I guess these could be > easily replaced with the AFS server > So 2 things here. 1) I've reached out to CentOS asking to enable rsync support on http://cloud.centos.org/ if they do that, I can easily enable rsync for it. 2) What about moving away from the centos diskimage-builder element and switch to centos-minimal element. I have an open review for this, but need help on actually testing this. It moves away from using the cloud image, and instead uses yumdownloader to prebuild the images. > Then we come to the parts where it will probably be more tricky to > move away from our own server > > o cached images - our nightly periodic jobs run tripleo ci with > master/HEAD for all openstack projects (using the most recent rdo > trunk repository), if the jobs pass then we upload the overcloud-full > and ipa images to the mirror server along with logging what jobs > passed, this happens at the end of toci_instack.sh[2], nothing else > happens at this point the files are just uploaded nothing starts using > them yet. > I suggest we move this to tarballs.o.o for now, this is what other projects are doing. I believe we are also considering moving this process into AFS too. > o promote script - hourly we then run the promote script[3], this > script is whats responsible for the promotion of the master rdo > repository that is used by tripleo ci (and devs), it checks to see if > images have been updated to the mirror server by the periodic jobs, > and if all of the jobs we care about (currently > periodic-tripleo-ci-centos-7-ovb-ha > periodic-tripleo-ci-centos-7-ovb-nonha[4]) passed then it does 2 > things > 1. updates the current-tripleo link on the mirror server[5] > 2. updates the current-tripleo link on the rdo trunk server[6] > By doing this we ensure that the the current-tripleo link on the rdo > trunk server is always pointing to something that has passed tripleo > ci jobs, and that tripleo ci is using cached images that were built > using this repository > Okay, I think we need to dive more into this. It might be possible to make this a post job or use mirror-update.openstack.org > We've had to run this promote script on the mirror server as the > individual jobs run independently and in oder to make the promote > decision we needed somewhere that is aware of the status of all the > jobs > > Hope this answers your questions, > Derek. > > [1] - > http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n40 > [2] - > http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_instack.sh#n198 > [3] - > http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/promote.sh > [4] - > http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n51 > [5] - http://8.43.87.241/builds/current-tripleo/ > [6] - > http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/ > > > > > [1] https://review.openstack.org/#/c/326143/ > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: [email protected]?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
