On Tue, 2012-07-31 at 14:02 -0400, Robert Middleswarth wrote: > On 07/31/2012 11:03 AM, Mike Burns wrote: > > On Tue, 2012-07-31 at 09:13 -0400, Robert Middleswarth wrote: > >> On 07/31/2012 08:51 AM, Eyal Edri wrote: > >>> ----- Original Message ----- > >>>> From: "Mike Burns" <[email protected]> > >>>> To: "Eyal Edri" <[email protected]> > >>>> Cc: "infra" <[email protected]> > >>>> Sent: Tuesday, July 31, 2012 3:33:01 PM > >>>> Subject: Re: how to get files from jenkins.ovirt.org to www.ovirt.org > >>>> > >>>> On Tue, 2012-07-31 at 02:46 -0400, Eyal Edri wrote: > >>>>> Hi, > >>>>> > >>>>> after looking into the process and testing it via > >>>>> jenkins.ovirt.info, > >>>>> here is my proposal: > >>>>> > >>>>> 1. Nightly job [1] will collect rpms from all ovirt projects (other > >>>>> jenkins jobs that build rpms per commit). > >>>>> 1.1 all files will be kept in a flat dir under $WORKSPCE/rpms > >>>>> 1.2 using 'publish over ssh' [2] plugin the job will then copy > >>>>> files to their destination according to their names. [3] > >>>>> for e.g 'ovirt-engine-3.1.0-3.fc17.noarch.rpm' ==> > >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/noarch > >>>>> 'ovirt-engine-3.1.0-3.fc17.src.rpm' ==> > >>>>> ovirt.org/var/www/html/releases/nightly/rpm/Fedora/17/src > >>>>> 'ovirt-engine-3.1.0-3.bz2' ==> > >>>>> ovirt.org/var/www/html/releases/nightly/src' > >>>>> 1.3 publish ssh plugin can execute a command after its done > >>>>> copying > >>>>> the file, so all is left is to run 'createrepo' on the relevant > >>>>> dirs. > >>>>> [4] > >>>>> > >>>>> > >>>>> 2. a separate cron cleaning script needs to run on ovirt.org to > >>>>> delete > >>>>> old rpms and run createrepo. > >>>>> > >>>> Maybe upload to a non-published directory first, > >>>> like /home/jenkins/nightly, and have the cron job handle cleanup of > >>>> old > >>>> rpms, adding of new rpms, and running createrepo? > >>> sure, i've got not problem with that.. > >>> > >>> there is one thing i don't understand, and sorry if not an expert in yum: > >>> > >>> we say we want to keep history for X builds, will all versions be kept > >>> under one repository? > >>> so 'createrepo' will run on all versions in that directory? > >>> > >>> so running yum install file.verX will work even if the repo has a x+1 > >>> version? > >> If you look at existing repo's many of them will include old versions of > >> the package yum / create repo will simple use the latest version in the > >> repo that meets all the requirements. Example when we first moved from > >> vdsm 4.9 to 4.10 the repo's got really screwy until all the packages > >> were rebuilt. In that case the older 4.9 builds would get used until > >> all the packages were in place then the newer versions would get used. > > > > If we still want to keep multiple night's versions, then we can have the > > cron job run and create a new directory called: > > > > cd /var/www/html/releases > > mkdir nightly-$(date +%Y%m%d) > > # move all content from /home/jenkins to new nightly-<date> directory > > ln -snf nightly-$(date +%Y%m%d) nightly > > rm -rf nightly-$(date -d "3 days ago" +%Y%m%d) > > > > Mike > Why recreate the wheel. www.hyperdrifter.com/software/tidy_rpm_cache.html > > Example usages: tidy-rpm-cache.py --dir=/tmp/packages --num-obsolete=3
Works for me... Mike > > Thanks > Robert > > > >>>> If not, how is the cron job going to cleanup old rpms? > >>>> > >>>> Uploading to a static location that isn't public (and is > >>>> under /home/jenkins) would make [3] and [4] unnecessary. I think I'd > >>>> rather avoid having a non-user with sudo and r/w on a publicly served > >>>> folder. > >>>> > >>> sounds right. > >>> > >>>> The publish ssh plugin can touch a file > >>>> /home/jenkins/nightly/FINISHED > >>>> when it's done, the cron job locally on ovirt.org can watch for that > >>>> file and then process the directory structure. > >> +1 This is the approach I was thinking about. > >>> good idea, we can have the cronjob monitor the dir for the FINISHED file, > >>> and delete it when it's done. > >>> > >>>> Mike > >>>> > >>>>> +1 for pushing this forward or comments? > >>>>> > >>>>> Eyal. > >>>>> > >>>>> > >>>>> [1] > >>>>> http://jenkins.ovirt.org/view/rpms/job/publish_ovirt_rpms_nightly/, > >>>>> this job currently works only for fedora17 rpms (that's what we > >>>>> have > >>>>> now). > >>>>> we'll need to improve it to be more generic (maybe as a matrix > >>>>> job) or duplicate it for each opertating system. > >>>>> [2] https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH > >>>>> +Plugin, using pub/private key with user jenkins to connect > >>>>> [3] jenkins user will need r/q permissions to the nightly folder > >>>>> [4] will need sudo access for it. > >>>>> _______________________________________________ > >>>>> Infra mailing list > >>>>> [email protected] > >>>>> http://lists.ovirt.org/mailman/listinfo/infra > >>>> > >>> _______________________________________________ > >>> Infra mailing list > >>> [email protected] > >>> http://lists.ovirt.org/mailman/listinfo/infra > >> > > > > _______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
