Arghhh... of course. I completely forgot about this ;( And I have updated the wiki release page, to avoid forgetting this in the future.
Synced the repodata folders to S3 - everything should be fine now. Thanks a lot for your help! Cos On Mon, Aug 31, 2015 at 03:37AM, Evans Ye wrote: > Hi Cos, > > I finally found the issue by reproducing a repo in my S3 buckets. > The answer is that the repodata needs to be regenerated after RPMs are > being signed. > > Here's my testing steps: > > 1. download the whole built repo from jenkins release built as archive.zip > and unzip it > 2. sign RPMs: > $ find -name *.rpm | xargs rpm --define="%_gpg_name Evans Ye" --addsign > *3. recreate the metadata* > * $ rm -rf repodata; createrepo -v .* > 3. upload to S3 bucket: > $ > aws s3 cp bigtop-1.0.0-rpm-centos-6-output/ > s3://evansye/bigtop/centos/6/x86_64 --recursive > > My repo: > [bigtop] > name=Bigtop > enabled=1 > gpgcheck=1 > baseurl=https://s3-us-west-1.amazonaws.com/evansye/bigtop/centos/6/x86_64 > gpgkey=https://s3-us-west-1.amazonaws.com/evansye/bigtop/KEYS > > Using my repo, the armored pgpkey will be automatically imported and the > packages are being installed without a problem. > > Here's the TODO summary for out official repos: > 1. Recreate metadata for yum repos > 2. Replace http://archive.apache.org/dist/bigtop/KEYS to an armored key > > I think the bigtop official repo is critical to our user experience and I > hope we can get it fixed ASAP. To be mentioned that Jazz Yao-Tsung Wang hits > the issue too and reported back to me as well. > If you're kind of busy I can help, too. Just let me know. Thanks! > > Evans > > 2015-08-24 1:48 GMT+08:00 Evans Ye <[email protected]>: > > > Hey Cos, thanks for updating the packages. > > I was testing the repos these days, but it still behaving strange. > > I constantly got "no more mirrors to try" when yum installing rpms from S3 > > repo: > > > > Error Downloading Packages: > > bigtop-utils-1.0.0-1.el6.noarch: failure: > > bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm from bigtop: [Errno > > 256] No more mirrors to try. > > > > However, the package exist and has been signed as well: > > > > $ wget > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm > > bigtop-utils-1.0.0-1.el6.noarch.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK > > (MISSING KEYS: (MD5) PGP#d0c3824f) > > > > I can't figure out why for days, so I just what to bring up the issue and > > see if you have any clue. > > Does it work on your end? > > > > > > 2015-08-21 11:32 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > >> Ok, my mistake - I have only signed src rpms ;( My bad - apologies for > >> wasting > >> your time. Now, with the key - I can really figure out what's going on > >> here. > >> The key seems to be fine, yet rpm --checksig isn't happy about it. I've > >> re-imported the key as you suggested, yet ir complains that the key is > >> missing > >> during the validation phase. I want to get to the bottom of it, but at > >> least > >> we have the packages signed now (the upload should be over in about an > >> hour, I > >> hope). > >> > >> Thanks, > >> Cos > >> > >> On Thu, Aug 20, 2015 at 01:31AM, Evans Ye wrote: > >> > Hi Cos, > >> > > >> > I've verified deb repos, and they're good as what you said, however, it > >> > seems that rpms are still not being signed. I don't have too much > >> knowledge > >> > on this so I did my homework and conduct the following evaluation. > >> > Here's my evaluation steps, please advise if any thing incorrect: > >> > > >> > First, download a rpm from S3, which should be already signed: > >> > > >> > $ wget > >> > > >> http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > >> > $ rpm --checksig bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm.1: sha1 md5 OK > >> > > >> > As shown above, the rpm does not being signed. > >> > A signed rpm should be looked like this: > >> > > >> > $ rpm --checksig puppetlabs-release-el-6.noarch.rpm > >> > puppetlabs-release-el-6.noarch.rpm: rsa sha1 (md5) pgp md5 OK > >> > > >> > And after I signed the rpm by my key, the rpm looks good now: > >> > > >> > $ rpm --addsign bigtop-utils-1.0.0-1.el6.noarch.rpm > >> > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm: rsa sha1 (md5) pgp md5 OK > >> > > >> > OTOH, it looks like the pgpkey for rpm packages needs to be armored when > >> > exporting, for example: > >> > > >> > $ gpg --armor --output KEYS --export 'Evans Ye' > >> > > >> > Otherwise, an error occurs when importing a non-armored key: > >> > > >> > $ rpm --import https://dist.apache.org/repos/dist/release/bigtop/KEYS > >> > error: https://dist.apache.org/repos/dist/release/bigtop/KEYS: key 2 > >> not an > >> > armored public key. > >> > > >> > A good thing for that is we can fix it in the cloud. > >> > Sorry for not discovering this at the very beginning. :( > >> > > >> > Evans > >> > > >> > > >> > 2015-08-19 4:12 GMT+08:00 Konstantin Boudnik <[email protected]>: > >> > > >> > > On Wed, Aug 19, 2015 at 02:12AM, Evans Ye wrote: > >> > > > Well, sorry. When I do manually test on those repos, I got package > >> is not > >> > > > signed message... > >> > > > It turns out that our puppet recipe sets pgpkey checking to false, > >> which > >> > > is > >> > > > why I got things working without a problem. > >> > > > I assume the repo should just work by dropping it into > >> /etc/yum.repos.d/. > >> > > > No need to import pgpkey manually, right? > >> > > > >> > > Hmm... with ubuntu repo the signatures are there because I had to run > >> > > apt-addkey manually to make apt recognize the signatures. Otherwise, > >> > > apt-get update didn't work. > >> > > > >> > > You might need to do something similar with yum - I am not really > >> sure. > >> > > But I > >> > > am positive that I have signed the packages per the insructions on our > >> > > release > >> > > page. > >> > > > >> > > Cos > >> > > > >> > > > 2015-08-18 2:39 GMT+08:00 Konstantin Boudnik <[email protected]>: > >> > > > > >> > > > > I think we are in good position - I was worries about keys being > >> not > >> > > found > >> > > > > or > >> > > > > something like this. If two of them are ok, then the rest should > >> be > >> > > fine > >> > > > > too. > >> > > > > > >> > > > > Thank you very much for the confirmation and testing - really > >> > > appreciate > >> > > > > it! > >> > > > > I will send the announcement shortly. > >> > > > > > >> > > > > Regards, > >> > > > > Cos > >> > > > > > >> > > > > On Tue, Aug 18, 2015 at 02:05AM, Evans Ye wrote: > >> > > > > > We're so close to the finished line! Thank you everybody! > >> > > > > > I took sometime to run the deployment test before I go to bed. > >> > > > > > Specifically I've tested centos-6, debian-8 repo, both work > >> like a > >> > > charm. > >> > > > > > It should be all good, if no hurry I'll do more test tomorrow. > >> :) > >> > > > > > > >> > > > > > 2015-08-17 14:00 GMT+08:00 Konstantin Boudnik <[email protected]>: > >> > > > > > > >> > > > > > > See? I can not even write word "dense" without making a > >> typo... > >> > > That's > >> > > > > how > >> > > > > > > hot > >> > > > > > > it is. Anyway, I have uploaded all signed packages to s3 (oh > >> boy, > >> > > was > >> > > > > it > >> > > > > > > fun?), and updated the repo files in the release under > >> > > > > > > > >> > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/ > >> > > > > > > > >> > > > > > > If you have a cycle or two - please do some quick validation > >> and I > >> > > will > >> > > > > > > send > >> > > > > > > the announcement to make the release final. > >> > > > > > > > >> > > > > > > Thanks everyone > >> > > > > > > Cos > >> > > > > > > > >> > > > > > > On Mon, Aug 17, 2015 at 03:18AM, Konstantin Boudnik wrote: > >> > > > > > > > Ah, of cource ... I am so sense when it's in the 100's F > >> > > outside ;( > >> > > > > > > Thanks! > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Sun, Aug 16, 2015 at 12:17PM, Roman Shaposhnik wrote: > >> > > > > > > > > On Sat, Aug 15, 2015 at 11:27 PM, Konstantin Boudnik < > >> > > > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > > > > Make sense - wgetting the stuff now. centos is done, > >> debs and > >> > > > > fedora > >> > > > > > > to go. > >> > > > > > > > > > Weirdly, deb repos are huge >2.5GB for whatever reason. > >> > > > > > > > > > >> > > > > > > > > IIRC, this had to do with double caching of packages. > >> IOW, you > >> > > only > >> > > > > > > > > need to make this available: > >> > > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-1.0.0-deb/BUILD_ENVIRONMENTS=debian-8,label=docker-slave-07/lastSuccessfulBuild/artifact/output/apt/ > >> > > > > > > > > > >> > > > > > > > > not the top level dir. > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > Roman. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> > > > >
