Hi Cos, I finally found the issue by reproducing a repo in my S3 buckets. The answer is that the repodata needs to be regenerated after RPMs are being signed.
Here's my testing steps: 1. download the whole built repo from jenkins release built as archive.zip and unzip it 2. sign RPMs: $ find -name *.rpm | xargs rpm --define="%_gpg_name Evans Ye" --addsign *3. recreate the metadata* * $ rm -rf repodata; createrepo -v .* 3. upload to S3 bucket: $ aws s3 cp bigtop-1.0.0-rpm-centos-6-output/ s3://evansye/bigtop/centos/6/x86_64 --recursive My repo: [bigtop] name=Bigtop enabled=1 gpgcheck=1 baseurl=https://s3-us-west-1.amazonaws.com/evansye/bigtop/centos/6/x86_64 gpgkey=https://s3-us-west-1.amazonaws.com/evansye/bigtop/KEYS Using my repo, the armored pgpkey will be automatically imported and the packages are being installed without a problem. Here's the TODO summary for out official repos: 1. Recreate metadata for yum repos 2. Replace http://archive.apache.org/dist/bigtop/KEYS to an armored key I think the bigtop official repo is critical to our user experience and I hope we can get it fixed ASAP. To be mentioned that Jazz Yao-Tsung Wang hits the issue too and reported back to me as well. If you're kind of busy I can help, too. Just let me know. Thanks! Evans 2015-08-24 1:48 GMT+08:00 Evans Ye <[email protected]>: > Hey Cos, thanks for updating the packages. > I was testing the repos these days, but it still behaving strange. > I constantly got "no more mirrors to try" when yum installing rpms from S3 > repo: > > Error Downloading Packages: > bigtop-utils-1.0.0-1.el6.noarch: failure: > bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm from bigtop: [Errno > 256] No more mirrors to try. > > However, the package exist and has been signed as well: > > $ wget > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm > bigtop-utils-1.0.0-1.el6.noarch.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK > (MISSING KEYS: (MD5) PGP#d0c3824f) > > I can't figure out why for days, so I just what to bring up the issue and > see if you have any clue. > Does it work on your end? > > > 2015-08-21 11:32 GMT+08:00 Konstantin Boudnik <[email protected]>: > >> Ok, my mistake - I have only signed src rpms ;( My bad - apologies for >> wasting >> your time. Now, with the key - I can really figure out what's going on >> here. >> The key seems to be fine, yet rpm --checksig isn't happy about it. I've >> re-imported the key as you suggested, yet ir complains that the key is >> missing >> during the validation phase. I want to get to the bottom of it, but at >> least >> we have the packages signed now (the upload should be over in about an >> hour, I >> hope). >> >> Thanks, >> Cos >> >> On Thu, Aug 20, 2015 at 01:31AM, Evans Ye wrote: >> > Hi Cos, >> > >> > I've verified deb repos, and they're good as what you said, however, it >> > seems that rpms are still not being signed. I don't have too much >> knowledge >> > on this so I did my homework and conduct the following evaluation. >> > Here's my evaluation steps, please advise if any thing incorrect: >> > >> > First, download a rpm from S3, which should be already signed: >> > >> > $ wget >> > >> http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm >> > $ rpm --checksig bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm >> > bigtop-utils-1.0.0-1.el6.noarch.rpm.1: sha1 md5 OK >> > >> > As shown above, the rpm does not being signed. >> > A signed rpm should be looked like this: >> > >> > $ rpm --checksig puppetlabs-release-el-6.noarch.rpm >> > puppetlabs-release-el-6.noarch.rpm: rsa sha1 (md5) pgp md5 OK >> > >> > And after I signed the rpm by my key, the rpm looks good now: >> > >> > $ rpm --addsign bigtop-utils-1.0.0-1.el6.noarch.rpm >> > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm >> > bigtop-utils-1.0.0-1.el6.noarch.rpm: rsa sha1 (md5) pgp md5 OK >> > >> > OTOH, it looks like the pgpkey for rpm packages needs to be armored when >> > exporting, for example: >> > >> > $ gpg --armor --output KEYS --export 'Evans Ye' >> > >> > Otherwise, an error occurs when importing a non-armored key: >> > >> > $ rpm --import https://dist.apache.org/repos/dist/release/bigtop/KEYS >> > error: https://dist.apache.org/repos/dist/release/bigtop/KEYS: key 2 >> not an >> > armored public key. >> > >> > A good thing for that is we can fix it in the cloud. >> > Sorry for not discovering this at the very beginning. :( >> > >> > Evans >> > >> > >> > 2015-08-19 4:12 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > >> > > On Wed, Aug 19, 2015 at 02:12AM, Evans Ye wrote: >> > > > Well, sorry. When I do manually test on those repos, I got package >> is not >> > > > signed message... >> > > > It turns out that our puppet recipe sets pgpkey checking to false, >> which >> > > is >> > > > why I got things working without a problem. >> > > > I assume the repo should just work by dropping it into >> /etc/yum.repos.d/. >> > > > No need to import pgpkey manually, right? >> > > >> > > Hmm... with ubuntu repo the signatures are there because I had to run >> > > apt-addkey manually to make apt recognize the signatures. Otherwise, >> > > apt-get update didn't work. >> > > >> > > You might need to do something similar with yum - I am not really >> sure. >> > > But I >> > > am positive that I have signed the packages per the insructions on our >> > > release >> > > page. >> > > >> > > Cos >> > > >> > > > 2015-08-18 2:39 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > > >> > > > > I think we are in good position - I was worries about keys being >> not >> > > found >> > > > > or >> > > > > something like this. If two of them are ok, then the rest should >> be >> > > fine >> > > > > too. >> > > > > >> > > > > Thank you very much for the confirmation and testing - really >> > > appreciate >> > > > > it! >> > > > > I will send the announcement shortly. >> > > > > >> > > > > Regards, >> > > > > Cos >> > > > > >> > > > > On Tue, Aug 18, 2015 at 02:05AM, Evans Ye wrote: >> > > > > > We're so close to the finished line! Thank you everybody! >> > > > > > I took sometime to run the deployment test before I go to bed. >> > > > > > Specifically I've tested centos-6, debian-8 repo, both work >> like a >> > > charm. >> > > > > > It should be all good, if no hurry I'll do more test tomorrow. >> :) >> > > > > > >> > > > > > 2015-08-17 14:00 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > > > > >> > > > > > > See? I can not even write word "dense" without making a >> typo... >> > > That's >> > > > > how >> > > > > > > hot >> > > > > > > it is. Anyway, I have uploaded all signed packages to s3 (oh >> boy, >> > > was >> > > > > it >> > > > > > > fun?), and updated the repo files in the release under >> > > > > > > >> > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/ >> > > > > > > >> > > > > > > If you have a cycle or two - please do some quick validation >> and I >> > > will >> > > > > > > send >> > > > > > > the announcement to make the release final. >> > > > > > > >> > > > > > > Thanks everyone >> > > > > > > Cos >> > > > > > > >> > > > > > > On Mon, Aug 17, 2015 at 03:18AM, Konstantin Boudnik wrote: >> > > > > > > > Ah, of cource ... I am so sense when it's in the 100's F >> > > outside ;( >> > > > > > > Thanks! >> > > > > > > > >> > > > > > > > >> > > > > > > > On Sun, Aug 16, 2015 at 12:17PM, Roman Shaposhnik wrote: >> > > > > > > > > On Sat, Aug 15, 2015 at 11:27 PM, Konstantin Boudnik < >> > > > > [email protected]> >> > > > > > > wrote: >> > > > > > > > > > Make sense - wgetting the stuff now. centos is done, >> debs and >> > > > > fedora >> > > > > > > to go. >> > > > > > > > > > Weirdly, deb repos are huge >2.5GB for whatever reason. >> > > > > > > > > >> > > > > > > > > IIRC, this had to do with double caching of packages. >> IOW, you >> > > only >> > > > > > > > > need to make this available: >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-1.0.0-deb/BUILD_ENVIRONMENTS=debian-8,label=docker-slave-07/lastSuccessfulBuild/artifact/output/apt/ >> > > > > > > > > >> > > > > > > > > not the top level dir. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Roman. >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> > > >> > >
