Just to wrap it up here as well. As reported on http://bit.ly/1XiYRRM - the situation has been resolved and the guilty part is RPM's long-standing inability to deal with subkeys.
Cos On Mon, Aug 31, 2015 at 12:57AM, Konstantin Boudnik wrote: > Arghhh... of course. I completely forgot about this ;( And I have updated the > wiki release page, to avoid forgetting this in the future. > > Synced the repodata folders to S3 - everything should be fine now. > Thanks a lot for your help! > Cos > > On Mon, Aug 31, 2015 at 03:37AM, Evans Ye wrote: > > Hi Cos, > > > > I finally found the issue by reproducing a repo in my S3 buckets. > > The answer is that the repodata needs to be regenerated after RPMs are > > being signed. > > > > Here's my testing steps: > > > > 1. download the whole built repo from jenkins release built as archive.zip > > and unzip it > > 2. sign RPMs: > > $ find -name *.rpm | xargs rpm --define="%_gpg_name Evans Ye" --addsign > > *3. recreate the metadata* > > * $ rm -rf repodata; createrepo -v .* > > 3. upload to S3 bucket: > > $ > > aws s3 cp bigtop-1.0.0-rpm-centos-6-output/ > > s3://evansye/bigtop/centos/6/x86_64 --recursive > > > > My repo: > > [bigtop] > > name=Bigtop > > enabled=1 > > gpgcheck=1 > > baseurl=https://s3-us-west-1.amazonaws.com/evansye/bigtop/centos/6/x86_64 > > gpgkey=https://s3-us-west-1.amazonaws.com/evansye/bigtop/KEYS > > > > Using my repo, the armored pgpkey will be automatically imported and the > > packages are being installed without a problem. > > > > Here's the TODO summary for out official repos: > > 1. Recreate metadata for yum repos > > 2. Replace http://archive.apache.org/dist/bigtop/KEYS to an armored key > > > > I think the bigtop official repo is critical to our user experience and I > > hope we can get it fixed ASAP. To be mentioned that Jazz Yao-Tsung Wang hits > > the issue too and reported back to me as well. > > If you're kind of busy I can help, too. Just let me know. Thanks! > > > > Evans > > > > 2015-08-24 1:48 GMT+08:00 Evans Ye <[email protected]>: > > > > > Hey Cos, thanks for updating the packages. > > > I was testing the repos these days, but it still behaving strange. > > > I constantly got "no more mirrors to try" when yum installing rpms from S3 > > > repo: > > > > > > Error Downloading Packages: > > > bigtop-utils-1.0.0-1.el6.noarch: failure: > > > bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm from bigtop: > > > [Errno > > > 256] No more mirrors to try. > > > > > > However, the package exist and has been signed as well: > > > > > > $ wget > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > > > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm > > > bigtop-utils-1.0.0-1.el6.noarch.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK > > > (MISSING KEYS: (MD5) PGP#d0c3824f) > > > > > > I can't figure out why for days, so I just what to bring up the issue and > > > see if you have any clue. > > > Does it work on your end? > > > > > > > > > 2015-08-21 11:32 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > >> Ok, my mistake - I have only signed src rpms ;( My bad - apologies for > > >> wasting > > >> your time. Now, with the key - I can really figure out what's going on > > >> here. > > >> The key seems to be fine, yet rpm --checksig isn't happy about it. I've > > >> re-imported the key as you suggested, yet ir complains that the key is > > >> missing > > >> during the validation phase. I want to get to the bottom of it, but at > > >> least > > >> we have the packages signed now (the upload should be over in about an > > >> hour, I > > >> hope). > > >> > > >> Thanks, > > >> Cos > > >> > > >> On Thu, Aug 20, 2015 at 01:31AM, Evans Ye wrote: > > >> > Hi Cos, > > >> > > > >> > I've verified deb repos, and they're good as what you said, however, it > > >> > seems that rpms are still not being signed. I don't have too much > > >> knowledge > > >> > on this so I did my homework and conduct the following evaluation. > > >> > Here's my evaluation steps, please advise if any thing incorrect: > > >> > > > >> > First, download a rpm from S3, which should be already signed: > > >> > > > >> > $ wget > > >> > > > >> http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > > >> > $ rpm --checksig > > >> > bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm > > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm.1: sha1 md5 OK > > >> > > > >> > As shown above, the rpm does not being signed. > > >> > A signed rpm should be looked like this: > > >> > > > >> > $ rpm --checksig puppetlabs-release-el-6.noarch.rpm > > >> > puppetlabs-release-el-6.noarch.rpm: rsa sha1 (md5) pgp md5 OK > > >> > > > >> > And after I signed the rpm by my key, the rpm looks good now: > > >> > > > >> > $ rpm --addsign bigtop-utils-1.0.0-1.el6.noarch.rpm > > >> > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm > > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm: rsa sha1 (md5) pgp md5 OK > > >> > > > >> > OTOH, it looks like the pgpkey for rpm packages needs to be armored > > >> > when > > >> > exporting, for example: > > >> > > > >> > $ gpg --armor --output KEYS --export 'Evans Ye' > > >> > > > >> > Otherwise, an error occurs when importing a non-armored key: > > >> > > > >> > $ rpm --import https://dist.apache.org/repos/dist/release/bigtop/KEYS > > >> > error: https://dist.apache.org/repos/dist/release/bigtop/KEYS: key 2 > > >> not an > > >> > armored public key. > > >> > > > >> > A good thing for that is we can fix it in the cloud. > > >> > Sorry for not discovering this at the very beginning. :( > > >> > > > >> > Evans > > >> > > > >> > > > >> > 2015-08-19 4:12 GMT+08:00 Konstantin Boudnik <[email protected]>: > > >> > > > >> > > On Wed, Aug 19, 2015 at 02:12AM, Evans Ye wrote: > > >> > > > Well, sorry. When I do manually test on those repos, I got package > > >> is not > > >> > > > signed message... > > >> > > > It turns out that our puppet recipe sets pgpkey checking to false, > > >> which > > >> > > is > > >> > > > why I got things working without a problem. > > >> > > > I assume the repo should just work by dropping it into > > >> /etc/yum.repos.d/. > > >> > > > No need to import pgpkey manually, right? > > >> > > > > >> > > Hmm... with ubuntu repo the signatures are there because I had to run > > >> > > apt-addkey manually to make apt recognize the signatures. Otherwise, > > >> > > apt-get update didn't work. > > >> > > > > >> > > You might need to do something similar with yum - I am not really > > >> sure. > > >> > > But I > > >> > > am positive that I have signed the packages per the insructions on > > >> > > our > > >> > > release > > >> > > page. > > >> > > > > >> > > Cos > > >> > > > > >> > > > 2015-08-18 2:39 GMT+08:00 Konstantin Boudnik <[email protected]>: > > >> > > > > > >> > > > > I think we are in good position - I was worries about keys being > > >> not > > >> > > found > > >> > > > > or > > >> > > > > something like this. If two of them are ok, then the rest should > > >> be > > >> > > fine > > >> > > > > too. > > >> > > > > > > >> > > > > Thank you very much for the confirmation and testing - really > > >> > > appreciate > > >> > > > > it! > > >> > > > > I will send the announcement shortly. > > >> > > > > > > >> > > > > Regards, > > >> > > > > Cos > > >> > > > > > > >> > > > > On Tue, Aug 18, 2015 at 02:05AM, Evans Ye wrote: > > >> > > > > > We're so close to the finished line! Thank you everybody! > > >> > > > > > I took sometime to run the deployment test before I go to bed. > > >> > > > > > Specifically I've tested centos-6, debian-8 repo, both work > > >> like a > > >> > > charm. > > >> > > > > > It should be all good, if no hurry I'll do more test tomorrow. > > >> :) > > >> > > > > > > > >> > > > > > 2015-08-17 14:00 GMT+08:00 Konstantin Boudnik > > >> > > > > > <[email protected]>: > > >> > > > > > > > >> > > > > > > See? I can not even write word "dense" without making a > > >> typo... > > >> > > That's > > >> > > > > how > > >> > > > > > > hot > > >> > > > > > > it is. Anyway, I have uploaded all signed packages to s3 (oh > > >> boy, > > >> > > was > > >> > > > > it > > >> > > > > > > fun?), and updated the repo files in the release under > > >> > > > > > > > > >> > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/ > > >> > > > > > > > > >> > > > > > > If you have a cycle or two - please do some quick validation > > >> and I > > >> > > will > > >> > > > > > > send > > >> > > > > > > the announcement to make the release final. > > >> > > > > > > > > >> > > > > > > Thanks everyone > > >> > > > > > > Cos > > >> > > > > > > > > >> > > > > > > On Mon, Aug 17, 2015 at 03:18AM, Konstantin Boudnik wrote: > > >> > > > > > > > Ah, of cource ... I am so sense when it's in the 100's F > > >> > > outside ;( > > >> > > > > > > Thanks! > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Sun, Aug 16, 2015 at 12:17PM, Roman Shaposhnik wrote: > > >> > > > > > > > > On Sat, Aug 15, 2015 at 11:27 PM, Konstantin Boudnik < > > >> > > > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > > Make sense - wgetting the stuff now. centos is done, > > >> debs and > > >> > > > > fedora > > >> > > > > > > to go. > > >> > > > > > > > > > Weirdly, deb repos are huge >2.5GB for whatever reason. > > >> > > > > > > > > > > >> > > > > > > > > IIRC, this had to do with double caching of packages. > > >> IOW, you > > >> > > only > > >> > > > > > > > > need to make this available: > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-1.0.0-deb/BUILD_ENVIRONMENTS=debian-8,label=docker-slave-07/lastSuccessfulBuild/artifact/output/apt/ > > >> > > > > > > > > > > >> > > > > > > > > not the top level dir. > > >> > > > > > > > > > > >> > > > > > > > > Thanks, > > >> > > > > > > > > Roman. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > > > > >
