Hi Cos, I randomly install fedora packages from the S3 repo and it's all good. Thank you so much for your effort and the quick fix. Finally. we now have our bigtop repos all set!
Evans 2015-09-08 13:36 GMT+08:00 Konstantin Boudnik <[email protected]>: > Just to make sure this doesn't fall through - I have flushed out and > synced F20 repo. > > Everything should be in order now. Thansk! > Cos > > On Mon, Sep 07, 2015 at 03:59PM, Konstantin Boudnik wrote: > > On Tue, Sep 08, 2015 at 02:21AM, Evans Ye wrote: > > > I ran deployment test on those 3 yum repos. > > > centos6, 7 are good, however fedora is still failing. > > > It looks like the exactly same issue is still in Fedora repo which is > some > > > of the rpm are still the old one signed by old key. > > > > > > # old package > > > $ rpm --checksig hadoop-2.6.0-1.fc20.x86_64.rpm > > > hadoop-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK > (MISSING > > > KEYS: (MD5) PGP#d0c3824f) > > > > > > # new package > > > $ rpm --checksig bigtop-utils-1.0.0-1.fc20.noarch.rpm > > > bigtop-utils-1.0.0-1.fc20.noarch.rpm: rsa sha1 (md5) pgp md5 OK > > > > > > Cos could you please wipe up and resync Fedora repo on S3? Thanks! > > > > Thanks for validating and being so patient - I will do this by the end > of the day today. > > > > Cos > > > > > > > > Evans > > > > > > 2015-09-06 9:39 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > And it is done now - fresh copy of all centos packages signed by the > > > > correct > > > > key (as validated locally). Thanks for keep finding those, Evans! > > > > > > > > Cos > > > > > > > > On Sun, Sep 06, 2015 at 02:19AM, Konstantin Boudnik wrote: > > > > > It is pretty nuts because the packages I have locally are all > signed > > > > with the > > > > > correct key. But when I download the package in question I see > that the > > > > local > > > > > one is different from the downloaded. And the latter is signed > with the > > > > wrong > > > > > key indeed. > > > > > > > > > > Considering that I had synced everything after re-signing, > there're only > > > > two > > > > > possibilities that I see: > > > > > - S3 eventual consistency bites us in the rear. Which might be > possible > > > > for > > > > > in the short run, but I don't see how it couldn't be updated > after > > > > all that > > > > > time > > > > > - s3cmd has screwed up and didn't updated some of the packages. I > am > > > > going to > > > > > wipe out _all_ rpm distros and resync it right away. This might > cause > > > > a > > > > > short interruption in the packages availability, but at least > we'll > > > > have > > > > > all correct stuff up there. > > > > > > > > > > Should be done in about 30 minutes. Stay tuned > > > > > Cos > > > > > > > > > > On Sun, Sep 06, 2015 at 12:10AM, Evans Ye wrote: > > > > > > Sorry guys. I'm back with the issue again. ;) > > > > > > > > > > > > Turns out that some of the rpms are good, some are not. Look at > my > > > > tests > > > > > > below: > > > > > > > > > > > > > > > > > > ### Centos 6 repo ### > > > > > > > > > > > > $ docker run -ti --rm bigtop/puppet:centos-6 bash -l > > > > > > > > > > > > $ wget > > > > > > > > > > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/repos/centos6/bigtop.repo > > > > > > -O /etc/yum.repos.d/bigtop.repo > > > > > > > > > > > > $ yum -y install bigtop-utils bigtop-groovy bigtop-jsvc > bigtop-tomcat > > > > > > zookeeper # Successfully installed > > > > > > > > > > > > $ yum -y install hadoop hadoop-hdfs > > > > > > > > > > > > ... > > > > > > > > > > > > Error Downloading Packages: > > > > > > > > > > > > hadoop-hdfs-2.6.0-1.el6.x86_64: failure: > > > > > > hadoop/x86_64/hadoop-hdfs-2.6.0-1.el6.x86_64.rpm from bigtop: > [Errno > > > > 256] > > > > > > No more mirrors to try. > > > > > > > > > > > > hadoop-2.6.0-1.el6.x86_64: failure: > > > > > > hadoop/x86_64/hadoop-2.6.0-1.el6.x86_64.rpm from bigtop: [Errno > 256] No > > > > > > more mirrors to try. > > > > > > > > > > > > > > > > > > I find the same set of packages(groovy, utils, jsvc, tomcat, > > > > zookeeper) can > > > > > > be successfully installed across centos 6, 7 and fedora repos > and the > > > > other > > > > > > same set of packages failed to install across the platforms. > > > > Therefore, I > > > > > > think there might be an issue happening during some sort of > automation > > > > > > steps. > > > > > > > > > > > > In addition, I suspect that those packages failed to install are > still > > > > > > signed by old key, hence the subkey issue found by Cos blocks the > > > > packages > > > > > > to be installed. > > > > > > > > > > > > > > > > > > [root@34696969ce7d /]# rpm --checksig > > > > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm > > > > > > > > > > > > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) md5 > NOT OK > > > > > > (MISSING KEYS: (MD5) PGP#d0c3824f) > > > > > > > > > > > > [root@34696969ce7d /]# rpm --checksig > > > > bigtop-groovy-2.3.8-1.fc20.noarch.rpm > > > > > > > > > > > > bigtop-groovy-2.3.8-1.fc20.noarch.rpm: rsa sha1 (md5) pgp md5 OK > > > > > > > > > > > > > > > > > > Cos can you first check that the hadoop* packages has been > successfully > > > > > > resigned by your newly generated code signing key? Thanks! > > > > > > > > > > > > > > > > > > Evans > > > > > > 2015年9月4日 上午2:23於 "Konstantin Boudnik" <[email protected]>寫道： > > > > > > > > > > > > > Appreciate the sentiment guys and thanks for kind words! > > > > > > > The irony here is that I don't even like this type of > packaging and > > > > not > > > > > > > using > > > > > > > it if I can help it ;) Oh well... > > > > > > > > > > > > > > To close this thread - I will try to put together a blog about > 1.0 > > > > later > > > > > > > today. Thanks everyone for the testing, patience, and - kudos > to > > > > Evans - > > > > > > > detailed instructions on how to reproduce the issue! > > > > > > > > > > > > > > Cos > > > > > > > > > > > > > > On Thu, Sep 03, 2015 at 01:48PM, Jay Vyas wrote: > > > > > > > > Yes thanks cos for getting this centos stuff figured out.! > > > > > > > > > > > > > > > > > On Sep 3, 2015, at 12:35 PM, Andrew Purtell < > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Thanks for sticking with it Cos. That's an annoying bug. > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Wed, Sep 2, 2015 at 9:31 PM, Konstantin Boudnik < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> Ok, as I suspected there's a long standing (at least from > 2006) > > > > bug > > > > > > > in RPM > > > > > > > > >> that doesn't allow to validate RPM signature if a subkey > has > > > > been > > > > > > > used for > > > > > > > > >> signing. > > > > > > > > >> > > > > > > > > >> I ended up generating a new key pair (just for this > purpose) and > > > > > > > resigning > > > > > > > > >> all > > > > > > > > >> binaries with it; then resyncing everything with s3. I > also have > > > > > > > updated > > > > > > > > >> KEYS > > > > > > > > >> file with the new one. I have quickly ran a test on > centos7 by > > > > > > > installing > > > > > > > > >> bigtop-utils on an empty container and everything worked, > > > > including > > > > > > > > >> automatic > > > > > > > > >> import of the keys and the validation/installation of the > > > > package. > > > > > > > Looks > > > > > > > > >> like > > > > > > > > >> we are in the clear. > > > > > > > > >> > > > > > > > > >> Please shout if you see otherwise. Thanks everyone for > your > > > > patience! > > > > > > > > >> Cos > > > > > > > > >> > > > > > > > > >>> On Wed, Sep 02, 2015 at 02:27PM, Konstantin Boudnik > wrote: > > > > > > > > >>> I think there's a difference between how you've signed > the > > > > pkgs and > > > > > > > how > > > > > > > > >> I did > > > > > > > > >>> it. I signed with sub-key (as I mentioned before) and yum > > > > doesn't > > > > > > > > >> recognize > > > > > > > > >>> it. Seemingly, it expects that the master key was used > for > > > > signing. > > > > > > > > >>> > > > > > > > > >>> Also, in your repo file below > > > > > > > > >>> gpgkey=http://archive.apache.org/dist/bigtop/KEYS > > > > > > > > >>> points to the old keys. The location should be > > > > > > > > >>> gpgkey= > > > > https://dist.apache.org/repos/dist/release/bigtop/KEYS > > > > > > > > >>> > > > > > > > > >>> I am pretty sure I have exported my key with --armor > option > > > > back in > > > > > > > the > > > > > > > > >> day. > > > > > > > > >>> But I will repeat it and see if I can fix the situation, > which > > > > I also > > > > > > > > >> observer > > > > > > > > >>> following your steps. If that's the only issue I will > update > > > > the KEYS > > > > > > > > >> and we > > > > > > > > >>> should be completed by tonight ;) > > > > > > > > >>> > > > > > > > > >>> Thanks for your help! > > > > > > > > >>> Cos > > > > > > > > >>> > > > > > > > > >>>> On Wed, Sep 02, 2015 at 03:11PM, Evans Ye wrote: > > > > > > > > >>>> This is the same issue we're trying to solve in the > mailing > > > > thread > > > > > > > > >>>> "convenience artifacts are signed and uploaded". I've > built a > > > > sample > > > > > > > > >> repo > > > > > > > > >>>> which works properly by using my own key "Evans Ye" to > sign > > > > and to > > > > > > > > >> export > > > > > > > > >>>> GPG KEY. So I believe the following steps should be the > right > > > > way to > > > > > > > > >> sign > > > > > > > > >>>> packages and export the gpgkey: > > > > > > > > >>>> > > > > > > > > >>>> $ find -name *.rpm | xargs rpm --define="%_gpg_name > Evans Ye" > > > > > > > --addsign > > > > > > > > >>>> > > > > > > > > >>>> $ gpg --armor --output KEYS --export 'Evans Ye' > > > > > > > > >>>> I've verified that the hash is matched now in our > official > > > > repo. > > > > > > > > >>>> So I guess the main issue left is using non-armored gpg > key, > > > > if we > > > > > > > > >> manually > > > > > > > > >>>> import the gpgkey in the repo file: > > > > > > > > >>>> > > > > > > > > >>>> [bigtop] > > > > > > > > >>>> name=Bigtop > > > > > > > > >>>> enabled=1 > > > > > > > > >>>> gpgcheck=1 > > > > > > > > >>>> type=NONE > > > > > > > > >>>> baseurl= > > > > > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64 > > > > > > > > >>>> gpgkey=http://archive.apache.org/dist/bigtop/KEYS > > > > > > > > >>>> > > > > > > > > >>>> [root@48723d98dc1b ~]# rpm --import > > > > > > > > >>>> https://dist.apache.org/repos/dist/release/bigtop/KEYS > > > > > > > > >>>> error: > https://dist.apache.org/repos/dist/release/bigtop/KEYS: > > > > key > > > > > > > 2 > > > > > > > > >> not an > > > > > > > > >>>> armored public key. > > > > > > > > >>>> > > > > > > > > >>>> It gets error. > > > > > > > > >>>> However, my own exported armored key can be imported > without > > > > an > > > > > > > error. > > > > > > > > >>>> That's the different. > > > > > > > > >>>> > > > > > > > > >>>> Can you confirm that the gpgkey( > > > > > > > > >> http://archive.apache.org/dist/bigtop/KEYS) > > > > > > > > >>>> is exported with --armor flag? > > > > > > > > >>>> > > > > > > > > >>>> 2015-09-02 13:25 GMT+08:00 Konstantin Boudnik < > [email protected] > > > > >: > > > > > > > > >>>> > > > > > > > > >>>>> Looks like I have figured out what's wrong with my > key. And > > > > it is > > > > > > > > >>>>> _nothing_. > > > > > > > > >>>>> However, it seems that I can not sign RPMs with subkey > as > > > > YUM can > > > > > > > > >> not find > > > > > > > > >>>>> the > > > > > > > > >>>>> key while importing. Can anyone confirm or disprove my > train > > > > of > > > > > > > > >> thoughts? > > > > > > > > >>>>> > > > > > > > > >>>>> Thanks! > > > > > > > > >>>>> Cos > > > > > > > > >>>>> > > > > > > > > >>>>>> On Wed, Sep 02, 2015 at 07:42AM, Konstantin Boudnik > wrote: > > > > > > > > >>>>>> I've resynced the repodata once again and I don't see > this > > > > issue > > > > > > > > >> on the > > > > > > > > >>>>>> centos7 anymore. However, yum still complains about > the key > > > > being > > > > > > > > >> no > > > > > > > > >>>>>> available, but there's a workaround by setting > gpgcheck=0 > > > > And I am > > > > > > > > >> going > > > > > > > > >>>>> to > > > > > > > > >>>>>> figure out what to do with it and why my key isn't > working > > > > as > > > > > > > > >> expected. > > > > > > > > >>>>>> > > > > > > > > >>>>>> I also have discovered that the gpgkey file URL is > using > > > > the old > > > > > > > > >>>>> incubation > > > > > > > > >>>>>> KEYS. Fixed that as well. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Please let me know if you still see the issue with > checksums > > > > > > > > >> mismatch. > > > > > > > > >>>>>> Thanks, > > > > > > > > >>>>>> Cos > > > > > > > > >>>>>> > > > > > > > > >>>>>>> On Tue, Sep 01, 2015 at 12:44PM, Konstantin Boudnik > wrote: > > > > > > > > >>>>>>> I think this is the consequences of me fighting with > the > > > > package > > > > > > > > >>>>> signing... ;( > > > > > > > > >>>>>>> A couple of days ago I have re-ran 'createrepo' for > all the > > > > > > > > >> RPM-based > > > > > > > > >>>>> distros > > > > > > > > >>>>>>> and uploaded new repo files to the release. Not sure > why > > > > the > > > > > > > > >> checksums > > > > > > > > >>>>> differ > > > > > > > > >>>>>>> now... > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> I will take a look into this again tonight. > > > > > > > > >>>>>>> Cos > > > > > > > > >>>>>>> > > > > > > > > >>>>>>>> On Tue, Sep 01, 2015 at 09:39PM, Olaf Flebbe wrote: > > > > > > > > >>>>>>>> I can second it: > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> I added to /etc/yum.repo.d/meins.repo > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> [meins] > > > > > > > > >>>>>>>> name=Bigtop epo > > > > > > > > >>>>>>>> baseurl= > > > > > > > > >>>>> > > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/ > > > > > > > > >>>>>>>> enabled=1 > > > > > > > > >>>>>>>> gpgcheck=0 > > > > > > > > >>>>>>>> priority=1 > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> and got > > > > > > > > >>>>>>>> ............ > > > > > > > > >>>>>>>> Downloading packages: > > > > > > > > >>>>>>>> hbase-0.98.12-1.el7.centos.noa FAILED > > > > > > > > >>>>> > =============================================-] 849 > > > > kB/s > > > > > > > > >> | 62 > > > > > > > > >>>>> MB 00:00:00 ETA > > > > > > > > >> > > > > > > > > > > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/hbase/noarch/hbase-0.98.12-1.el7.centos.noarch.rpm > > > > > > > > >> : > > > > > > > > >>>>> [Errno -1] Package does not match intended download. > > > > Suggestion: > > > > > > > run > > > > > > > > >> yum > > > > > > > > >>>>> --enablerepo=meins clean metadata > > > > > > > > >>>>>>>> Trying other mirror. > > > > > > > > >>>>>>>> ............. > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> Olaf > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > - Andy > > > > > > > > > > > > > > > > > > Problems worthy of attack prove their worth by hitting > back. - > > > > Piet > > > > > > > Hein > > > > > > > > > (via Tom White) > > > > > > > > > > > > > > > > > > > >
