Indeee we do! Thanks for tireless testing and catching these mistakes! ;)
On September 8, 2015 5:35:07 AM PDT, Evans Ye <[email protected]> wrote: >Hi Cos, > >I randomly install fedora packages from the S3 repo and it's all good. >Thank you so much for your effort and the quick fix. >Finally. we now have our bigtop repos all set! > >Evans > >2015-09-08 13:36 GMT+08:00 Konstantin Boudnik <[email protected]>: > >> Just to make sure this doesn't fall through - I have flushed out and >> synced F20 repo. >> >> Everything should be in order now. Thansk! >> Cos >> >> On Mon, Sep 07, 2015 at 03:59PM, Konstantin Boudnik wrote: >> > On Tue, Sep 08, 2015 at 02:21AM, Evans Ye wrote: >> > > I ran deployment test on those 3 yum repos. >> > > centos6, 7 are good, however fedora is still failing. >> > > It looks like the exactly same issue is still in Fedora repo >which is >> some >> > > of the rpm are still the old one signed by old key. >> > > >> > > # old package >> > > $ rpm --checksig hadoop-2.6.0-1.fc20.x86_64.rpm >> > > hadoop-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK >> (MISSING >> > > KEYS: (MD5) PGP#d0c3824f) >> > > >> > > # new package >> > > $ rpm --checksig bigtop-utils-1.0.0-1.fc20.noarch.rpm >> > > bigtop-utils-1.0.0-1.fc20.noarch.rpm: rsa sha1 (md5) pgp md5 OK >> > > >> > > Cos could you please wipe up and resync Fedora repo on S3? >Thanks! >> > >> > Thanks for validating and being so patient - I will do this by the >end >> of the day today. >> > >> > Cos >> > >> > > >> > > Evans >> > > >> > > 2015-09-06 9:39 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > >> > > > And it is done now - fresh copy of all centos packages signed >by the >> > > > correct >> > > > key (as validated locally). Thanks for keep finding those, >Evans! >> > > > >> > > > Cos >> > > > >> > > > On Sun, Sep 06, 2015 at 02:19AM, Konstantin Boudnik wrote: >> > > > > It is pretty nuts because the packages I have locally are all >> signed >> > > > with the >> > > > > correct key. But when I download the package in question I >see >> that the >> > > > local >> > > > > one is different from the downloaded. And the latter is >signed >> with the >> > > > wrong >> > > > > key indeed. >> > > > > >> > > > > Considering that I had synced everything after re-signing, >> there're only >> > > > two >> > > > > possibilities that I see: >> > > > > - S3 eventual consistency bites us in the rear. Which might >be >> possible >> > > > for >> > > > > in the short run, but I don't see how it couldn't be >updated >> after >> > > > all that >> > > > > time >> > > > > - s3cmd has screwed up and didn't updated some of the >packages. I >> am >> > > > going to >> > > > > wipe out _all_ rpm distros and resync it right away. This >might >> cause >> > > > a >> > > > > short interruption in the packages availability, but at >least >> we'll >> > > > have >> > > > > all correct stuff up there. >> > > > > >> > > > > Should be done in about 30 minutes. Stay tuned >> > > > > Cos >> > > > > >> > > > > On Sun, Sep 06, 2015 at 12:10AM, Evans Ye wrote: >> > > > > > Sorry guys. I'm back with the issue again. ;) >> > > > > > >> > > > > > Turns out that some of the rpms are good, some are not. >Look at >> my >> > > > tests >> > > > > > below: >> > > > > > >> > > > > > >> > > > > > ### Centos 6 repo ### >> > > > > > >> > > > > > $ docker run -ti --rm bigtop/puppet:centos-6 bash -l >> > > > > > >> > > > > > $ wget >> > > > > > >> > > > >> >https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/repos/centos6/bigtop.repo >> > > > > > -O /etc/yum.repos.d/bigtop.repo >> > > > > > >> > > > > > $ yum -y install bigtop-utils bigtop-groovy bigtop-jsvc >> bigtop-tomcat >> > > > > > zookeeper # Successfully installed >> > > > > > >> > > > > > $ yum -y install hadoop hadoop-hdfs >> > > > > > >> > > > > > ... >> > > > > > >> > > > > > Error Downloading Packages: >> > > > > > >> > > > > > hadoop-hdfs-2.6.0-1.el6.x86_64: failure: >> > > > > > hadoop/x86_64/hadoop-hdfs-2.6.0-1.el6.x86_64.rpm from >bigtop: >> [Errno >> > > > 256] >> > > > > > No more mirrors to try. >> > > > > > >> > > > > > hadoop-2.6.0-1.el6.x86_64: failure: >> > > > > > hadoop/x86_64/hadoop-2.6.0-1.el6.x86_64.rpm from bigtop: >[Errno >> 256] No >> > > > > > more mirrors to try. >> > > > > > >> > > > > > >> > > > > > I find the same set of packages(groovy, utils, jsvc, >tomcat, >> > > > zookeeper) can >> > > > > > be successfully installed across centos 6, 7 and fedora >repos >> and the >> > > > other >> > > > > > same set of packages failed to install across the >platforms. >> > > > Therefore, I >> > > > > > think there might be an issue happening during some sort of >> automation >> > > > > > steps. >> > > > > > >> > > > > > In addition, I suspect that those packages failed to >install are >> still >> > > > > > signed by old key, hence the subkey issue found by Cos >blocks the >> > > > packages >> > > > > > to be installed. >> > > > > > >> > > > > > >> > > > > > [root@34696969ce7d /]# rpm --checksig >> > > > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm >> > > > > > >> > > > > > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) >md5 >> NOT OK >> > > > > > (MISSING KEYS: (MD5) PGP#d0c3824f) >> > > > > > >> > > > > > [root@34696969ce7d /]# rpm --checksig >> > > > bigtop-groovy-2.3.8-1.fc20.noarch.rpm >> > > > > > >> > > > > > bigtop-groovy-2.3.8-1.fc20.noarch.rpm: rsa sha1 (md5) pgp >md5 OK >> > > > > > >> > > > > > >> > > > > > Cos can you first check that the hadoop* packages has been >> successfully >> > > > > > resigned by your newly generated code signing key? Thanks! >> > > > > > >> > > > > > >> > > > > > Evans >> > > > > > 2015年9月4日 上午2:23於 "Konstantin Boudnik" <[email protected]>寫道： >> > > > > > >> > > > > > > Appreciate the sentiment guys and thanks for kind words! >> > > > > > > The irony here is that I don't even like this type of >> packaging and >> > > > not >> > > > > > > using >> > > > > > > it if I can help it ;) Oh well... >> > > > > > > >> > > > > > > To close this thread - I will try to put together a blog >about >> 1.0 >> > > > later >> > > > > > > today. Thanks everyone for the testing, patience, and - >kudos >> to >> > > > Evans - >> > > > > > > detailed instructions on how to reproduce the issue! >> > > > > > > >> > > > > > > Cos >> > > > > > > >> > > > > > > On Thu, Sep 03, 2015 at 01:48PM, Jay Vyas wrote: >> > > > > > > > Yes thanks cos for getting this centos stuff figured >out.! >> > > > > > > > >> > > > > > > > > On Sep 3, 2015, at 12:35 PM, Andrew Purtell < >> [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > Thanks for sticking with it Cos. That's an annoying >bug. >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> On Wed, Sep 2, 2015 at 9:31 PM, Konstantin Boudnik < >> > > > [email protected]> >> > > > > > > wrote: >> > > > > > > > >> >> > > > > > > > >> Ok, as I suspected there's a long standing (at least >from >> 2006) >> > > > bug >> > > > > > > in RPM >> > > > > > > > >> that doesn't allow to validate RPM signature if a >subkey >> has >> > > > been >> > > > > > > used for >> > > > > > > > >> signing. >> > > > > > > > >> >> > > > > > > > >> I ended up generating a new key pair (just for this >> purpose) and >> > > > > > > resigning >> > > > > > > > >> all >> > > > > > > > >> binaries with it; then resyncing everything with s3. >I >> also have >> > > > > > > updated >> > > > > > > > >> KEYS >> > > > > > > > >> file with the new one. I have quickly ran a test on >> centos7 by >> > > > > > > installing >> > > > > > > > >> bigtop-utils on an empty container and everything >worked, >> > > > including >> > > > > > > > >> automatic >> > > > > > > > >> import of the keys and the validation/installation >of the >> > > > package. >> > > > > > > Looks >> > > > > > > > >> like >> > > > > > > > >> we are in the clear. >> > > > > > > > >> >> > > > > > > > >> Please shout if you see otherwise. Thanks everyone >for >> your >> > > > patience! >> > > > > > > > >> Cos >> > > > > > > > >> >> > > > > > > > >>> On Wed, Sep 02, 2015 at 02:27PM, Konstantin Boudnik >> wrote: >> > > > > > > > >>> I think there's a difference between how you've >signed >> the >> > > > pkgs and >> > > > > > > how >> > > > > > > > >> I did >> > > > > > > > >>> it. I signed with sub-key (as I mentioned before) >and yum >> > > > doesn't >> > > > > > > > >> recognize >> > > > > > > > >>> it. Seemingly, it expects that the master key was >used >> for >> > > > signing. >> > > > > > > > >>> >> > > > > > > > >>> Also, in your repo file below >> > > > > > > > >>> >gpgkey=http://archive.apache.org/dist/bigtop/KEYS >> > > > > > > > >>> points to the old keys. The location should be >> > > > > > > > >>> gpgkey= >> > > > https://dist.apache.org/repos/dist/release/bigtop/KEYS >> > > > > > > > >>> >> > > > > > > > >>> I am pretty sure I have exported my key with >--armor >> option >> > > > back in >> > > > > > > the >> > > > > > > > >> day. >> > > > > > > > >>> But I will repeat it and see if I can fix the >situation, >> which >> > > > I also >> > > > > > > > >> observer >> > > > > > > > >>> following your steps. If that's the only issue I >will >> update >> > > > the KEYS >> > > > > > > > >> and we >> > > > > > > > >>> should be completed by tonight ;) >> > > > > > > > >>> >> > > > > > > > >>> Thanks for your help! >> > > > > > > > >>> Cos >> > > > > > > > >>> >> > > > > > > > >>>> On Wed, Sep 02, 2015 at 03:11PM, Evans Ye wrote: >> > > > > > > > >>>> This is the same issue we're trying to solve in >the >> mailing >> > > > thread >> > > > > > > > >>>> "convenience artifacts are signed and uploaded". >I've >> built a >> > > > sample >> > > > > > > > >> repo >> > > > > > > > >>>> which works properly by using my own key "Evans >Ye" to >> sign >> > > > and to >> > > > > > > > >> export >> > > > > > > > >>>> GPG KEY. So I believe the following steps should >be the >> right >> > > > way to >> > > > > > > > >> sign >> > > > > > > > >>>> packages and export the gpgkey: >> > > > > > > > >>>> >> > > > > > > > >>>> $ find -name *.rpm | xargs rpm >--define="%_gpg_name >> Evans Ye" >> > > > > > > --addsign >> > > > > > > > >>>> >> > > > > > > > >>>> $ gpg --armor --output KEYS --export 'Evans Ye' >> > > > > > > > >>>> I've verified that the hash is matched now in our >> official >> > > > repo. >> > > > > > > > >>>> So I guess the main issue left is using >non-armored gpg >> key, >> > > > if we >> > > > > > > > >> manually >> > > > > > > > >>>> import the gpgkey in the repo file: >> > > > > > > > >>>> >> > > > > > > > >>>> [bigtop] >> > > > > > > > >>>> name=Bigtop >> > > > > > > > >>>> enabled=1 >> > > > > > > > >>>> gpgcheck=1 >> > > > > > > > >>>> type=NONE >> > > > > > > > >>>> baseurl= >> > > > > > > >http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64 >> > > > > > > > >>>> gpgkey=http://archive.apache.org/dist/bigtop/KEYS >> > > > > > > > >>>> >> > > > > > > > >>>> [root@48723d98dc1b ~]# rpm --import >> > > > > > > > >>>> >https://dist.apache.org/repos/dist/release/bigtop/KEYS >> > > > > > > > >>>> error: >> https://dist.apache.org/repos/dist/release/bigtop/KEYS: >> > > > key >> > > > > > > 2 >> > > > > > > > >> not an >> > > > > > > > >>>> armored public key. >> > > > > > > > >>>> >> > > > > > > > >>>> It gets error. >> > > > > > > > >>>> However, my own exported armored key can be >imported >> without >> > > > an >> > > > > > > error. >> > > > > > > > >>>> That's the different. >> > > > > > > > >>>> >> > > > > > > > >>>> Can you confirm that the gpgkey( >> > > > > > > > >> http://archive.apache.org/dist/bigtop/KEYS) >> > > > > > > > >>>> is exported with --armor flag? >> > > > > > > > >>>> >> > > > > > > > >>>> 2015-09-02 13:25 GMT+08:00 Konstantin Boudnik < >> [email protected] >> > > > >: >> > > > > > > > >>>> >> > > > > > > > >>>>> Looks like I have figured out what's wrong with >my >> key. And >> > > > it is >> > > > > > > > >>>>> _nothing_. >> > > > > > > > >>>>> However, it seems that I can not sign RPMs with >subkey >> as >> > > > YUM can >> > > > > > > > >> not find >> > > > > > > > >>>>> the >> > > > > > > > >>>>> key while importing. Can anyone confirm or >disprove my >> train >> > > > of >> > > > > > > > >> thoughts? >> > > > > > > > >>>>> >> > > > > > > > >>>>> Thanks! >> > > > > > > > >>>>> Cos >> > > > > > > > >>>>> >> > > > > > > > >>>>>> On Wed, Sep 02, 2015 at 07:42AM, Konstantin >Boudnik >> wrote: >> > > > > > > > >>>>>> I've resynced the repodata once again and I >don't see >> this >> > > > issue >> > > > > > > > >> on the >> > > > > > > > >>>>>> centos7 anymore. However, yum still complains >about >> the key >> > > > being >> > > > > > > > >> no >> > > > > > > > >>>>>> available, but there's a workaround by setting >> gpgcheck=0 >> > > > And I am >> > > > > > > > >> going >> > > > > > > > >>>>> to >> > > > > > > > >>>>>> figure out what to do with it and why my key >isn't >> working >> > > > as >> > > > > > > > >> expected. >> > > > > > > > >>>>>> >> > > > > > > > >>>>>> I also have discovered that the gpgkey file URL >is >> using >> > > > the old >> > > > > > > > >>>>> incubation >> > > > > > > > >>>>>> KEYS. Fixed that as well. >> > > > > > > > >>>>>> >> > > > > > > > >>>>>> Please let me know if you still see the issue >with >> checksums >> > > > > > > > >> mismatch. >> > > > > > > > >>>>>> Thanks, >> > > > > > > > >>>>>> Cos >> > > > > > > > >>>>>> >> > > > > > > > >>>>>>> On Tue, Sep 01, 2015 at 12:44PM, Konstantin >Boudnik >> wrote: >> > > > > > > > >>>>>>> I think this is the consequences of me fighting >with >> the >> > > > package >> > > > > > > > >>>>> signing... ;( >> > > > > > > > >>>>>>> A couple of days ago I have re-ran 'createrepo' >for >> all the >> > > > > > > > >> RPM-based >> > > > > > > > >>>>> distros >> > > > > > > > >>>>>>> and uploaded new repo files to the release. Not >sure >> why >> > > > the >> > > > > > > > >> checksums >> > > > > > > > >>>>> differ >> > > > > > > > >>>>>>> now... >> > > > > > > > >>>>>>> >> > > > > > > > >>>>>>> I will take a look into this again tonight. >> > > > > > > > >>>>>>> Cos >> > > > > > > > >>>>>>> >> > > > > > > > >>>>>>>> On Tue, Sep 01, 2015 at 09:39PM, Olaf Flebbe >wrote: >> > > > > > > > >>>>>>>> I can second it: >> > > > > > > > >>>>>>>> >> > > > > > > > >>>>>>>> I added to /etc/yum.repo.d/meins.repo >> > > > > > > > >>>>>>>> >> > > > > > > > >>>>>>>> [meins] >> > > > > > > > >>>>>>>> name=Bigtop epo >> > > > > > > > >>>>>>>> baseurl= >> > > > > > > > >>>>> >> > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/ >> > > > > > > > >>>>>>>> enabled=1 >> > > > > > > > >>>>>>>> gpgcheck=0 >> > > > > > > > >>>>>>>> priority=1 >> > > > > > > > >>>>>>>> >> > > > > > > > >>>>>>>> and got >> > > > > > > > >>>>>>>> ............ >> > > > > > > > >>>>>>>> Downloading packages: >> > > > > > > > >>>>>>>> hbase-0.98.12-1.el7.centos.noa FAILED >> > > > > > > > >>>>> >> =============================================-] 849 >> > > > kB/s >> > > > > > > > >> | 62 >> > > > > > > > >>>>> MB 00:00:00 ETA >> > > > > > > > >> >> > > > > > > >> > > > >> >http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/hbase/noarch/hbase-0.98.12-1.el7.centos.noarch.rpm >> > > > > > > > >> : >> > > > > > > > >>>>> [Errno -1] Package does not match intended >download. >> > > > Suggestion: >> > > > > > > run >> > > > > > > > >> yum >> > > > > > > > >>>>> --enablerepo=meins clean metadata >> > > > > > > > >>>>>>>> Trying other mirror. >> > > > > > > > >>>>>>>> ............. >> > > > > > > > >>>>>>>> >> > > > > > > > >>>>>>>> Olaf >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > -- >> > > > > > > > > Best regards, >> > > > > > > > > >> > > > > > > > > - Andy >> > > > > > > > > >> > > > > > > > > Problems worthy of attack prove their worth by >hitting >> back. - >> > > > Piet >> > > > > > > Hein >> > > > > > > > > (via Tom White) >> > > > > > > >> > > > >> > > > >> > > > >>
