I ran deployment test on those 3 yum repos. centos6, 7 are good, however fedora is still failing. It looks like the exactly same issue is still in Fedora repo which is some of the rpm are still the old one signed by old key.
# old package $ rpm --checksig hadoop-2.6.0-1.fc20.x86_64.rpm hadoop-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK (MISSING KEYS: (MD5) PGP#d0c3824f) # new package $ rpm --checksig bigtop-utils-1.0.0-1.fc20.noarch.rpm bigtop-utils-1.0.0-1.fc20.noarch.rpm: rsa sha1 (md5) pgp md5 OK Cos could you please wipe up and resync Fedora repo on S3? Thanks! Evans 2015-09-06 9:39 GMT+08:00 Konstantin Boudnik <[email protected]>: > And it is done now - fresh copy of all centos packages signed by the > correct > key (as validated locally). Thanks for keep finding those, Evans! > > Cos > > On Sun, Sep 06, 2015 at 02:19AM, Konstantin Boudnik wrote: > > It is pretty nuts because the packages I have locally are all signed > with the > > correct key. But when I download the package in question I see that the > local > > one is different from the downloaded. And the latter is signed with the > wrong > > key indeed. > > > > Considering that I had synced everything after re-signing, there're only > two > > possibilities that I see: > > - S3 eventual consistency bites us in the rear. Which might be possible > for > > in the short run, but I don't see how it couldn't be updated after > all that > > time > > - s3cmd has screwed up and didn't updated some of the packages. I am > going to > > wipe out _all_ rpm distros and resync it right away. This might cause > a > > short interruption in the packages availability, but at least we'll > have > > all correct stuff up there. > > > > Should be done in about 30 minutes. Stay tuned > > Cos > > > > On Sun, Sep 06, 2015 at 12:10AM, Evans Ye wrote: > > > Sorry guys. I'm back with the issue again. ;) > > > > > > Turns out that some of the rpms are good, some are not. Look at my > tests > > > below: > > > > > > > > > ### Centos 6 repo ### > > > > > > $ docker run -ti --rm bigtop/puppet:centos-6 bash -l > > > > > > $ wget > > > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/repos/centos6/bigtop.repo > > > -O /etc/yum.repos.d/bigtop.repo > > > > > > $ yum -y install bigtop-utils bigtop-groovy bigtop-jsvc bigtop-tomcat > > > zookeeper # Successfully installed > > > > > > $ yum -y install hadoop hadoop-hdfs > > > > > > ... > > > > > > Error Downloading Packages: > > > > > > hadoop-hdfs-2.6.0-1.el6.x86_64: failure: > > > hadoop/x86_64/hadoop-hdfs-2.6.0-1.el6.x86_64.rpm from bigtop: [Errno > 256] > > > No more mirrors to try. > > > > > > hadoop-2.6.0-1.el6.x86_64: failure: > > > hadoop/x86_64/hadoop-2.6.0-1.el6.x86_64.rpm from bigtop: [Errno 256] No > > > more mirrors to try. > > > > > > > > > I find the same set of packages(groovy, utils, jsvc, tomcat, > zookeeper) can > > > be successfully installed across centos 6, 7 and fedora repos and the > other > > > same set of packages failed to install across the platforms. > Therefore, I > > > think there might be an issue happening during some sort of automation > > > steps. > > > > > > In addition, I suspect that those packages failed to install are still > > > signed by old key, hence the subkey issue found by Cos blocks the > packages > > > to be installed. > > > > > > > > > [root@34696969ce7d /]# rpm --checksig > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm > > > > > > hadoop-hdfs-2.6.0-1.fc20.x86_64.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK > > > (MISSING KEYS: (MD5) PGP#d0c3824f) > > > > > > [root@34696969ce7d /]# rpm --checksig > bigtop-groovy-2.3.8-1.fc20.noarch.rpm > > > > > > bigtop-groovy-2.3.8-1.fc20.noarch.rpm: rsa sha1 (md5) pgp md5 OK > > > > > > > > > Cos can you first check that the hadoop* packages has been successfully > > > resigned by your newly generated code signing key? Thanks! > > > > > > > > > Evans > > > 2015年9月4日 上午2:23於 "Konstantin Boudnik" <[email protected]>寫道： > > > > > > > Appreciate the sentiment guys and thanks for kind words! > > > > The irony here is that I don't even like this type of packaging and > not > > > > using > > > > it if I can help it ;) Oh well... > > > > > > > > To close this thread - I will try to put together a blog about 1.0 > later > > > > today. Thanks everyone for the testing, patience, and - kudos to > Evans - > > > > detailed instructions on how to reproduce the issue! > > > > > > > > Cos > > > > > > > > On Thu, Sep 03, 2015 at 01:48PM, Jay Vyas wrote: > > > > > Yes thanks cos for getting this centos stuff figured out.! > > > > > > > > > > > On Sep 3, 2015, at 12:35 PM, Andrew Purtell <[email protected] > > > > > > wrote: > > > > > > > > > > > > Thanks for sticking with it Cos. That's an annoying bug. > > > > > > > > > > > > > > > > > >> On Wed, Sep 2, 2015 at 9:31 PM, Konstantin Boudnik < > [email protected]> > > > > wrote: > > > > > >> > > > > > >> Ok, as I suspected there's a long standing (at least from 2006) > bug > > > > in RPM > > > > > >> that doesn't allow to validate RPM signature if a subkey has > been > > > > used for > > > > > >> signing. > > > > > >> > > > > > >> I ended up generating a new key pair (just for this purpose) and > > > > resigning > > > > > >> all > > > > > >> binaries with it; then resyncing everything with s3. I also have > > > > updated > > > > > >> KEYS > > > > > >> file with the new one. I have quickly ran a test on centos7 by > > > > installing > > > > > >> bigtop-utils on an empty container and everything worked, > including > > > > > >> automatic > > > > > >> import of the keys and the validation/installation of the > package. > > > > Looks > > > > > >> like > > > > > >> we are in the clear. > > > > > >> > > > > > >> Please shout if you see otherwise. Thanks everyone for your > patience! > > > > > >> Cos > > > > > >> > > > > > >>> On Wed, Sep 02, 2015 at 02:27PM, Konstantin Boudnik wrote: > > > > > >>> I think there's a difference between how you've signed the > pkgs and > > > > how > > > > > >> I did > > > > > >>> it. I signed with sub-key (as I mentioned before) and yum > doesn't > > > > > >> recognize > > > > > >>> it. Seemingly, it expects that the master key was used for > signing. > > > > > >>> > > > > > >>> Also, in your repo file below > > > > > >>> gpgkey=http://archive.apache.org/dist/bigtop/KEYS > > > > > >>> points to the old keys. The location should be > > > > > >>> gpgkey= > https://dist.apache.org/repos/dist/release/bigtop/KEYS > > > > > >>> > > > > > >>> I am pretty sure I have exported my key with --armor option > back in > > > > the > > > > > >> day. > > > > > >>> But I will repeat it and see if I can fix the situation, which > I also > > > > > >> observer > > > > > >>> following your steps. If that's the only issue I will update > the KEYS > > > > > >> and we > > > > > >>> should be completed by tonight ;) > > > > > >>> > > > > > >>> Thanks for your help! > > > > > >>> Cos > > > > > >>> > > > > > >>>> On Wed, Sep 02, 2015 at 03:11PM, Evans Ye wrote: > > > > > >>>> This is the same issue we're trying to solve in the mailing > thread > > > > > >>>> "convenience artifacts are signed and uploaded". I've built a > sample > > > > > >> repo > > > > > >>>> which works properly by using my own key "Evans Ye" to sign > and to > > > > > >> export > > > > > >>>> GPG KEY. So I believe the following steps should be the right > way to > > > > > >> sign > > > > > >>>> packages and export the gpgkey: > > > > > >>>> > > > > > >>>> $ find -name *.rpm | xargs rpm --define="%_gpg_name Evans Ye" > > > > --addsign > > > > > >>>> > > > > > >>>> $ gpg --armor --output KEYS --export 'Evans Ye' > > > > > >>>> I've verified that the hash is matched now in our official > repo. > > > > > >>>> So I guess the main issue left is using non-armored gpg key, > if we > > > > > >> manually > > > > > >>>> import the gpgkey in the repo file: > > > > > >>>> > > > > > >>>> [bigtop] > > > > > >>>> name=Bigtop > > > > > >>>> enabled=1 > > > > > >>>> gpgcheck=1 > > > > > >>>> type=NONE > > > > > >>>> baseurl= > > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64 > > > > > >>>> gpgkey=http://archive.apache.org/dist/bigtop/KEYS > > > > > >>>> > > > > > >>>> [root@48723d98dc1b ~]# rpm --import > > > > > >>>> https://dist.apache.org/repos/dist/release/bigtop/KEYS > > > > > >>>> error: https://dist.apache.org/repos/dist/release/bigtop/KEYS: > key > > > > 2 > > > > > >> not an > > > > > >>>> armored public key. > > > > > >>>> > > > > > >>>> It gets error. > > > > > >>>> However, my own exported armored key can be imported without > an > > > > error. > > > > > >>>> That's the different. > > > > > >>>> > > > > > >>>> Can you confirm that the gpgkey( > > > > > >> http://archive.apache.org/dist/bigtop/KEYS) > > > > > >>>> is exported with --armor flag? > > > > > >>>> > > > > > >>>> 2015-09-02 13:25 GMT+08:00 Konstantin Boudnik <[email protected] > >: > > > > > >>>> > > > > > >>>>> Looks like I have figured out what's wrong with my key. And > it is > > > > > >>>>> _nothing_. > > > > > >>>>> However, it seems that I can not sign RPMs with subkey as > YUM can > > > > > >> not find > > > > > >>>>> the > > > > > >>>>> key while importing. Can anyone confirm or disprove my train > of > > > > > >> thoughts? > > > > > >>>>> > > > > > >>>>> Thanks! > > > > > >>>>> Cos > > > > > >>>>> > > > > > >>>>>> On Wed, Sep 02, 2015 at 07:42AM, Konstantin Boudnik wrote: > > > > > >>>>>> I've resynced the repodata once again and I don't see this > issue > > > > > >> on the > > > > > >>>>>> centos7 anymore. However, yum still complains about the key > being > > > > > >> no > > > > > >>>>>> available, but there's a workaround by setting gpgcheck=0 > And I am > > > > > >> going > > > > > >>>>> to > > > > > >>>>>> figure out what to do with it and why my key isn't working > as > > > > > >> expected. > > > > > >>>>>> > > > > > >>>>>> I also have discovered that the gpgkey file URL is using > the old > > > > > >>>>> incubation > > > > > >>>>>> KEYS. Fixed that as well. > > > > > >>>>>> > > > > > >>>>>> Please let me know if you still see the issue with checksums > > > > > >> mismatch. > > > > > >>>>>> Thanks, > > > > > >>>>>> Cos > > > > > >>>>>> > > > > > >>>>>>> On Tue, Sep 01, 2015 at 12:44PM, Konstantin Boudnik wrote: > > > > > >>>>>>> I think this is the consequences of me fighting with the > package > > > > > >>>>> signing... ;( > > > > > >>>>>>> A couple of days ago I have re-ran 'createrepo' for all the > > > > > >> RPM-based > > > > > >>>>> distros > > > > > >>>>>>> and uploaded new repo files to the release. Not sure why > the > > > > > >> checksums > > > > > >>>>> differ > > > > > >>>>>>> now... > > > > > >>>>>>> > > > > > >>>>>>> I will take a look into this again tonight. > > > > > >>>>>>> Cos > > > > > >>>>>>> > > > > > >>>>>>>> On Tue, Sep 01, 2015 at 09:39PM, Olaf Flebbe wrote: > > > > > >>>>>>>> I can second it: > > > > > >>>>>>>> > > > > > >>>>>>>> I added to /etc/yum.repo.d/meins.repo > > > > > >>>>>>>> > > > > > >>>>>>>> [meins] > > > > > >>>>>>>> name=Bigtop epo > > > > > >>>>>>>> baseurl= > > > > > >>>>> > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/ > > > > > >>>>>>>> enabled=1 > > > > > >>>>>>>> gpgcheck=0 > > > > > >>>>>>>> priority=1 > > > > > >>>>>>>> > > > > > >>>>>>>> and got > > > > > >>>>>>>> ............ > > > > > >>>>>>>> Downloading packages: > > > > > >>>>>>>> hbase-0.98.12-1.el7.centos.noa FAILED > > > > > >>>>> =============================================-] 849 > kB/s > > > > > >> | 62 > > > > > >>>>> MB 00:00:00 ETA > > > > > >> > > > > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/7/x86_64/hbase/noarch/hbase-0.98.12-1.el7.centos.noarch.rpm > > > > > >> : > > > > > >>>>> [Errno -1] Package does not match intended download. > Suggestion: > > > > run > > > > > >> yum > > > > > >>>>> --enablerepo=meins clean metadata > > > > > >>>>>>>> Trying other mirror. > > > > > >>>>>>>> ............. > > > > > >>>>>>>> > > > > > >>>>>>>> Olaf > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > > > > > > > > > > - Andy > > > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - > Piet > > > > Hein > > > > > > (via Tom White) > > > > > > >
