Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

> @andrewclausen The Sequoia backend should process revocation certificates 
> correctly. The internal backend does not, which is one of multiple reasons it 
> is deprecated.

It's not just a question of the backend.  For example, how would the revocation 
certificates be distributed?

My main point here is: security is hard, and users (like me) definitely 
appreciate seeing that everything has been thought through.

If so, I think @nwalfield's proposal should work well.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1652685398
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

> > ```
> > * Another example is the [Google's Linux signing 
> > key](https://dl.google.com/linux/linux_signing_key.pub), which is actually 
> > a collection of public keys bundled inside a single certificate.  For the 
> > record, its contents are:
> >   ```
> >   $ cat linux_signing_key.pub | gpg --show-keys --with-subkey-fingerprint
> >   pub   dsa1024 2007-03-08 [SC]
> > 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
> >   uid  Google, Inc. Linux Package Signing Key 
> > 
> >   sub   elg2048 2007-03-08 [E]
> > 9534C9C4130B4DC9927992BF4F30B6B4C07CB649
> >   
> >   pub   rsa4096 2016-04-12 [SC]
> > EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
> >   uid  Google Inc. (Linux Packages Signing Authority) 
> > 
> >   sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
> > 3B068FB4789ABE4AEFA3BB491397BC53640DB551
> >   sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
> > 3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
> >   sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
> > 2F528D36D67B69EDF998D85778BD65473CB3BD13
> >   sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
> > 8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
> >   sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
> > A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2
> >   ```
> > ```
> 
> This is not how the term certificate is commonly used in the OpenPGP 
> ecosystem. This file includes two certificates: 
> 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 and 
> EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796.

Thank you for your patience!

They are bundled together into a single file, and into a single (virtual?) RPM 
package gpg-pubkey-38ab71f4-60242b08.  Is there standard terminology for this 
situation?

> > > Can you explain what it means for "a key master" to "publish keys with 
> > > separate certificates"?
> > 
> > My first example above -- the Fedora 36 signing key -- is distributed in 
> > its own certificate. Fedora 37 has a separate key and a separate 
> > certificate. This separation means that the user can install any 
> > combination of them at their leisure. If for some reason they want to check 
> > old signatures, they can do so by installing old certificates containing 
> > old keys. Since Fedora publishes keys in separate certificates, there is no 
> > need to merge Fedora's certificates.
> 
> You've unfortunately lost me again. In OpenPGP, a key is not separate from a 
> certificate. A key is a component.

What I mean is that the Fedora 36 signing key can be installed separately from 
the Fedora 37 signing key.  They are in separate virtual RPM packages, and they 
have separate keys and certificates.  This contrasts with the Google situation 
where it's different versions of the same certificate.

> > On the other hand, Google publishes many keys within a single certificate 
> > (the second example). If a new version of the certificate removes some old 
> > keys, this would prevent the user from verifying old signatures. For 
> > example, the key that was issued on 2016-04-12 (and expired in 2019) might 
> > get removed from future versions of this certificate. If this happens, then 
> > the user would have no obvious way of verifying packages signed by this 
> > key. Your proposal of merging the new certificate with previously installed 
> > ones is one way of addressing this problem. But I think it comes with a 
> > serious downside that the user has no way removing revoked keys. If the 
> > 2016-04-12 key gets compromised, your proposal might leave the user 
> > vulnerable to attacker-signed packages. (The fact that they key has expired 
> > might help, but it's not the end of the story; e.g. what if a more recent 
> > key gets compromised?)
> 
> I don't understand why a user would want to remove a revoked key. If it is 
> revoked, the user should just import the revocation certificate and then it 
> can't be used to verify packages any more.

Good point.  I guess I was worried that the key master might not distribute a 
revocation certificate, or that DNF / RPM might not acquire and process the 
revocation certificate correctly.  Has this been tested?


-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1651519766
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

> Is a "key master" the entity that controls the key?
>
> I think we are using the word certificate in different ways. According to RFC 
> 4880, a certificate in OpenPGP (as a first approximation) is a Public Key 
> packet, zero or more direct key signatures, zero or more User ID packets with 
> any accompanying self signatures, and zero or more Subkey packets with any 
> accompanying self signatures. Is that how you are using the word certificate?

You are right, I am not on top of OpenPGP terminology, and I suspect much of 
the documentation floating around (including GPG's own manual) is not entirely 
aligned with it.  Thank you for being patient with me!

By *key master*, I mean the person (or team of people) who manage the signing 
keys at Google.

I think I was using the word *certificate* inconsistently, but mostly in the 
same way you are.  I think it's helpful to use examples to make 100% sure we 
are talking about the same thing.  Examples of certificates include:

 * Each item listed in the output of ``rpm -qa gpg-pubkey*`` contains a single 
PGP public key block.  As far as I know, each of these contains a single 
certificate.  For example, one of these on my system is 
``gpg-pubkey-38ab71f4-60242b08``.  The contents -- including subkeys -- can be 
viewed by typing:

   ```
   $ rpm -qi gpg-pubkey-38ab71f4-60242b08 | gpg --show-keys 
--with-subkey-fingerprint
   pub   rsa4096 2021-02-10 [SCE]
 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
   uid  Fedora (36) 
   ```

 * Another example is the [Google's Linux signing 
key](https://dl.google.com/linux/linux_signing_key.pub), which is actually a 
collection of public keys bundled inside a single certificate.  For the record, 
its contents are:

   ```
   $ cat linux_signing_key.pub | gpg --show-keys --with-subkey-fingerprint
   pub   dsa1024 2007-03-08 [SC]
 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
   uid  Google, Inc. Linux Package Signing Key 

   sub   elg2048 2007-03-08 [E]
 9534C9C4130B4DC9927992BF4F30B6B4C07CB649
   
   pub   rsa4096 2016-04-12 [SC]
 EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
   uid  Google Inc. (Linux Packages Signing Authority) 

   sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
 3B068FB4789ABE4AEFA3BB491397BC53640DB551
   sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
 3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
   sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
 2F528D36D67B69EDF998D85778BD65473CB3BD13
   sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
 8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
   sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
 A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2
   ```


> Can you explain what it means for "a key master" to "publish keys with 
> separate certificates"?

My first example above -- the Fedora 36 signing key -- is distributed in its 
own certificate.  Fedora 37 has a separate key and a separate certificate.  
This separation means that the user can install any combination of them at 
their leisure.  If for some reason they want to check old signatures, they can 
do so by installing old certificates containing old keys.  Since Fedora 
publishes keys in separate certificates, there is no need to merge Fedora's 
certificates.

On the other hand, Google publishes many keys within a single certificate (the 
second example).  If a new version of the certificate removes some old keys, 
this would prevent the user from verifying old signatures.  For example, the 
key that was issued on 2016-04-12 (and expired in 2019) might get removed from 
future versions of this certificate.  If this happens, then the user would have 
no obvious way of verifying packages signed by this key.  Your proposal of 
merging the new certificate with previously installed ones is one way of 
addressing this problem.  But I think it comes with a serious downside that the 
user has no way removing revoked keys.  If the 2016-04-12 key gets compromised, 
your proposal might leave the user vulnerable to attacker-signed packages.  
(The fact that they key has expired might help, but it's not the end of the 
story; e.g. what if a more recent key gets compromised?)

Sorry for the long letter.  But hopefully my writing (and my thinking!) is a 
bit clearer now.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1646809963
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

>> I suppose there might be some situations where that is helpful. But it 
>> sounds dangerous as a default behaviour. Old keys would never get retired / 
>> revoked. This could leave users vulnerable to attack. Otherwise, what's the 
>> point of rotating keys?
>
> I see. If I understand you correctly, you're suggesting that the old 
> component would be dropped. Are you suggesting that people resign their old 
> messages when they retire an signing subkey (that sounds like a lot of work)? 
> If not, how would a third party verify an old signature?

Yes, I think the old component should be dropped by default.

No, I don't think old packages should be re-signed by the new key.  Rather, I 
think that the key master ought to keep old keys inside the certificate unless 
they want to revoke it (e.g. due to an attack).  The old keys would still have 
expiry dates, so RPM would complain about old signatures -- but this can be 
overriden.

To reiterate, I think the common scenarios would be one of:

1. The key master keeps a long history of keys (as subkeys) inside the 
certificate.  This is what Google does.

2. The key master publishes keys with separate certificates, so they can be 
installed simultaneously.  I think this is the more common practice.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1646670658
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

> Can you explain what you are thinking or worried about here?

I'm not especially worried.  It just sounds like a lot of work!  But it sounds 
like you have already done the work, so there is no problem.

> >  Can you think of any use cases where this would cause a problem?
>
> Yes. If the new certificate is missing some components that the existing 
> version has, signatures that could once be verified may no longer be 
> verifiable.

I suppose there might be some situations where that is helpful.  But it sounds 
dangerous as a default behaviour.  Old keys would never get retired / revoked.  
This could leave users vulnerable to attack.  Otherwise, what's the point of 
rotating keys?


-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1646600750
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

@nwalfield, merging certificates sounds like a relatively hard problem to solve 
in general.  In this situation, if the user types

```
sudo rpm --import linux_signing_key.pub
```

then they probably just wants to replace the pre-existing key/certificate with 
the supplied one.

Can you think of any use cases where this would cause a problem?



-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1646543232
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

I should mention that I am not the first person to encounter this problem -- I 
imagine many people are affected.  See for example this Reddit thread: 





-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577#issuecomment-1646334035
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] rpm --import does not replace old keys with new keys (Issue #2577)

[Andrew Clausen]

# Encountering the problem with Google Linux Signing keys

When I try to install new Google Linux Signing keys in Fedora 37 (with the 
latest stable release of ``rpm``, namely 4.18.1), it fails.  However, when I 
delete the old (pre-installed) keys, and try again, it works.  Here is what I 
typed:

```
wget https://dl.google.com/linux/linux_signing_key.pub
sudo rpm --import linux_signing_key.pub
rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint
rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} %{PACKAGER}\n' | grep 
'linux-packages-keymas...@google.com' | sed 's/ .*$//' | sudo xargs rpm -e
sudo rpm --import linux_signing_key.pub
rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys --with-subkey-fingerprint
```

Here is the output.  Notice that the first attempt at importing the keys fails 
silently, whereas the second attempt (after deleting old keys) succeeds.

```
[user@disp8080 ~]$ wget https://dl.google.com/linux/linux_signing_key.pub
--2023-07-21 23:26:12--  https://dl.google.com/linux/linux_signing_key.pub
Resolving dl.google.com (dl.google.com)... 142.250.178.14, 
2a00:1450:4009:81f::200e
Connecting to dl.google.com (dl.google.com)|142.250.178.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14605 (14K) [application/octet-stream]
Saving to: ‘linux_signing_key.pub’

linux_signing_key.p 100%[===>]  14.26K  --.-KB/sin 0s  

2023-07-21 23:26:14 (43.0 MB/s) - ‘linux_signing_key.pub’ saved [14605/14605]

[user@disp8080 ~]$ sudo rpm --import linux_signing_key.pub
[user@disp8080 ~]$ rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys 
--with-subkey-fingerprint
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
pub   rsa4096 2016-04-12 [SC]
  EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
uid  Google Inc. (Linux Packages Signing Authority) 

sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
  3B068FB4789ABE4AEFA3BB491397BC53640DB551
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
  3E50F6D3EC278FDEB655C8CA6494C6D6997C215E

[user@disp8080 ~]$ rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} 
%{PACKAGER}\n' | grep 'linux-packages-keymas...@google.com' | sed 's/ .*$//' | 
sudo xargs rpm -e
[user@disp8080 ~]$ sudo rpm --import linux_signing_key.pub
[user@disp8080 ~]$ rpm -qi gpg-pubkey-d38b4796-* | gpg --show-keys 
--with-subkey-fingerprint
pub   rsa4096 2016-04-12 [SC]
  EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
uid  Google Inc. (Linux Packages Signing Authority) 

sub   rsa4096 2016-04-12 [S] [expired: 2019-04-12]
  3B068FB4789ABE4AEFA3BB491397BC53640DB551
sub   rsa4096 2017-01-24 [S] [expired: 2020-01-24]
  3E50F6D3EC278FDEB655C8CA6494C6D6997C215E
sub   rsa4096 2019-07-22 [S] [expired: 2022-07-21]
  2F528D36D67B69EDF998D85778BD65473CB3BD13
sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
  8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
sub   rsa4096 2023-02-15 [S] [expires: 2026-02-14]
  A5F483CD733A4EBAEA378B2AE88979FB9B30ACF2
```

# The root of the problem

Google's Linux Signing keys have an unusual configuration: they retain the same 
master key, but replace the subkeys which do the actual signing.  Since ``rpm`` 
identifies the whole collection of keys by the master key, the new key looks 
like the pre-existing key.  So it ignores it.

# Work around

The work around -- as alluded to above -- is to delete the old keys before 
trying to import the new ones.  This can be done with:

```
rpm -qa gpg-pubkey* --qf '%{NAME}-%{VERSION}-%{RELEASE} %{PACKAGER}\n' | grep 
'linux-packages-keymas...@google.com' | sed 's/ .*$//' | sudo xargs rpm -e
```

# Discussion

This bug is potentially quite serious.  Millions of Linux users use Google 
Chrome, and right now, there is no way for them to install or upgrade it.  They 
will just see that signature verification fails.  Most people will give up, or 
worse, disable signature checks.   Those that try to check the signature 
manually will be thwarted by the same bug again.

This means there is a serious usability impact (not being able to install 
Google Chrome), as well as a serious security impact (not getting security 
updates, or installing unverified updates for Google Chrome).

# Solution

Please either fix ``rpm`` so that it correctly imports keys (e.g. by using a 
timestamp as the key version), or deprecate the use of subkeys.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/2577
You are receiving this because you are subscribed to this thread.

Message ID: ___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint