Re: Transparency into private keys of Debian
Hans-Christoph Steiner writes: >> In business, such things are confirmed (often badly) by independent >> audit. For a volunteer-driven community effort, we have to rely on >> everyone to exercise their best judgement in these sorts of matters. > > Debian could also get independent, professional audits. I think it > would be a good use of the Debian pot of money, for example. Or > someone could submit a proposal to get Debian audited. I'll be either > Open Tech Fund or NLnet would do it: > > https://www.opentech.fund/labs/red-team-lab/ > > Open Tech Fund already funds Tails, which is based on Debian. That would be useful for the important keys like the apt release keys, and would set an example for others to follow. If there are things to improve, it would be better if we know about them than allowing attackers to find out on their own. For DD keys, as Jemery noticed, I don't think it is useful: uses of DD keys leave a quite auditable flow already. /Simon signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
On 2024-02-08 23:44:21 +0100 (+0100), Hans-Christoph Steiner wrote: > > In business, such things are confirmed (often badly) by independent > > audit. For a volunteer-driven community effort, we have to rely on > > everyone to exercise their best judgement in these sorts of matters. > > Debian could also get independent, professional audits. [...] Perhaps of a sort, but not the kind I'm accustomed to being performed in the corporate world. Having auditors visit every DD's home to watch them upload packages and confirm they're following the claimed secure workflows seems entirely intractable. Sure you could ask every DD to fill out a questionnaire, but if you don't trust them to all follow documented practices then why would you trust them to accurately answer survey questions either? -- Jeremy Stanley signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
> In business, such things are confirmed (often badly) by independent > audit. For a volunteer-driven community effort, we have to rely on > everyone to exercise their best judgement in these sorts of matters. Debian could also get independent, professional audits. I think it would be a good use of the Debian pot of money, for example. Or someone could submit a proposal to get Debian audited. I'll be either Open Tech Fund or NLnet would do it: https://www.opentech.fund/labs/red-team-lab/ Open Tech Fund already funds Tails, which is based on Debian. .hc
Re: Transparency into private keys of Debian
On 2/1/24 10:38, Simon Josefsson wrote: Hi I'm exploring how to defend against an attacker who can create valid signatures for cryptographic private keys (e.g., PGP) that users need to trust when using an operating system such as Debian. A signature like that can be used in a targetted attacks against one victim. For example, apt does not have any protection against this threat scenario, and often unprotected ftp:// or http:// transports are used, which are easy to man-in-the-middle. Even with https:// there is a large number of mirror sites out there that can replace content you get. There is a risk that use of a compromised trusted apt PGP key will not be noticed. Attackers are also in a good position to deny themselves out of their actions if they are careful. hello, thanks for raising this. I've spent some time exploring this in 2022 and figured, though I don't have the resources to do anything meaningful about this, I can still help by making reasoning about this attack easier by implementing it. I authored a malicious update server for multiple package managers including apt (the contrib/ folder has a few examples, some of them being incomplete or defunct tho, please do not make assumptions unless you tested them, many of them have a `check:` section for automatic integration testing): https://github.com/kpcyrd/sh4d0wup The name is derived from "shadow updates" that carry a valid signature (through private key abuse) but are fed directly to an update client and therefore never show up in Debian. It has some features to illustrate how I would assume targeted attacks to work (switch the Release/InRelease files based on source IP address and/or user-agent, leading them to a different /by-hash/ url that nobody else knows about). From what I've learned the most interesting keys are (for each respective release): - Debian Archive Automatic Signing Key - Debian Security Archive Automatic Signing Key - Debian Stable Release Key The DM/DD keys probably also have some value, but would leave a permanent record in Debian which might get noticed at some (distant or not so distant) point in the future, however when abusing release signing keys you are almost guaranteed to get away with it. --- Having built all of this (and also hence having proven writing the software for this is well within reach of a single, semi-determined rando), I then spent some time in 2023 trying to think up a system that is as annoying as possible for sh4d0wup users and developed apt-swarm. It's my first attempt of an autonomous p2p network and it uses the configured public-keys for a "proof of authority", gossip-sync protocol to attempt an eventually-consistent view of all Release/InRelease files. However: - I need an embedded database with specific properties for the sync protocol (allowing fast, ordered lookups by key-prefix), I picked the `sled` library as suggested by a debian-rust friend at FOSDEM 2023, but I later learned sled needs to be able to fit the entire database into RAM, so running a node currently requires ~8GB RAM (and *will* require more in the future). - Because of this, the entire p2p network is currently running from a single server (that is provided to me by a friend from a local hackerspace). You can join your own nodes at any time but so far nobody has done so. - Despite it's name, it doesn't know about apt and only stores opaque signed documents, the Release/InRelease files. It does not store any copies of any referenced package indexes (or packages), which limits the amount of data you get to triage incidents with. The last part is important because availability of this kind of data is going to set the scene in case you ever need to actually investigate something. I found that rekor from the sigstore project is a great public resource, but will only tell you "there's a signature you don't have any records of", but nowhere to go from there. It doesn't complement the snapshot service and it's impossible to tell apart if somebody generated themselves the inclusion-proof for a malicious update or if snapshot.debian.org simply missed/lost a snapshot. With apt-swarm you can look at the alleged `Date:` field of the release file but you will (likely) still be missing the package index that was fed into user's apt clients (and the code they executed) unless you recognize the referenced hashes. Development of this project has stalled due to the mentioned RAM usage issue, little interest for this kind of tech (even inside the security community) and cyberpunk 2077 being a really good game. Throughout the year apt-swarm has collected (among others): - 20,459 signatures from `Debian Archive Automatic Signing Key (10/buster)` - 18,085 signatures from `Debian Archive Automatic Signing Key (9/stretch)` - 10,463 signatures from `Debian Archive Automatic Signing Key (11/bullseye)` - 6,843 signatures from `Debian Security Archive Automatic Signing Key
Re: Transparency into private keys of Debian
> > I've looked at Sigstore, it looks nice. It seems to be architected > > for use > > cases that assume highly reliable and unblocked single domains. > > That's a > > showstopper for us. Also, the official client app is 100% JVM code > > right now > > (Java+Kotlin), so integrating Go binaries would really be an option > > of last > > resort. Its almost a no go for many reasons. > > Agreed -- having a C, perl or python client for Sigstore or Sigsum > would be nice. I'll bring this up with them. I should have checked before sending the previous email, there is this client: https://git.glasklar.is/sigsum/core/sigsum-py I suppose for apt python is more relevant, and if the sigsum proof is included alongside apt metadata, it would allow offline verification inside apt. If apt doesn't support this natively, the proof would have to be distributed through some other channel, but that is relatively easy to do as a Proof-of-Concept (i.e., how apt-sigstore + debdistcanary already works via gitlab) but scaling it to decentralized etc remains to be resolved, if relevant. /Simon > > /Simon > > > > > Hans > > > > > > > > /Simon > > > > > > > And publishing the jurisdictions could be enough info to > > > > deanonymize > > > > the key holder, e.g. if it is Germany, then there are many DDs > > > > there. > > > > If it is Iceland, then govs can easily find who to target. > > > > > > > > .hc > > > > > > > > > Hi > > > > > I'm exploring how to defend against an attacker who can > > > > > create > > > > > valid > > > > > signatures for cryptographic private keys (e.g., PGP) that > > > > > users need to > > > > > trust when using an operating system such as Debian. A > > > > > signature like > > > > > that can be used in a targetted attacks against one victim. > > > > > For example, apt does not have any protection against this > > > > > threat > > > > > scenario, and often unprotected ftp:// or http:// transports > > > > > are used, > > > > > which are easy to man-in-the-middle. Even with https:// > > > > > there > > > > > is a > > > > > large number of mirror sites out there that can replace > > > > > content > > > > > you get. > > > > > There is a risk that use of a compromised trusted apt PGP key > > > > > will not > > > > > be noticed. Attackers are also in a good position to deny > > > > > themselves > > > > > out of their actions if they are careful. > > > > > Part of my effort is to inventory all trusted private keys > > > > > that > > > > > a > > > > > distribution needs to manage on their own, or depend on that > > > > > are managed > > > > > by others, and gain some insight how these keys are handled. > > > > > Does the Debian project, or any members, publish information > > > > > on > > > > > these > > > > > topics? Has this been discussed before? > > > > > Some questions that I think would be useful to answer > > > > > include: > > > > > 1) List of relevant private keys, held by the organization, > > > > > its > > > > > members, > > > > > or someone else. As far as I can tell from Debian, the > > > > > list includes > > > > > the following PGP trust anchors: > > > > > B8B8 0B5B 623E AB6A D877 5C45 B7C5 D7D6 3509 47F8 > > > > > 05AB 9034 0C0C 5E79 7F44 A8C8 254C F3B5 AEC0 A8F0 > > > > > 4D64 FEC1 19C2 0290 67D6 E791 F8D2 585B 8783 D481 > > > > > 1F89 983E 0081 FDE0 18F3 CC96 73A4 F27B 8DD4 7936 > > > > > AC53 0D52 0F2F 3269 F5E9 8313 A484 4904 4AAD 5C5D > > > > > A428 5295 FC7B 1A81 6000 62A9 605C 66F0 0D6C 9793 > > > > > 80D1 5823 B7FD 1561 F9F7 BCDD DC30 D7C2 3CBB ABEE > > > > > 5E61 B217 265D A980 7A23 C5FF 4DFA B270 CAA9 6DFA > > > > > 6D33 866E DD8F FA41 C014 3AED DCC9 EFBF 77E1 1517 > > > > > Compromising any single key on this list leads to > > > > > trouble. > > > > > However I > > > > > don't think this list is complete. What other keys are > > > > > there? > > > > > I believe there are keys used to sign some component of > > > > > the > > > > > boot > > > > > phase, compare the 'shim-signed' and 'grub-efi-amd64- > > > > > signed' > > > > > packages. What private keys held by Debian are involved > > > > > here? What > > > > > keys held by others are involved? What other similar > > > > > packages > > > > > exists? > > > > > 2) For each private key, information about its management and > > > > > lifecycle. > > > > > Relevant questions include: > > > > > a) How was the key generated? By whom? On what hardware? > > > > > What > > > > > software? In what environment? What legal jurisdiction > > > > > apply to > > > > > people involved? > > > > > b) How is the key stored and protected during its > > > > > lifetime? > > > > > What > > > > > media > > > > > is used? Who control the physical storage of the key? > > > > > How are they > > > > > stored and transported? What jurisdiction? > > > > > c) Under what policy is the key used?
Re: Transparency into private keys of Debian
tis 2024-02-06 klockan 16:50 +0100 skrev Hans-Christoph Steiner: > > > Simon Josefsson: > > Hans-Christoph Steiner writes: > > > > > Thanks for digging in here, its very important work! I'd be > > > happy to > > > contribute where I can. I'm a DD and a core contributor to F- > > > Droid, > > > where we wrestle with basically the same issues. So we've > > > thought a > > > lot about these kinds of things, but definitely do not have all > > > the > > > answers. Since F-Droid started much later than Debian, we were > > > able > > > to build in some nice protections from the beginning, like > > > requiring > > > HTTPS. We've also made the decision to stick with a single > > > jurisdiction for the singing keys, even though it is not the best > > > one > > > for our needs. We'd of course love to move them to the best > > > jurisdiction but that is actually quite a bit of work, so we have > > > to > > > stay grounded in reality. > > > > > > For me the hard part of all this is how to be transparent as > > > possible > > > without putting a giant target on the heads of the people who > > > control > > > the keys. I think it is clear that publishing private key usage > > > information is safe, like this key signed these things at these > > > times. > > > Publishing all the details about how the key was generated could > > > help > > > those who want to attack it more than those who rely on it. But > > > that > > > kind of information would be good to share internally to the > > > project > > > at the very least. > > > > Good aspect -- and indeed this is a trade-off, and thanks for > > caring > > about these issues for f-droid! It seems that if you would verify > > signatures via a transparency log, you would reduce the risk on the > > people controlling the keys? Then you (and they) could feel more > > comfortable publishing information how your process work. That > > would be > > valuable to publish (even de-personalized or generalized) as a best > > practices approach. The reason is that anyone stealing the keys > > from > > these persons would ALSO have to upload signatures of whatever they > > wanted to use these keys for into the transparency log, which would > > provide evidence to what they did, so they may be less likely to > > follow > > through on their plans. > > Yes, I agree, and am familiar with this architecture. The hard part > is the > availability of the transparency log needs to be as good as the > signed index's > availability, otherwise attackers can just block the transparency log > to stop > the whole process. The official F-Droid client does a lot of things > automatically in order to find a working mirror. And we're expanding > on that. > I have yet to see a transparency log set up for decentralized, > flexible > distribution. F-Droid's mirror handling will automatically choose > from: > > * f-droid.org > * Any of the official mirrors > * Any mirrors that the user has added locally > * Files hosted on IPFS > * Mirrors on local storage (SD cards, thumb drives, etc) > > We're expanding to: > * Mirrors on cloud file storage like Nextcloud, Google Drive, etc. > * Signed JSON on public code distribution platforms and CDNs I believe sigsum allows for offline verification of the inclusion proof. Just include the transparency checksum proof with the app or meta-data around the app. Not sure about Sigstore. > (As a side note: I would like to see Debian/apt adopt this approach, > at least > optionally. The key part is automatic operation and dead simple > setup, e.g. for > non-technical users.) +1 > > What would be involved is to 1) during signing of artifacts, also > > sign > > and upload into Sigstore/Sigsum, and 2) during verification in the > > f-droid app, also verify that the signature has been committed to > > the > > Sigstore/Sigsum logs. Both projects have clients written in Go > > which > > should work on Android, but the rest of the details are sketchy to > > me. > > I'm happy to continue discuss and help with design if you are > > interested, to understand what the limitations of your environments > > are > > and how to resolve them. > > I've looked at Sigstore, it looks nice. It seems to be architected > for use > cases that assume highly reliable and unblocked single domains. > That's a > showstopper for us. Also, the official client app is 100% JVM code > right now > (Java+Kotlin), so integrating Go binaries would really be an option > of last > resort. Its almost a no go for many reasons. Agreed -- having a C, perl or python client for Sigstore or Sigsum would be nice. I'll bring this up with them. /Simon > > Hans > > > > > /Simon > > > > > And publishing the jurisdictions could be enough info to > > > deanonymize > > > the key holder, e.g. if it is Germany, then there are many DDs > > > there. > > > If it is Iceland, then govs can easily find who to target. > > > > > > .hc > > > > > > > Hi > > > > I'm exploring how to defend
Re: Transparency into private keys of Debian
On 2024-02-06 14:35:34 +0800 (+0800), Simon khng wrote: [...] > 3) Use 'password enabled key store' to prevent unauthorized access to > digital keys. > 4) Use 'password enabled signing' to prevent unauthorized usage of digital > keys. > The use of number 3 and 4 are the steps for developers to upload > application packages as part of the > verification process by Google for the 'Play store' used in Android OS > devices. [...] This is already a standard practice for anyone using OpenPGP/GnuPG keys. I personally know no one who keeps their developer private keys or subkeys unencrypted. That said, it's not enforceable (or detectable), so it can at best only be part of documented workflow process. There is no way to actually guarantee everyone follows it. In business, such things are confirmed (often badly) by independent audit. For a volunteer-driven community effort, we have to rely on everyone to exercise their best judgement in these sorts of matters. -- Jeremy Stanley signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
Simon Josefsson: Hans-Christoph Steiner writes: Thanks for digging in here, its very important work! I'd be happy to contribute where I can. I'm a DD and a core contributor to F-Droid, where we wrestle with basically the same issues. So we've thought a lot about these kinds of things, but definitely do not have all the answers. Since F-Droid started much later than Debian, we were able to build in some nice protections from the beginning, like requiring HTTPS. We've also made the decision to stick with a single jurisdiction for the singing keys, even though it is not the best one for our needs. We'd of course love to move them to the best jurisdiction but that is actually quite a bit of work, so we have to stay grounded in reality. For me the hard part of all this is how to be transparent as possible without putting a giant target on the heads of the people who control the keys. I think it is clear that publishing private key usage information is safe, like this key signed these things at these times. Publishing all the details about how the key was generated could help those who want to attack it more than those who rely on it. But that kind of information would be good to share internally to the project at the very least. Good aspect -- and indeed this is a trade-off, and thanks for caring about these issues for f-droid! It seems that if you would verify signatures via a transparency log, you would reduce the risk on the people controlling the keys? Then you (and they) could feel more comfortable publishing information how your process work. That would be valuable to publish (even de-personalized or generalized) as a best practices approach. The reason is that anyone stealing the keys from these persons would ALSO have to upload signatures of whatever they wanted to use these keys for into the transparency log, which would provide evidence to what they did, so they may be less likely to follow through on their plans. Yes, I agree, and am familiar with this architecture. The hard part is the availability of the transparency log needs to be as good as the signed index's availability, otherwise attackers can just block the transparency log to stop the whole process. The official F-Droid client does a lot of things automatically in order to find a working mirror. And we're expanding on that. I have yet to see a transparency log set up for decentralized, flexible distribution. F-Droid's mirror handling will automatically choose from: * f-droid.org * Any of the official mirrors * Any mirrors that the user has added locally * Files hosted on IPFS * Mirrors on local storage (SD cards, thumb drives, etc) We're expanding to: * Mirrors on cloud file storage like Nextcloud, Google Drive, etc. * Signed JSON on public code distribution platforms and CDNs (As a side note: I would like to see Debian/apt adopt this approach, at least optionally. The key part is automatic operation and dead simple setup, e.g. for non-technical users.) What would be involved is to 1) during signing of artifacts, also sign and upload into Sigstore/Sigsum, and 2) during verification in the f-droid app, also verify that the signature has been committed to the Sigstore/Sigsum logs. Both projects have clients written in Go which should work on Android, but the rest of the details are sketchy to me. I'm happy to continue discuss and help with design if you are interested, to understand what the limitations of your environments are and how to resolve them. I've looked at Sigstore, it looks nice. It seems to be architected for use cases that assume highly reliable and unblocked single domains. That's a showstopper for us. Also, the official client app is 100% JVM code right now (Java+Kotlin), so integrating Go binaries would really be an option of last resort. Its almost a no go for many reasons. Hans /Simon And publishing the jurisdictions could be enough info to deanonymize the key holder, e.g. if it is Germany, then there are many DDs there. If it is Iceland, then govs can easily find who to target. .hc Hi I'm exploring how to defend against an attacker who can create valid signatures for cryptographic private keys (e.g., PGP) that users need to trust when using an operating system such as Debian. A signature like that can be used in a targetted attacks against one victim. For example, apt does not have any protection against this threat scenario, and often unprotected ftp:// or http:// transports are used, which are easy to man-in-the-middle. Even with https:// there is a large number of mirror sites out there that can replace content you get. There is a risk that use of a compromised trusted apt PGP key will not be noticed. Attackers are also in a good position to deny themselves out of their actions if they are careful. Part of my effort is to inventory all trusted private keys that a distribution needs to manage on their own, or depend on that are managed by others, and
Re: Transparency into private keys of Debian
Hello there, I have read a little on this discussion and feel like sharing my thoughts. I think the current lacking procedures are number 3 and 4 from my summarization based on the current standards adopted for PKI: 1) Chain of trust from developer, [intermediaries,] to root CA. 2) Ensure multiple signing of packages at the above layers (intermediaries, root). |__This allows for a buffer for key revocation whether by user choice or the other party holding the authority (intermediaries, root). 3) Use 'password enabled key store' to prevent unauthorized access to digital keys. 4) Use 'password enabled signing' to prevent unauthorized usage of digital keys. The use of number 3 and 4 are the steps for developers to upload application packages as part of the verification process by Google for the 'Play store' used in Android OS devices. I am not sure about the others like Apple, Windows, Amazon etc, but they all probably have the same process. Nothing new is being invented here and none being reinvented to complicate matters. The existing security framework is quite simple yet sophisticated which is probably good. Cheers, Simon khng On Tue, Feb 6, 2024 at 1:12 AM Stephan Verbücheln wrote: > Your work is valuable. Many of the things have probably evolved over > time and could use some analysis based on modern cryptography and > security practices. I just wanted to point out that there are subtle > but important differences outside of the key and signature formats. > > The most important distinction is probably the one of personal keys on > the one hand which purpose is to identify a developer and the Release > keys which are stored on some build servers to create Release files. > > You cannot only have PGP keys signing each other (like CAs and leaf > certificates in X.509 PKI). PGP has subkeys and they could be used in > the release process to mitigate risks. > > Example: > 1. Debian creates a PGP key for releases. > 2. The public key is installed in Debian to verify releases. > 3. The release team creates subkeys for signing. > 4. The main private key is stored in a restricted place. > 5. The build server only uses the subkeys to sign releases. > > The subkeys could be expired and rotated all the time without changing > the PGP fingerprint and therefore without changing the trusted key ring > in the Debian installations. I do not know whether Debian actually > makes use of this, or it has been discussed before. > > Regards > Stephan >
Re: Transparency into private keys of Debian
On 2024-02-05 08:58, Simon Josefsson wrote: What would be involved is to 1) during signing of artifacts, also sign and upload into Sigstore/Sigsum, and 2) during verification in the f-droid app, also verify that the signature has been committed to the Sigstore/Sigsum logs. Both projects have clients written in Go which should work on Android, but the rest of the details are sketchy to me. I'm happy to continue discuss and help with design if you are interested, to understand what the limitations of your environments are and how to resolve them. One weirdness with the release keys we use is that they are technically able to sign independently, but practically only ever appear next to the archive signing key. That gives us an escape hatch to update the key set when we managed to lose the archive key, I guess. Obviously you'd need to have an efficient way to test against a transparency log - I don't think it's sufficient for just tracking the signatures if your worry is that someone MITMs you with a malicious signature. I'll note that Firefox also still does not implement Certificate Transparency checks. (Which I find quite surprising and makes it less secure in my book.) Kind regards Philipp Kern
Re: Transparency into private keys of Debian
Your work is valuable. Many of the things have probably evolved over time and could use some analysis based on modern cryptography and security practices. I just wanted to point out that there are subtle but important differences outside of the key and signature formats. The most important distinction is probably the one of personal keys on the one hand which purpose is to identify a developer and the Release keys which are stored on some build servers to create Release files. You cannot only have PGP keys signing each other (like CAs and leaf certificates in X.509 PKI). PGP has subkeys and they could be used in the release process to mitigate risks. Example: 1. Debian creates a PGP key for releases. 2. The public key is installed in Debian to verify releases. 3. The release team creates subkeys for signing. 4. The main private key is stored in a restricted place. 5. The build server only uses the subkeys to sign releases. The subkeys could be expired and rotated all the time without changing the PGP fingerprint and therefore without changing the trusted key ring in the Debian installations. I do not know whether Debian actually makes use of this, or it has been discussed before. Regards Stephan signature.asc Description: This is a digitally signed message part
Re: Transparency into private keys of Debian
Stephan Verbücheln writes: > II. Typical Debian case > > 1. Debian developer signs source tarballs and upload them > 2. The signature only has to be secure until the code lands in the FTP > 3. Debian builds the binary packages > 4. Debian creates Release files with hashes of the packages > 5. The Release file is signed by Debian APT keys > 6. The signature has to be secure until the next Release file > 7. Security updates have a separate Release file with expiration time > > This strategy effectively prevents attackers from injecting outdated > programs with known vulnerabilities into the updates. Even mirrors > cannot do that. TLS for mirrors is not required for integrity and > authenticity. Thanks for summarizing. I'm not concerned with injecting outdated programs -- I'm concerned with injecting malware indirectly signed by the Debian APT keys, and want to explore ways to mitigate against those attacks. Understanding the private key lifecycle is one way to increase confidence in what the cost of those attacks are -- right now all we can say is that the cost of forging signatures for Debian APT keys is likely higher than $0 and is likely lower than, let's say, $100B to pay someone to get the keys or inject code for malicious remote signing. Tightening these bounds, using some rational methodology, gives us knowledge about safety. Using transparency logging of the signatures created by the private keys is another way to allow detection when this occur, which also increase confidence in the overall security, and reduce the risk with the private keys (since they are longer useful for deniable attacks). It may be that publishing more details about the private key lifecycle is not a good idea unless we have measurements to protect ourselves, but at least establishing that fact would be an improvement, and an incentive to invest in these improvements. /Simon signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
Code signing is not equal to code signing. There are a lot of differences between different code-signing strategies, many of which are often overlooked. Example: I. Typical Windows case 1. Third-party developer gets a key from a CA. 2. Third-party developer signs a program binary. 3. The user obtains the program. 4. The signature is checked at execution time. 5. The signature has to be secure for many years if not decades. 6. The user has to figure out whether he has the latest version. 7. Revocation is an unreliable mess. One could even say that this strategy of code signing is literally useless, because malware can always just exploit a signed binary of an outdated version with known vulnerabilities. II. Typical Debian case 1. Debian developer signs source tarballs and upload them 2. The signature only has to be secure until the code lands in the FTP 3. Debian builds the binary packages 4. Debian creates Release files with hashes of the packages 5. The Release file is signed by Debian APT keys 6. The signature has to be secure until the next Release file 7. Security updates have a separate Release file with expiration time This strategy effectively prevents attackers from injecting outdated programs with known vulnerabilities into the updates. Even mirrors cannot do that. TLS for mirrors is not required for integrity and authenticity. Secure Boot is a different issue. From the security vulnerabilities which are being published, it looks to me that it has the same problem with malware exploiting vulnerable binaries with valid signatures. Regards Stephan PS: Another related discussion is about very old package uploads and the corresponding (expired, inactive, weak, etc.) developer signatures. https://lists.debian.org/debian-devel/2023/12/msg0.html signature.asc Description: This is a digitally signed message part
Re: Transparency into private keys of Debian
Bill Allombert writes: > On Mon, Feb 05, 2024 at 08:49:09AM +0100, Simon Josefsson wrote: >> Bill Allombert writes: >> >> > Le Thu, Feb 01, 2024 at 10:38:03AM +0100, Simon Josefsson a écrit : >> >> Hi >> >> >> >> I'm exploring how to defend against an attacker who can create valid >> >> signatures for cryptographic private keys (e.g., PGP) that users need to >> >> trust when using an operating system such as Debian. A signature like >> >> that can be used in a targetted attacks against one victim. >> >> >> >> For example, apt does not have any protection against this threat >> >> scenario, >> > >> > Is not apt-key a protection ? >> >> No, the current implementation protects against missing and/or invalid >> signatures. Compare how in the WebPKI world some CA issued a valid >> *.google.com certificate, and how that (and other incidents) lead to >> setup of Certificate Transparency, which helps mitigate these issues. > > The difference is that with apt-key, the list of valid public keys is stored > on > the user system (in /etc/apt/trusted.gpg.d/), not a list of root certificates, > and that the users are notified when the keys are updated, which is not the > case with CA. Nobody can generate a new signature that will be accepted by > apt-key out of the box. Right, there are differences, but I believe the underlying problem is the same: if someone mis-use private keys I trust, or is able to forge valid signatures somehow (e.g., crypto attack), apt-key wouldn't notice but in the WebPKI world there are mechanisms to mitigate this via Certificate Transparency. It is a subjective choice to care about this attack scenario, and some may regard it as out of scope, but others can believe it is in scope and that improvements are warranted. /Simon signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
On Mon, Feb 05, 2024 at 08:49:09AM +0100, Simon Josefsson wrote: > Bill Allombert writes: > > > Le Thu, Feb 01, 2024 at 10:38:03AM +0100, Simon Josefsson a écrit : > >> Hi > >> > >> I'm exploring how to defend against an attacker who can create valid > >> signatures for cryptographic private keys (e.g., PGP) that users need to > >> trust when using an operating system such as Debian. A signature like > >> that can be used in a targetted attacks against one victim. > >> > >> For example, apt does not have any protection against this threat > >> scenario, > > > > Is not apt-key a protection ? > > No, the current implementation protects against missing and/or invalid > signatures. Compare how in the WebPKI world some CA issued a valid > *.google.com certificate, and how that (and other incidents) lead to > setup of Certificate Transparency, which helps mitigate these issues. The difference is that with apt-key, the list of valid public keys is stored on the user system (in /etc/apt/trusted.gpg.d/), not a list of root certificates, and that the users are notified when the keys are updated, which is not the case with CA. Nobody can generate a new signature that will be accepted by apt-key out of the box. Cheers, -- Bill. Imagine a large red swirl here.
Re: Transparency into private keys of Debian
Hans-Christoph Steiner writes: > Thanks for digging in here, its very important work! I'd be happy to > contribute where I can. I'm a DD and a core contributor to F-Droid, > where we wrestle with basically the same issues. So we've thought a > lot about these kinds of things, but definitely do not have all the > answers. Since F-Droid started much later than Debian, we were able > to build in some nice protections from the beginning, like requiring > HTTPS. We've also made the decision to stick with a single > jurisdiction for the singing keys, even though it is not the best one > for our needs. We'd of course love to move them to the best > jurisdiction but that is actually quite a bit of work, so we have to > stay grounded in reality. > > For me the hard part of all this is how to be transparent as possible > without putting a giant target on the heads of the people who control > the keys. I think it is clear that publishing private key usage > information is safe, like this key signed these things at these times. > Publishing all the details about how the key was generated could help > those who want to attack it more than those who rely on it. But that > kind of information would be good to share internally to the project > at the very least. Good aspect -- and indeed this is a trade-off, and thanks for caring about these issues for f-droid! It seems that if you would verify signatures via a transparency log, you would reduce the risk on the people controlling the keys? Then you (and they) could feel more comfortable publishing information how your process work. That would be valuable to publish (even de-personalized or generalized) as a best practices approach. The reason is that anyone stealing the keys from these persons would ALSO have to upload signatures of whatever they wanted to use these keys for into the transparency log, which would provide evidence to what they did, so they may be less likely to follow through on their plans. What would be involved is to 1) during signing of artifacts, also sign and upload into Sigstore/Sigsum, and 2) during verification in the f-droid app, also verify that the signature has been committed to the Sigstore/Sigsum logs. Both projects have clients written in Go which should work on Android, but the rest of the details are sketchy to me. I'm happy to continue discuss and help with design if you are interested, to understand what the limitations of your environments are and how to resolve them. /Simon > And publishing the jurisdictions could be enough info to deanonymize > the key holder, e.g. if it is Germany, then there are many DDs there. > If it is Iceland, then govs can easily find who to target. > > .hc > >> Hi >> I'm exploring how to defend against an attacker who can create valid >> signatures for cryptographic private keys (e.g., PGP) that users need to >> trust when using an operating system such as Debian. A signature like >> that can be used in a targetted attacks against one victim. >> For example, apt does not have any protection against this threat >> scenario, and often unprotected ftp:// or http:// transports are used, >> which are easy to man-in-the-middle. Even with https:// there is a >> large number of mirror sites out there that can replace content you get. >> There is a risk that use of a compromised trusted apt PGP key will not >> be noticed. Attackers are also in a good position to deny themselves >> out of their actions if they are careful. >> Part of my effort is to inventory all trusted private keys that a >> distribution needs to manage on their own, or depend on that are managed >> by others, and gain some insight how these keys are handled. >> Does the Debian project, or any members, publish information on >> these >> topics? Has this been discussed before? >> Some questions that I think would be useful to answer include: >> 1) List of relevant private keys, held by the organization, its >> members, >>or someone else. As far as I can tell from Debian, the list includes >>the following PGP trust anchors: >> B8B8 0B5B 623E AB6A D877 5C45 B7C5 D7D6 3509 47F8 >> 05AB 9034 0C0C 5E79 7F44 A8C8 254C F3B5 AEC0 A8F0 >> 4D64 FEC1 19C2 0290 67D6 E791 F8D2 585B 8783 D481 >> 1F89 983E 0081 FDE0 18F3 CC96 73A4 F27B 8DD4 7936 >> AC53 0D52 0F2F 3269 F5E9 8313 A484 4904 4AAD 5C5D >> A428 5295 FC7B 1A81 6000 62A9 605C 66F0 0D6C 9793 >> 80D1 5823 B7FD 1561 F9F7 BCDD DC30 D7C2 3CBB ABEE >> 5E61 B217 265D A980 7A23 C5FF 4DFA B270 CAA9 6DFA >> 6D33 866E DD8F FA41 C014 3AED DCC9 EFBF 77E1 1517 >>Compromising any single key on this list leads to trouble. >> However I >>don't think this list is complete. What other keys are there? >>I believe there are keys used to sign some component of the boot >>phase, compare the 'shim-signed' and 'grub-efi-amd64-signed' >>packages. What private keys held by Debian are involved here? What >>keys
Re: Transparency into private keys of Debian
Bill Allombert writes: > Le Thu, Feb 01, 2024 at 10:38:03AM +0100, Simon Josefsson a écrit : >> Hi >> >> I'm exploring how to defend against an attacker who can create valid >> signatures for cryptographic private keys (e.g., PGP) that users need to >> trust when using an operating system such as Debian. A signature like >> that can be used in a targetted attacks against one victim. >> >> For example, apt does not have any protection against this threat >> scenario, > > Is not apt-key a protection ? No, the current implementation protects against missing and/or invalid signatures. Compare how in the WebPKI world some CA issued a valid *.google.com certificate, and how that (and other incidents) lead to setup of Certificate Transparency, which helps mitigate these issues. It is possible to implement similar features for the relevant private keys used to sign Debian too; Sigstore and Sigsum are two publicly available projects. /Simon signature.asc Description: PGP signature
Re: Transparency into private keys of Debian
Le Thu, Feb 01, 2024 at 10:38:03AM +0100, Simon Josefsson a écrit : > Hi > > I'm exploring how to defend against an attacker who can create valid > signatures for cryptographic private keys (e.g., PGP) that users need to > trust when using an operating system such as Debian. A signature like > that can be used in a targetted attacks against one victim. > > For example, apt does not have any protection against this threat > scenario, Is not apt-key a protection ? Cheers, -- Bill. Imagine a large red swirl here.
Re: Transparency into private keys of Debian
Thanks for digging in here, its very important work! I'd be happy to contribute where I can. I'm a DD and a core contributor to F-Droid, where we wrestle with basically the same issues. So we've thought a lot about these kinds of things, but definitely do not have all the answers. Since F-Droid started much later than Debian, we were able to build in some nice protections from the beginning, like requiring HTTPS. We've also made the decision to stick with a single jurisdiction for the singing keys, even though it is not the best one for our needs. We'd of course love to move them to the best jurisdiction but that is actually quite a bit of work, so we have to stay grounded in reality. For me the hard part of all this is how to be transparent as possible without putting a giant target on the heads of the people who control the keys. I think it is clear that publishing private key usage information is safe, like this key signed these things at these times. Publishing all the details about how the key was generated could help those who want to attack it more than those who rely on it. But that kind of information would be good to share internally to the project at the very least. And publishing the jurisdictions could be enough info to deanonymize the key holder, e.g. if it is Germany, then there are many DDs there. If it is Iceland, then govs can easily find who to target. .hc Hi I'm exploring how to defend against an attacker who can create valid signatures for cryptographic private keys (e.g., PGP) that users need to trust when using an operating system such as Debian. A signature like that can be used in a targetted attacks against one victim. For example, apt does not have any protection against this threat scenario, and often unprotected ftp:// or http:// transports are used, which are easy to man-in-the-middle. Even with https:// there is a large number of mirror sites out there that can replace content you get. There is a risk that use of a compromised trusted apt PGP key will not be noticed. Attackers are also in a good position to deny themselves out of their actions if they are careful. Part of my effort is to inventory all trusted private keys that a distribution needs to manage on their own, or depend on that are managed by others, and gain some insight how these keys are handled. Does the Debian project, or any members, publish information on these topics? Has this been discussed before? Some questions that I think would be useful to answer include: 1) List of relevant private keys, held by the organization, its members, or someone else. As far as I can tell from Debian, the list includes the following PGP trust anchors: B8B8 0B5B 623E AB6A D877 5C45 B7C5 D7D6 3509 47F8 05AB 9034 0C0C 5E79 7F44 A8C8 254C F3B5 AEC0 A8F0 4D64 FEC1 19C2 0290 67D6 E791 F8D2 585B 8783 D481 1F89 983E 0081 FDE0 18F3 CC96 73A4 F27B 8DD4 7936 AC53 0D52 0F2F 3269 F5E9 8313 A484 4904 4AAD 5C5D A428 5295 FC7B 1A81 6000 62A9 605C 66F0 0D6C 9793 80D1 5823 B7FD 1561 F9F7 BCDD DC30 D7C2 3CBB ABEE 5E61 B217 265D A980 7A23 C5FF 4DFA B270 CAA9 6DFA 6D33 866E DD8F FA41 C014 3AED DCC9 EFBF 77E1 1517 Compromising any single key on this list leads to trouble. However I don't think this list is complete. What other keys are there? I believe there are keys used to sign some component of the boot phase, compare the 'shim-signed' and 'grub-efi-amd64-signed' packages. What private keys held by Debian are involved here? What keys held by others are involved? What other similar packages exists? 2) For each private key, information about its management and lifecycle. Relevant questions include: a) How was the key generated? By whom? On what hardware? What software? In what environment? What legal jurisdiction apply to people involved? b) How is the key stored and protected during its lifetime? What media is used? Who control the physical storage of the key? How are they stored and transported? What jurisdiction? c) Under what policy is the key used? What should it sign? Who authorize the signing? What hardware and software is used? What jurisdiction? d) For externally held keys, what are the legal terms we use the keys under? What insight into key transparency questions do we have? What of those can we make public? How do they restrict what we are allowed to do? 3) Which less visible private keys are indirectly trusted by users of the distribution? For example, all DD PGP keys are indirectly trusted since they are permitted to make uploads into the archive. Host keys used to authorized access to sensitive systems may also be relevant. I'm primarily thinking SSH private keys to a system that may have online access to a private key signing oracle. Indirectly, questions about authentication protocols and authorization
Transparency into private keys of Debian
Hi I'm exploring how to defend against an attacker who can create valid signatures for cryptographic private keys (e.g., PGP) that users need to trust when using an operating system such as Debian. A signature like that can be used in a targetted attacks against one victim. For example, apt does not have any protection against this threat scenario, and often unprotected ftp:// or http:// transports are used, which are easy to man-in-the-middle. Even with https:// there is a large number of mirror sites out there that can replace content you get. There is a risk that use of a compromised trusted apt PGP key will not be noticed. Attackers are also in a good position to deny themselves out of their actions if they are careful. Part of my effort is to inventory all trusted private keys that a distribution needs to manage on their own, or depend on that are managed by others, and gain some insight how these keys are handled. Does the Debian project, or any members, publish information on these topics? Has this been discussed before? Some questions that I think would be useful to answer include: 1) List of relevant private keys, held by the organization, its members, or someone else. As far as I can tell from Debian, the list includes the following PGP trust anchors: B8B8 0B5B 623E AB6A D877 5C45 B7C5 D7D6 3509 47F8 05AB 9034 0C0C 5E79 7F44 A8C8 254C F3B5 AEC0 A8F0 4D64 FEC1 19C2 0290 67D6 E791 F8D2 585B 8783 D481 1F89 983E 0081 FDE0 18F3 CC96 73A4 F27B 8DD4 7936 AC53 0D52 0F2F 3269 F5E9 8313 A484 4904 4AAD 5C5D A428 5295 FC7B 1A81 6000 62A9 605C 66F0 0D6C 9793 80D1 5823 B7FD 1561 F9F7 BCDD DC30 D7C2 3CBB ABEE 5E61 B217 265D A980 7A23 C5FF 4DFA B270 CAA9 6DFA 6D33 866E DD8F FA41 C014 3AED DCC9 EFBF 77E1 1517 Compromising any single key on this list leads to trouble. However I don't think this list is complete. What other keys are there? I believe there are keys used to sign some component of the boot phase, compare the 'shim-signed' and 'grub-efi-amd64-signed' packages. What private keys held by Debian are involved here? What keys held by others are involved? What other similar packages exists? 2) For each private key, information about its management and lifecycle. Relevant questions include: a) How was the key generated? By whom? On what hardware? What software? In what environment? What legal jurisdiction apply to people involved? b) How is the key stored and protected during its lifetime? What media is used? Who control the physical storage of the key? How are they stored and transported? What jurisdiction? c) Under what policy is the key used? What should it sign? Who authorize the signing? What hardware and software is used? What jurisdiction? d) For externally held keys, what are the legal terms we use the keys under? What insight into key transparency questions do we have? What of those can we make public? How do they restrict what we are allowed to do? 3) Which less visible private keys are indirectly trusted by users of the distribution? For example, all DD PGP keys are indirectly trusted since they are permitted to make uploads into the archive. Host keys used to authorized access to sensitive systems may also be relevant. I'm primarily thinking SSH private keys to a system that may have online access to a private key signing oracle. Indirectly, questions about authentication protocols and authorization methods are relevant. To the extent that symmetric shared secrets (rather than asymmetric public/private keys) are involved, the same question applies. Other aspects worth exploring? I understand commercial distributions have different incentives related to making this information public. They have a commercial interest to defend, and has usually entered various legal arrangements that create obstacles related to releasing information. As we've seen with the WebPKI CA business, that model does not inspire trust and leads to systematic failures. More transparent approaches like Certificate Transparency and LetsEncrypt evolved as a consequence. I believe that Debian is in a good position to improve things and, if there is interest, could lead the way by documenting a better approach, and describe how to deal with these concerns in a more transparent way than what we do today. Thoughts? /Simon signature.asc Description: PGP signature
