Re: Online exposed keys database
On 27/12/2018 10:35, Matt Palmer via dev-security-policy wrote:
> Hmm, Rob's reply never made it to my inbox. I'll reply to that separately
> now I know it's a thing.
Hi Matt. I'm consistently receiving "Undelivered Mail Returned to
Sender" messages from your mailserver, which is presumably why you
didn't see my reply. The body of each message is as follows:
This is the mail system at host mail.hezmatt.org.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
: Command time limit exceeded: "exec
/usr/bin/procmail -t
-a "${EXTENSION}""
> On Thu, Dec 27, 2018 at 05:56:08PM +0900, Hector Martin 'marcan' via
> dev-security-policy wrote:
>> On 19/12/2018 20:09, Rob Stradling via dev-security-policy wrote:
>>> I'm wondering how I might add a pwnedkeys check to crt.sh. I think I'd
>>> prefer to have a table of SHA-256(SPKI) stored locally on the crt.sh DB.
>>
>> Yes, I think the right approach for an upstream source is to provide a big
>> list of hashes. People can then postprocess that into whatever database or
>> filter format they want.
>
> The reason I haven't provided that (yet) is because, unlike pwnedpasswords,
> the set of pwned keys increases in real-time, as my scrapers go out into the
> world and find more keys. Thus, a once-off dump of what's in the database
> today isn't going to be very useful tomorrow, or next week, or next month.
>
> I don't want to put up a dump of everything until I've got a solid mechanism
> for people to retrieve and load updates of the dataset. The last thing I
> want to do is give people any encouragement to use a stale data set.
>
> Implementation of an auto-update mechanism is on the todo list, but it's
> quite a bit lower down the priority list than other things, like improving
> key scraping, and implementing a bloom filter of keys, which I feel is more
> useful, because you've got to hit the API anyway to get the attestation of
> compromise, so something with a bit of a false-positive rate isn't a big
> deal.
>
> - Matt
--
Rob Stradling
Senior Research & Development Scientist
Sectigo Limited
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On Wed, 19 Dec 2018 05:09:11 -0600, Rob Stradling wrote: > How do you handle malformed SPKIs? (e.g., the algorithm parameters > field for an RSA public key is missing, whereas it should be present and > should contain an ASN.1 NULL). > > Presumably your server/database only deals with correctly encoded SPKIs, > and it's up to the caller to ensure that they repair an incorrectly > encoded SPKI before they call your API? Yes, that's right. I'm not doing exhaustive checks of SPKIs at present, but the intention is to provide hashes for the canonically correct form(s) of keys. NIST EC keys annoying have *two* canonical forms (compressed and uncompressed point), and I've taken the approach of listing the key under both hashes, for completeness. However, where there's only *supposed* to be one form of key, that's all I'm providing a hash for in the API. Frontends can do the heavy lifting of detecting incorrect key encodings, and either refuse to proceed, or try and do a patch job to get a key to query. > I'm wondering how I might add a pwnedkeys check to crt.sh. I think I'd > prefer to have a table of SHA-256(SPKI) stored locally on the crt.sh DB. As I mentioned in my reply to Hector, a raw dump of the existing keys (or, rather, fingerprints thereof) is a low(ish) priority. There's lots of more important things to be working on at the moment, and this is only a spare-time project. Scrapers first! > This was the gist of the idea: > https://www.mail-archive.com/trans@ietf.org/msg02705.html Unfortunately that doesn't really give much in the "why" and "how" dimensions, either. I'd love it if you could expand on what benefits a verifiable data structure provides, in your opinion, over and above a straightforward database. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
Hmm, Rob's reply never made it to my inbox. I'll reply to that separately now I know it's a thing. On Thu, Dec 27, 2018 at 05:56:08PM +0900, Hector Martin 'marcan' via dev-security-policy wrote: > On 19/12/2018 20:09, Rob Stradling via dev-security-policy wrote: > > I'm wondering how I might add a pwnedkeys check to crt.sh. I think I'd > > prefer to have a table of SHA-256(SPKI) stored locally on the crt.sh DB. > > Yes, I think the right approach for an upstream source is to provide a big > list of hashes. People can then postprocess that into whatever database or > filter format they want. The reason I haven't provided that (yet) is because, unlike pwnedpasswords, the set of pwned keys increases in real-time, as my scrapers go out into the world and find more keys. Thus, a once-off dump of what's in the database today isn't going to be very useful tomorrow, or next week, or next month. I don't want to put up a dump of everything until I've got a solid mechanism for people to retrieve and load updates of the dataset. The last thing I want to do is give people any encouragement to use a stale data set. Implementation of an auto-update mechanism is on the todo list, but it's quite a bit lower down the priority list than other things, like improving key scraping, and implementing a bloom filter of keys, which I feel is more useful, because you've got to hit the API anyway to get the attestation of compromise, so something with a bit of a false-positive rate isn't a big deal. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On 19/12/2018 20:09, Rob Stradling via dev-security-policy wrote: I'm wondering how I might add a pwnedkeys check to crt.sh. I think I'd prefer to have a table of SHA-256(SPKI) stored locally on the crt.sh DB. Yes, I think the right approach for an upstream source is to provide a big list of hashes. People can then postprocess that into whatever database or filter format they want. For example, this is how Pwned Passwords does things, and I wrote a bloom filter implementation to import that for production usage (with parameters tuned for my personal taste of false positive rate, etc). -- Hector Martin "marcan" Public key: https://mrcn.st/pub ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On Wed, Dec 19, 2018 at 10:08:51AM +0100, Kurt Roeckx via dev-security-policy wrote: > On 2018-12-18 11:44, Matt Palmer wrote: > > It's currently loaded with great piles of Debian weak keys (from multiple > > architectures, etc), as well as some keys I've picked up at various times. > > I'm also developing scrapers for various sites where keys routinely get > > dropped. > > You might for instance also want to look at > https://github.com/devttys0/littleblackbox, I'm not sure how useful it is. You might also want to contact the people from https://factorable.net/ Kurt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
I threw together a quick Go library for using this API to see how it works in a larger app. https://github.com/adamdecaf/pwnedkeys On Wed, Dec 19, 2018 at 3:34 AM Matt Palmer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Wed, Dec 19, 2018 at 11:30:47AM +0100, Kurt Roeckx via > dev-security-policy wrote: > > I'm not sure how you feel about listing keys where you don't have the > > private key for, but are known to be compromised anyway. One potential > > source for such information might be CRLs where the reason for revocation > > was keyCompromise. > > At *this* stage, I'm really only interested in providing proof of key > exposure, via signatures. Just listing keys and saying "trust me, these > are > compromised" just seems... weak, somehow. Also, trawling revocation lists > for keys requires matching up the issuer+serial number to a cert in another > store (since CRLs only record serial numbers), which is just *annoying*. > > > If you don't want to publish the private keys, distributing the public > keys > > might be an option. > > For a "bulk" export, yes, that is a possibility. > > - Matt > > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On Wed, Dec 19, 2018 at 11:30:47AM +0100, Kurt Roeckx via dev-security-policy wrote: > I'm not sure how you feel about listing keys where you don't have the > private key for, but are known to be compromised anyway. One potential > source for such information might be CRLs where the reason for revocation > was keyCompromise. At *this* stage, I'm really only interested in providing proof of key exposure, via signatures. Just listing keys and saying "trust me, these are compromised" just seems... weak, somehow. Also, trawling revocation lists for keys requires matching up the issuer+serial number to a cert in another store (since CRLs only record serial numbers), which is just *annoying*. > If you don't want to publish the private keys, distributing the public keys > might be an option. For a "bulk" export, yes, that is a possibility. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
Hi Matt. This is great. A few comments inline... On 19/12/2018 09:00, Matt Palmer via dev-security-policy wrote: > Hi Ryan, > > On Tue, Dec 18, 2018 at 08:24:48PM -0800, Ryan Hurst via dev-security-policy > wrote: >> My first thought is by using SPKI you have limited the service >> unnecessarily to X.509 related keys, I imagined something like this >> covering PGP, JWT as well as other formats. It would be nice to see the >> scope increased accordingly. > > I had to use *something* to hash to get a fingerprint, and SPKI just > happened to be the easiest to get out of Ruby's OpenSSL library. It's also > been well-defined for a very long time and so, I hope, is fairly widely > supported. > > Converting keys which happen to be in other formats into SPKI is a Simple > Matter of Programming (or at least as simple as anything involving ASN.1 > ever is). They're all the same key parameters underneath, after all (I > noticed everyone seems to use the EC point encoding format from SEC 1, > even). > > I've already written code to do SSH public keys > (https://github.com/pwnedkeys/pwnedkeys-tools/blob/master/lib/openssl/pkey.rb), > and it wasn't particularly onerous. Extracting public keys out of PGP or > JWK wouldn't, I presume, be a Herculean task, either. > > At any rate, if I'd chosen some other format to represent the data to be > hashed, then I'd have needed to write code to convert from SPKI to *that* > format to search for keys in certificates, so it's all much of a muchness, > modulo relative distaste for particular formats. How do you handle malformed SPKIs? (e.g., the algorithm parameters field for an RSA public key is missing, whereas it should be present and should contain an ASN.1 NULL). Presumably your server/database only deals with correctly encoded SPKIs, and it's up to the caller to ensure that they repair an incorrectly encoded SPKI before they call your API? >> It would be ideal if it were possible to download the database also, the >> latency of the use of a third-party service while issuing certs is >> potentially too much for a CA to eat at issuance time; something that >> could optionally be used on-prem wouldn't leak affiliation and address >> this. > > I don't think the security community would be overly happy if I just dumped > all the keys in a tarball for world+dog to download, but I *did* consider > the possibility of providing something like a bloom filter of fingerprints, > for people who really need to do high-volume, low-latency lookups. I'd like > to know that someone actually *wants* to do high-volume, low-latency lookups > of exposed keys before I go building it (and the infrastructure for updating > the filter, distributing the updates, etc), though. I'm wondering how I might add a pwnedkeys check to crt.sh. I think I'd prefer to have a table of SHA-256(SPKI) stored locally on the crt.sh DB. >> As long as its limited to X.509, or at least as long as it supports it and >> uses SPKI, it would be interesting to have the website use PKIjs to let >> you browse to a cert, csr or key and the SPKI calculated for you. Happy >> to help with that if your interested. > > My JavaScript skills are even worse than my Golang skills, so yeah, it would > probably be better for everyone if someone else was willing to build that. > > The command-line querying tool I wrote > (https://github.com/pwnedkeys/pwnedkeys-tools/blob/master/bin/pwnedkeys-query) > takes all X.509-key-bearing forms and SSH public keys already, but I never > expected it to be the primary querying tool for anyone other than the > hard-core amongst us. A more user-friendly frontend would, no doubt, be > appreciated by many. > >> Personally I prefer https://api.pwnedkeys.com/v1/ to >> https://v1.pwnedkeys.com/. > > I did it that way mostly so I can run future versions of the API on separate > infrastructure. The current version is very lightweight and scalable; if a > future API version required more processing power to run at scale, I'd > prefer to be able to manage them separately. Not everyone has Google > Frontends they can put everything behind... It's a minor detail, > though, and one which should impact all of about a dozen people (whoever's > writing pwnedkeys querying libraries). If it turns out I was > being unnecessarily pessimistic, v2 of the API can always move under > `api.pwnedkeys.com`, because the URL will change anyway for every new > protocol version, regardless of how it's versioned. > >> I see your using JWS; I had been planning on building mine on top of >> Trillian (https://github.com/google/trillian) so you could have an >> auditable low trust mechanism to do this. Let me know if your interested >> in that and I would be happy to help there. > > As far as using JWS, I *really* wanted to include a signature in the > attestation of compromise, so there was iron-clad proof that the key was in > the control of someone who wanted to say it was pwned (hence why the signed
Re: Online exposed keys database
On 2018-12-19 10:55, Matt Palmer wrote: On Wed, Dec 19, 2018 at 10:08:51AM +0100, Kurt Roeckx via dev-security-policy wrote: On 2018-12-18 11:44, Matt Palmer wrote: It's currently loaded with great piles of Debian weak keys (from multiple architectures, etc), as well as some keys I've picked up at various times. I'm also developing scrapers for various sites where keys routinely get dropped. You might for instance also want to look at https://github.com/devttys0/littleblackbox, I'm not sure how useful it is. Oh my, that's an interesting trove. I was a bit worried at first that the private keys weren't included, but it looks like there's at least some in there. I'm not sure how you feel about listing keys where you don't have the private key for, but are known to be compromised anyway. One potential source for such information might be CRLs where the reason for revocation was keyCompromise. If you don't want to publish the private keys, distributing the public keys might be an option. Kurt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On Wed, Dec 19, 2018 at 10:08:51AM +0100, Kurt Roeckx via dev-security-policy wrote: > On 2018-12-18 11:44, Matt Palmer wrote: > > It's currently loaded with great piles of Debian weak keys (from multiple > > architectures, etc), as well as some keys I've picked up at various times. > > I'm also developing scrapers for various sites where keys routinely get > > dropped. > > You might for instance also want to look at > https://github.com/devttys0/littleblackbox, I'm not sure how useful it is. Oh my, that's an interesting trove. I was a bit worried at first that the private keys weren't included, but it looks like there's at least some in there, although they're pretty much all 512 bit keys, with a few 1024s and only one 2048. Not surprising, given the age of the repo, but hey, an exposed key's still a key! Thanks for the pointer, I've imported the 2048 bit key. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
Ryan Hurst via dev-security-policy writes: >My first thought is by using SPKI you have limited the service unnecessarily >to X.509 related keys, I imagined something like this covering PGP, JWT as >well as other formats. It would be nice to see the scope increased >accordingly. You can't do it for PGP, that hashes in a pile of additional stuff unrelated to the key so there's no way to uniquely identify a specific key, only "the key and this specific set of metadata". Using the SPKI for the hash is the best option, I use that internally as the unique ID for keys, including PGP ones. Peter. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
On 2018-12-18 11:44, Matt Palmer wrote: It's currently loaded with great piles of Debian weak keys (from multiple architectures, etc), as well as some keys I've picked up at various times. I'm also developing scrapers for various sites where keys routinely get dropped. You might for instance also want to look at https://github.com/devttys0/littleblackbox, I'm not sure how useful it is. Kurt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Online exposed keys database
Hi Ryan, On Tue, Dec 18, 2018 at 08:24:48PM -0800, Ryan Hurst via dev-security-policy wrote: > My first thought is by using SPKI you have limited the service > unnecessarily to X.509 related keys, I imagined something like this > covering PGP, JWT as well as other formats. It would be nice to see the > scope increased accordingly. I had to use *something* to hash to get a fingerprint, and SPKI just happened to be the easiest to get out of Ruby's OpenSSL library. It's also been well-defined for a very long time and so, I hope, is fairly widely supported. Converting keys which happen to be in other formats into SPKI is a Simple Matter of Programming (or at least as simple as anything involving ASN.1 ever is). They're all the same key parameters underneath, after all (I noticed everyone seems to use the EC point encoding format from SEC 1, even). I've already written code to do SSH public keys (https://github.com/pwnedkeys/pwnedkeys-tools/blob/master/lib/openssl/pkey.rb), and it wasn't particularly onerous. Extracting public keys out of PGP or JWK wouldn't, I presume, be a Herculean task, either. At any rate, if I'd chosen some other format to represent the data to be hashed, then I'd have needed to write code to convert from SPKI to *that* format to search for keys in certificates, so it's all much of a muchness, modulo relative distaste for particular formats. > It would be ideal if it were possible to download the database also, the > latency of the use of a third-party service while issuing certs is > potentially too much for a CA to eat at issuance time; something that > could optionally be used on-prem wouldn't leak affiliation and address > this. I don't think the security community would be overly happy if I just dumped all the keys in a tarball for world+dog to download, but I *did* consider the possibility of providing something like a bloom filter of fingerprints, for people who really need to do high-volume, low-latency lookups. I'd like to know that someone actually *wants* to do high-volume, low-latency lookups of exposed keys before I go building it (and the infrastructure for updating the filter, distributing the updates, etc), though. > As long as its limited to X.509, or at least as long as it supports it and > uses SPKI, it would be interesting to have the website use PKIjs to let > you browse to a cert, csr or key and the SPKI calculated for you. Happy > to help with that if your interested. My JavaScript skills are even worse than my Golang skills, so yeah, it would probably be better for everyone if someone else was willing to build that. The command-line querying tool I wrote (https://github.com/pwnedkeys/pwnedkeys-tools/blob/master/bin/pwnedkeys-query) takes all X.509-key-bearing forms and SSH public keys already, but I never expected it to be the primary querying tool for anyone other than the hard-core amongst us. A more user-friendly frontend would, no doubt, be appreciated by many. > Personally I prefer https://api.pwnedkeys.com/v1/ to > https://v1.pwnedkeys.com/. I did it that way mostly so I can run future versions of the API on separate infrastructure. The current version is very lightweight and scalable; if a future API version required more processing power to run at scale, I'd prefer to be able to manage them separately. Not everyone has Google Frontends they can put everything behind... It's a minor detail, though, and one which should impact all of about a dozen people (whoever's writing pwnedkeys querying libraries). If it turns out I was being unnecessarily pessimistic, v2 of the API can always move under `api.pwnedkeys.com`, because the URL will change anyway for every new protocol version, regardless of how it's versioned. > I see your using JWS; I had been planning on building mine on top of > Trillian (https://github.com/google/trillian) so you could have an > auditable low trust mechanism to do this. Let me know if your interested > in that and I would be happy to help there. As far as using JWS, I *really* wanted to include a signature in the attestation of compromise, so there was iron-clad proof that the key was in the control of someone who wanted to say it was pwned (hence why the signed data includes a statement to that effect, rather than a signature over some arbitrary data that could potentially be spoofed). As I mentioned in the FAQ, I first thought of formatting the signatures into X.509 certs or CSRs, but I didn't want to create any chance of confusion with legitimate X.509 objects. Also, I really don't *want* this to be an X.509-only thing, and this way I could annoy people who don't like JOSE as well as the people who don't like X.509. With regards to using Trillian (or verifiable data structures in general), given my background in working with CT, you can rest assured I spent a *lot* of time pondering how a transparency approach could provide value. Honestly, I wish I could have figured out some way to profitably use
Re: Online exposed keys database
On Tuesday, December 18, 2018 at 2:44:22 AM UTC-8, Matt Palmer wrote: > Hi all, > > I'd like to make everyone aware of a service I've just stood up, called > pwnedkeys.com. It's intended to serve as a clearinghouse of known-exposed > private keys, so that services that accept public keys from external > entities (such as -- relevant to mdsp's interests -- CAs) can make one call > to get a fairly authoritative answer to the question "has the private key > I'm being asked to interact with in some way been exposed?". > > It's currently loaded with great piles of Debian weak keys (from multiple > architectures, etc), as well as some keys I've picked up at various times. > I'm also developing scrapers for various sites where keys routinely get > dropped. > > The eventual intention is to be able to go from "private key is on The > Public Internet somewhere" to "shows up in pwnedkeys.com" automatically and > in double-quick time. > > I know there are a number of very clever people on this list who have found > and extracted keys from more esoteric places than Google search, and I'd be > really interested in talking to you (privately, I'd imagine) about getting > specimens of those keys to add to the database. > > I'd also welcome comments from anyone about the query API, the attestation > format, the documentation, or anything else vaguely relevant to the service. > Probably best to take that off-list, though. > > I do have plans to develop a PR against (the AWS Labs') certlint to cause it > to query the API, so there's no need for anyone to get deep into that unless > they're feeling especially frisky. Other linting tools will *probably* have > to do their own development, as my Go skills are... rudimentary at best, > shall we say. I'd be happy to give guidance or any other necessary help to > anyone looking at building those, though. > > Finally, if any CAs are interested in integrating the pwnedkeys database > into their issuance pipelines, I'd love to discuss how we can work together. > > Thanks, > - Matt This is great. I purchased keycompromise.com ages ago to build something just like this. Im very glad to see you took the time to make this. My first thought is by using SPKI you have limited the service unnecessarily to X.509 related keys, I imagined something like this covering PGP, JWT as well as other formats. It would be nice to see the scope increased accordingly. It would be ideal if it were possible to download the database also, the latency of the use of a third-party service while issuing certs is potentially too much for a CA to eat at issuance time; something that could optionally be used on-prem wouldn't leak affiliation and address this. As long as its limited to X.509, or at least as long as it supports it and uses SPKI, it would be interesting to have the website use PKIjs to let you browse to a cert, csr or key and the SPKI calculated for you. Happy to help with that if your interested. Personally I prefer https://api.pwnedkeys.com/v1/ to https://v1.pwnedkeys.com/. I see your using JWS; I had been planning on building mine on top of Trillian (https://github.com/google/trillian) so you could have an auditable low trust mechanism to do this. Let me know if your interested in that and I would be happy to help there. Anyways thanks for doing this. Ryan Hurst (personal) ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Online exposed keys database
Hi all, I'd like to make everyone aware of a service I've just stood up, called pwnedkeys.com. It's intended to serve as a clearinghouse of known-exposed private keys, so that services that accept public keys from external entities (such as -- relevant to mdsp's interests -- CAs) can make one call to get a fairly authoritative answer to the question "has the private key I'm being asked to interact with in some way been exposed?". It's currently loaded with great piles of Debian weak keys (from multiple architectures, etc), as well as some keys I've picked up at various times. I'm also developing scrapers for various sites where keys routinely get dropped. The eventual intention is to be able to go from "private key is on The Public Internet somewhere" to "shows up in pwnedkeys.com" automatically and in double-quick time. I know there are a number of very clever people on this list who have found and extracted keys from more esoteric places than Google search, and I'd be really interested in talking to you (privately, I'd imagine) about getting specimens of those keys to add to the database. I'd also welcome comments from anyone about the query API, the attestation format, the documentation, or anything else vaguely relevant to the service. Probably best to take that off-list, though. I do have plans to develop a PR against (the AWS Labs') certlint to cause it to query the API, so there's no need for anyone to get deep into that unless they're feeling especially frisky. Other linting tools will *probably* have to do their own development, as my Go skills are... rudimentary at best, shall we say. I'd be happy to give guidance or any other necessary help to anyone looking at building those, though. Finally, if any CAs are interested in integrating the pwnedkeys database into their issuance pipelines, I'd love to discuss how we can work together. Thanks, - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
