Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Tue, Mar 31, 2020 at 01:34:27PM +1100, Matt Palmer wrote: > If someone would like to make the argument that it's a gray area because I > submitted the revocation requests via ACME, rather than the CPS-provided > e-mail address, well, I can switch to sending e-mails, but having a human > process all the revocation requests is unlikely to be a better use of > everyone's time. ... aaand they did (https://bugzilla.mozilla.org/show_bug.cgi?id=1625322#c2). Oh well, I guess someone at Let's Encrypt is going to have a bit more to do from now on. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Mon, Mar 30, 2020 at 06:01:58PM -0400, Ryan Sleevi wrote: > On Mon, Mar 30, 2020 at 5:43 PM Matt Palmer via dev-security-policy > wrote: > > > > On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy > > wrote: > > > Matt - It would be helpful if you could report issues like this to the CA > > > in question, not just to mdsp. > > > > Helpful to *whom*, exactly? I don't write up these reports to be helpful to > > the CA in question; I write them to be helpful to the community. I don't > > see how reporting these problems to an individual CA is helpful to anyone > > except that one CA -- which, as I said, is not a goal I am aiming for here. > > I don't think that's quite a particularly helpful stance to take :) > Or, put differently, "why not both" Yes, I might have put it a bit harshly, and I apologise for that. I was somewhat taken aback by the implication of hiding problems by reporting to them to the CA, rather than to the community. > That said, your specific incident was in the gray area, where you'd > already previously reported compromise and the CA issued certs with > known compromised keys. You shouldn't "have" to report those new keys, > but it's still good form. I *have* been reporting additional certificates using the same compromised key, using the ACME revocation endpoint provided by the CA, and indicating that the reason for requesting certificate revocation was key compromise. I don't see how it's a "gray area", though, except insofar as multiple CAs have misinterpreted the BRs in roughly the same way. If someone would like to make the argument that it's a gray area because I submitted the revocation requests via ACME, rather than the CPS-provided e-mail address, well, I can switch to sending e-mails, but having a human process all the revocation requests is unlikely to be a better use of everyone's time. > > At any rate, since (as I understand it) all CAs are supposed to be watching > > mdsp anyway, sending a report here should be equivalent to sending it to all > > CAs -- including Let's Encrypt -- anyway. > > Ish? https://wiki.mozilla.org/CA/Incident_Dashboard specifically > encourages reporters to file a new incident bug. I don't see that that page *discourages* reporters from posting to mdsp. My point, though, is that Josh asked for a report to be sent to Let's Encrypt, all CAs are supposed to watch mdsp, therefore sending to mdsp should satisfy Josh's request for a copy to be sent to Let's Encrypt. > https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report > allows CAs to post to m.d.s.p. and a member will convert to a bug, but > I don't think it should be, nor do I want, m.d.s.p. to be the general > catch-all reporting mechanism for general users :) For one, as you can > see by my timeliness to the threads, it makes it hard to respond and > triage appropriately. I did look into creating CA Compliance bugs directly, however I'm not 100% confident of what counts as a compliance issue (as you've seen with some of my past posts). Also, bugzilla uses a mail relay which is blocked on my mail server (AWS' Spam Emission Service), and there's nothing in the SMTP transaction I can use to whitelist *just* Bugzilla's emails. So I can't create an account, and so I can't create bugs (or make snarky comments about why CAs haven't updated their open bugs in three weeks -- which you may count as a positive, perhaps?) - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Mon, Mar 30, 2020 at 5:43 PM Matt Palmer via dev-security-policy wrote: > > On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy > wrote: > > Matt - It would be helpful if you could report issues like this to the CA > > in question, not just to mdsp. > > Helpful to *whom*, exactly? I don't write up these reports to be helpful to > the CA in question; I write them to be helpful to the community. I don't > see how reporting these problems to an individual CA is helpful to anyone > except that one CA -- which, as I said, is not a goal I am aiming for here. I don't think that's quite a particularly helpful stance to take :) Or, put differently, "why not both" That said, your specific incident was in the gray area, where you'd already previously reported compromise and the CA issued certs with known compromised keys. You shouldn't "have" to report those new keys, but it's still good form. > At any rate, since (as I understand it) all CAs are supposed to be watching > mdsp anyway, sending a report here should be equivalent to sending it to all > CAs -- including Let's Encrypt -- anyway. Ish? https://wiki.mozilla.org/CA/Incident_Dashboard specifically encourages reporters to file a new incident bug. https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report allows CAs to post to m.d.s.p. and a member will convert to a bug, but I don't think it should be, nor do I want, m.d.s.p. to be the general catch-all reporting mechanism for general users :) For one, as you can see by my timeliness to the threads, it makes it hard to respond and triage appropriately. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Mon, Mar 30, 2020 at 01:48:28PM -0700, Josh Aas via dev-security-policy wrote: > Matt - It would be helpful if you could report issues like this to the CA > in question, not just to mdsp. Helpful to *whom*, exactly? I don't write up these reports to be helpful to the CA in question; I write them to be helpful to the community. I don't see how reporting these problems to an individual CA is helpful to anyone except that one CA -- which, as I said, is not a goal I am aiming for here. At any rate, since (as I understand it) all CAs are supposed to be watching mdsp anyway, sending a report here should be equivalent to sending it to all CAs -- including Let's Encrypt -- anyway. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Monday, March 30, 2020 at 4:48:38 PM UTC-4, Josh Aas wrote: > On Thursday, March 26, 2020 at 6:27:10 PM UTC-4, Ryan Sleevi wrote: > > Apologies for the delay here. I filed > > https://bugzilla.mozilla.org/show_bug.cgi?id=1625322 for this. > > We are looking into this. > > Matt - It would be helpful if you could report issues like this to the CA in > question, not just to mdsp. We can review and respond faster. So far as I > know we were never contacted. You can use our secur...@letsencrypt.org email > address in the future. Thanks. It seems that Matt did report to our community forums. I'd like to encourage people to report known or potential security issues to our security@ address. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Thursday, March 26, 2020 at 6:27:10 PM UTC-4, Ryan Sleevi wrote: > Apologies for the delay here. I filed > https://bugzilla.mozilla.org/show_bug.cgi?id=1625322 for this. We are looking into this. Matt - It would be helpful if you could report issues like this to the CA in question, not just to mdsp. We can review and respond faster. So far as I know we were never contacted. You can use our secur...@letsencrypt.org email address in the future. Thanks. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
Apologies for the delay here. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1625322 for this. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Paessler (was Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours)
On Sun, Mar 22, 2020 at 07:47:49AM +0100, Hanno Böck via dev-security-policy wrote: > FWIW: Given that with the private key it's easily possible to revoke > certificates from Let's Encrypt I took the key yesterday and iterated > over all of them and called the revoke command of certbot. Yes, I play revocation whack-a-mole every day or two. I hammer crt.sh's pgsql replicas each time to get an up-to-date list of all new certs with keys in the pwnedkeys database, and do the needful. > I strongly recommend Let's Encrypt (and probably all other CAs) > blacklists that key if they haven't already done so. That'll always be the dream... but since at least one CA can't seem to prevent a customer from getting a new certificate for the same key *while they're revoking a cert for the same name with the same key because it's compromised*, I think it's going to take a BR change that forbids reusing a reported-compromised key before CAs bake in any sort of sensible key blacklisting. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Paessler (was Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours)
On Sat, 21 Mar 2020 19:20:27 + Nick Lamb via dev-security-policy wrote: > Rather than mint an RSA key pair and self-signed certificate to > bootstrap each install, they just supply a (presumably randomly > generated) key and certificate right in the install data. FWIW: Given that with the private key it's easily possible to revoke certificates from Let's Encrypt I took the key yesterday and iterated over all of them and called the revoke command of certbot. They were all already revoked except for the latest [1], which was issued on the 20th of march. Now there's this [2] certificate with the same key that apparently got revoked on the 19th. I strongly recommend Let's Encrypt (and probably all other CAs) blacklists that key if they haven't already done so. [1] https://crt.sh/?id=2603336468 [2] https://crt.sh/?id=2574981982 -- Hanno Böck https://hboeck.de/ ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Paessler (was Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours)
On Sat, Mar 21, 2020 at 07:20:27PM +, Nick Lamb wrote: > On Sat, 21 Mar 2020 13:40:21 +1100 > Matt Palmer via dev-security-policy > wrote: > > There's also this one, which is another reuse-after-revocation, but > > the prior history of this key suggests that there's something *far* > > more interesting going on, given the variety of CAs and domain names > > it has been used for (and its current residence, on a Taiwanese > > traffic stats server): > > > > > > https://crt.sh/?spkisha256=69fc5edbd904577629121b09c49b711e201c46213e5b175bbee08a4d1d30b3c7 > > > > If anyone figures out the story with that last key, I'd be most > > pleased to hear about it. > > Sure. [snip story] Ha ha! Nice detective work. It was the old wildcard for `*.new-access.net` that threw me for a loop, but I suppose if someone's going to reuse a key, why not reuse one for a wildcard? Thanks, I can now sleep a little bit sounder now that I know there isn't another Debian-style weak PRNG out there. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Paessler (was Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours)
On Sat, 21 Mar 2020 13:40:21 +1100 Matt Palmer via dev-security-policy wrote: > Oh the facepalm, it burns (probably too much hand sanitizer)... let > me try that again. Use soap and water where practical. And, as the BBC Comedy TV show "That Mitchell & Webb Look" put it many years ago "Remain indoors". > There's also this one, which is another reuse-after-revocation, but > the prior history of this key suggests that there's something *far* > more interesting going on, given the variety of CAs and domain names > it has been used for (and its current residence, on a Taiwanese > traffic stats server): > > > https://crt.sh/?spkisha256=69fc5edbd904577629121b09c49b711e201c46213e5b175bbee08a4d1d30b3c7 > > If anyone figures out the story with that last key, I'd be most > pleased to hear about it. Sure. This requires a small degree of insight into how little ordinary people (even say IT people) understand about public key cryptography. These servers are running PRTG - a network monitoring tool from an outfit named Paessler. The software offers a web interface with SSL. PRTG is supplied as Windows software, and I have just installed it on my games PC (hopefully uninstalling it will be easy because this is no time to go out shopping for a PC) to verify the following: Rather than mint an RSA key pair and self-signed certificate to bootstrap each install, they just supply a (presumably randomly generated) key and certificate right in the install data. They don't have one of those (often rather archaic but functional) UIs where it mints new RSA keys and gives you a CSR for them. Instead it offers either a tool that will convert keys and certificates and install them, or you can just paste the files into the right place and restart the software. Now, for you or me the provided default RSA key is obviously no use and you'd mint your own with your preferred tools before requesting a publicly trusted certificate or indeed using your own in-house CA. But if you don't know much about this stuff and you find there's a perfectly nice RSA key supplied with the software it seems natural to use it. Whereupon of course now your "real" publicly trusted certificate is for a key which in reality is available to anybody with the insight to guess which software you're using. Oops. Here's their demo certificate, the associated Private Key is freely available to download as part of their software, but there's no need for me to paste it here. -BEGIN CERTIFICATE- MIIDpjCCAo6gAwIBAgIJAMM2JGwQ4/iqMA0GCSqGSIb3DQEBBQUAMEAxHjAcBgNV BAoTFVBSVEcgRGVtbyBDZXJ0aWZpY2F0ZTEeMBwGA1UEAxMVUFJURyBEZW1vIENl cnRpZmljYXRlMB4XDTEzMDcwODExMTUwNVoXDTIzMDcwNjExMTUwNVowQDEeMBwG A1UEChMVUFJURyBEZW1vIENlcnRpZmljYXRlMR4wHAYDVQQDExVQUlRHIERlbW8g Q2VydGlmaWNhdGUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsR6TJ IF2cRzoUfElst4CxY3q6vnWzZ0U0wrO6pdrVbrWqVcmofC9bxiLpW8AtlsQ0cVAQ r64juKbivQV6ggpIrFpYE505VDbu6tvqYR8nY2wtNJZNwKhT0hpBNmgujceaDc/q ghTIzaZGbtzas7HX1g8bBs81mw0TUI+IJNDAz+tbQM0NPxl/BY0LSuRX7ApUp/jn veUWXzpBb8BbCriQXPeykQuVXF2oWZ4d5B6X8mxl4GhzjmoQsTr0xGi0pWz1Tc0h Wkcd0hU633Hw1tjL82j8x5uEwy/nrb3ShMOzKtVpsoFA0TBc5BaIgbQvJpBk0Qd6 cfCxnLPjZQj4+AcFAgMBAAGjgaIwgZ8wHQYDVR0OBBYEFO6ncMKuxL4p7cwozSn1 USIYzEK5MHAGA1UdIwRpMGeAFO6ncMKuxL4p7cwozSn1USIYzEK5oUSkQjBAMR4w HAYDVQQKExVQUlRHIERlbW8gQ2VydGlmaWNhdGUxHjAcBgNVBAMTFVBSVEcgRGVt byBDZXJ0aWZpY2F0ZYIJAMM2JGwQ4/iqMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcN AQEFBQADggEBAKm6SueZqG7mVSyls2D/kFPoxsh1inctOeQPHbwMAVMCD68KGlJh kSicHq7bISy0aSioGRZe6rS12bcYRkqtgg0DjQ+ZmtHPBJTgrXIqZW0jHuqN6vyS d4IDCNQGrQQgQ+uC6V71EDcM6WDULuDygqdvM2D1gc8u2di8Rp3MpKfHAi8n0yRu 00B+01aqce/EA0b0dBPeJciKfB1cAU3CEGoLNVS/F8skumn7Q/kWwbuyjz0Nb66m 3WJOu1yAXPalEdRHQIiXEbnJgT5YrNU1R74CSdOATSKjk6kkWromGH63onF8wSS0 hh/btapuzGY6VPSscqMh3k9ji0+sPdxy3+U= -END CERTIFICATE- Nick. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Sat, Mar 21, 2020 at 01:53:31AM +, Nick Lamb wrote: > On Sat, 21 Mar 2020 09:25:26 +1100 > Matt Palmer via dev-security-policy > wrote: > > > These two certificates: > > > > https://crt.sh/?id=2602048478=ocsp > > https://crt.sh/?id=2601324532=ocsp > > > > Were issued by Let's Encrypt more than 24 hours ago, and remain > > unrevoked, despite the revocation of the below two certificates, > > which use the same private key, for keyCompromise prior to the above > > two certificates being issued: > > > > https://crt.sh/?id=2602048478=ocsp > > https://crt.sh/?id=2599226028=ocsp > > > > As per recent discussions here on m.d.s.p, I believe this is a breach > > of BR s4.9.1.1. > > I haven't looked at the substance of your concern yet, but the 1st and > 3rd links you gave above both look identical to me whereas your text > implies they should differ. Perhaps this is a copy-paste error? Oh the facepalm, it burns (probably too much hand sanitizer)... let me try that again. Recently issued and as-yet-unrevoked certificate, the first: https://crt.sh/?id=2602048478=ocsp Previously revoked certificate for the same key: https://crt.sh/?id=2599363087=ocsp Recently issued and as-yet-unrevoked certificate, the second: https://crt.sh/?id=2601324532=ocsp Previously revoked certificate for the same key: https://crt.sh/?id=2599226028=ocsp I've also, since my initial report, come across some more keys that have been successfully re-used by Let's Encrypt customers after being revoked for key compromise. You can pull the details out of the recent history: https://crt.sh/?spkisha256=c5b2c5acc5a35409cb18c7f820b93a3d53e2fd17d99df165875881d60ff91ca2 https://crt.sh/?spkisha256=35e61785dc449d235568dc5919f9f4bca31a234f0768e6c057f1d9e39491d76d https://crt.sh/?spkisha256=bb84a7d81dafd4e59877bb31595545eb5a205a4cc7db881b027fa499c5086c1c There's also this one, which is another reuse-after-revocation, but the prior history of this key suggests that there's something *far* more interesting going on, given the variety of CAs and domain names it has been used for (and its current residence, on a Taiwanese traffic stats server): https://crt.sh/?spkisha256=69fc5edbd904577629121b09c49b711e201c46213e5b175bbee08a4d1d30b3c7 If anyone figures out the story with that last key, I'd be most pleased to hear about it. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Let's Encrypt: Failure to revoke key-compromised certificates within 24 hours
On Sat, 21 Mar 2020 09:25:26 +1100 Matt Palmer via dev-security-policy wrote: > These two certificates: > > https://crt.sh/?id=2602048478=ocsp > https://crt.sh/?id=2601324532=ocsp > > Were issued by Let's Encrypt more than 24 hours ago, and remain > unrevoked, despite the revocation of the below two certificates, > which use the same private key, for keyCompromise prior to the above > two certificates being issued: > > https://crt.sh/?id=2602048478=ocsp > https://crt.sh/?id=2599226028=ocsp > > As per recent discussions here on m.d.s.p, I believe this is a breach > of BR s4.9.1.1. > Hi Matt, I haven't looked at the substance of your concern yet, but the 1st and 3rd links you gave above both look identical to me whereas your text implies they should differ. Perhaps this is a copy-paste error? Nick. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
