Re: ssl.com: Certificate with Debian weak key
On Mon, Mar 16, 2020 at 12:11:57PM -0700, Chris Kemmerer via
dev-security-policy wrote:
> On Wednesday, March 11, 2020 at 5:41:00 PM UTC-5, Matt Palmer wrote:
> > On Wed, Mar 11, 2020 at 10:46:05AM -0700, Chris Kemmerer via
> > dev-security-policy wrote:
> > > On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote:
> > > > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via
> > > > dev-security-policy wrote:
> > > For what it's worth, we believe that the current language in the BRs could
> > > be less ambiguous as far as the Debian weak keys are concerned. For
> > > example, it seems that the community's expectations are for CAs to detect
> > > and block weak Debian keys generated by vulnerable RNG using OpenSSL in
> > > popular architectures.
> >
> > The problem with using the argument that "the BRs are ambiguous" to try and
> > defend a breach of them is that there are always potential ambiguities in
> > all language -- in many ways, "ambiguity is in the eye of the beholder"
> > ("ambiguer"?). My understanding of the consensus from past discussions on
> > this list is that if a CA believes there is an ambiguity in the BRs, the
> > correct action is to raise that in the CA/B Forum *before* they fall foul of
> > it.
> >
> > CAs should be reading the BRs, as I understand it, in a "defensive" mode,
> > looking for requirements that could be read multiple ways, and when they are
> > found, the CA needs to ensure either that they are complying with the
> > strictest possible reading, or else bringing the ambiguity to the attention
> > of the CA/B Forum and suggesting less ambiguous wording.
> >
> > At any rate, it would be helpful to know what, precisely, SSL.com's
> > understanding of this requirement of the BRs prior to the commencement of
> > this incident. Can you share this with us? Presumably SSL.com did a
> > careful analysis of all aspects of the BRs, and came to a conclusion as to
> > precisely what was considered "compliant". With regards to this
> > requirement, what was SSL.com's position as to what was necessary to be
> > compliant with this aspect of the BRs?
>
> We have already described our understanding of the expectations expressed
> in BR 6.1.1.3 and the steps we took to comply with it.
Sorry, I must have missed the description of SSL.com's understanding. Could
you quote or reference it here, for clarity?
> Our implementation did not meet these expectations, as it was missing
> direct checks of keys matching the "openssl-blacklist" package.
> Immediately upon our coming to this understanding,
This is SSL.com's post-incident understanding; what I believe is important
to also know is SSL.com's *pre*-incident understanding of the BR
requirements.
> > > That could be added in the BRs:
> > >
> > > Change:
> > > "The CA SHALL reject a certificate request if the requested Public Key
> > > does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> > > it has a known weak Private Key (such as a Debian weak key, see
> > > http://wiki.debian.org/SSLkeys)"
> > >
> > > to something like:
> > > "The CA SHALL reject a certificate request if the requested Public Key
> > > does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> > > it has a known weak Private Key using, as a minimum set the Debian weak
> > > keys produced by OpenSSL in i386 and x64 architectures (see
> > > http://wiki.debian.org/SSLkeys)"
> >
> > It would appear that SSL.com is a member in good standing of the CA/B
> > Forum.
> > Is there any intention on the part of SSL.com to propose this change as a
> > ballot? While you're at it, if you could include a fix for the issue
> > described in https://github.com/cabforum/documents/issues/164, that would be
> > appreciated, since it is the same sentences that need modification, and for
> > much the same reasons.
>
> Yes, this is reasonable, and we treated such key as compromised, revoking
> it within 24 hours.
>
> We would support a ballot that makes this clear. We also monitor the
> discussion in https://github.com/cabforum/documents/issues/164.
As Ryan mentioned, "we would support a ballot" is not the same as "we intend
to propose a ballot". Thank you for clarifying that SSL.com intends to
propse a ballot in your reply to Ryan.
> > > Then, it would be clear that all CAs would need to block "at least" the
> > > vulnerable keys produced by OpenSSL and could add other keys produced by
> > > OpenSSH or other applications if they wanted a "more complete" list.
> >
> > Well, you're still missing the rnd/nornd/noreadrnd variations, and there's
> > no specification as to what key sizes are considered the bare minimum.
>
> We mention these variations in bug 1620772, but thank you for repeating it
> here for completeness.
>
> For the record, this fact (“there's no specification as to what key sizes
> are considered the bare minimum”) is exactly our point too.
I think you misunderstood my point here. I was
Re: ssl.com: Certificate with Debian weak key
On Monday, March 16, 2020 at 2:46:46 PM UTC-5, Ryan Sleevi wrote: > On Mon, Mar 16, 2020 at 3:12 PM Chris Kemmerer via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > > It would appear that SSL.com is a member in good standing of the CA/B > > Forum. > > > Is there any intention on the part of SSL.com to propose this change as a > > > ballot? While you're at it, if you could include a fix for the issue > > > described in https://github.com/cabforum/documents/issues/164, that > > would be > > > appreciated, since it is the same sentences that need modification, and > > for > > > much the same reasons. > > > > Yes, this is reasonable, and we treated such key as compromised, revoking > > it within 24 hours. > > > > We would support a ballot that makes this clear. We also monitor the > > discussion in https://github.com/cabforum/documents/issues/164. > > > > While you answered "Yes", you followed with a different clarification. "We > would support" seems different from "Is there any intention on SSL.com to > propose" Yes, we are happy to propose a ballot change to the BR language pertaining to this issue. > > > We have responded as to how we interpreted and implemented this > > requirement. Our blacklist did not include the entire set of Debian weak > > keys. We have been completely transparent about this issue and we have > > improved our weak key detection mechanism to include the openssl-blacklist > > package. > > > > We examined similar failures/incidents, such as: > > > > - https://bugzilla.mozilla.org/show_bug.cgi?id=1472052 > > - https://bugzilla.mozilla.org/show_bug.cgi?id=1435770 > > - > > https://community.letsencrypt.org/t/2017-09-09-late-weak-key-revocation/42519 > > > > The last one shows that at least one more CA had a similar interpretation > > of the BRs. > > > > Having read these, while I appreciate SSL.com highlighting them, I fail to > see them supporting the claim being made here. Could you please elaborate > why/how you believe this statement to be true? They seem rather remarkably > different, and certainly, the last one highlights a CA actively working to > be comprehensive, while much of SSL.com's reply seems to be "We shouldn't > have to be comprehensive" We were merely pointing out previous discussions on this topic. I can see why you may have such an impression, but "we shouldn't have to be comprehensive" is not what we are thinking or believe. On the contrary, we've increased our efforts to be as comprehensive as possible, and we will continue to expand our weak keys list even after this issue is closed. We believe in a robustly secure ecosystem, not in half measures. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Mon, Mar 16, 2020 at 3:12 PM Chris Kemmerer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > It would appear that SSL.com is a member in good standing of the CA/B > Forum. > > Is there any intention on the part of SSL.com to propose this change as a > > ballot? While you're at it, if you could include a fix for the issue > > described in https://github.com/cabforum/documents/issues/164, that > would be > > appreciated, since it is the same sentences that need modification, and > for > > much the same reasons. > > Yes, this is reasonable, and we treated such key as compromised, revoking > it within 24 hours. > > We would support a ballot that makes this clear. We also monitor the > discussion in https://github.com/cabforum/documents/issues/164. > While you answered "Yes", you followed with a different clarification. "We would support" seems different from "Is there any intention on SSL.com to propose" > We have responded as to how we interpreted and implemented this > requirement. Our blacklist did not include the entire set of Debian weak > keys. We have been completely transparent about this issue and we have > improved our weak key detection mechanism to include the openssl-blacklist > package. > > We examined similar failures/incidents, such as: > > - https://bugzilla.mozilla.org/show_bug.cgi?id=1472052 > - https://bugzilla.mozilla.org/show_bug.cgi?id=1435770 > - > https://community.letsencrypt.org/t/2017-09-09-late-weak-key-revocation/42519 > > The last one shows that at least one more CA had a similar interpretation > of the BRs. > Having read these, while I appreciate SSL.com highlighting them, I fail to see them supporting the claim being made here. Could you please elaborate why/how you believe this statement to be true? They seem rather remarkably different, and certainly, the last one highlights a CA actively working to be comprehensive, while much of SSL.com's reply seems to be "We shouldn't have to be comprehensive" ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Wednesday, March 11, 2020 at 5:41:00 PM UTC-5, Matt Palmer wrote:
> On Wed, Mar 11, 2020 at 10:46:05AM -0700, Chris Kemmerer via
> dev-security-policy wrote:
> > On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote:
> > > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via
> > > dev-security-policy wrote:
> > > > For the purpose of identifying whether a Private Key is weak, SSL.com
> > > > uses
> > > > a set of Debian weak keys that was provided by our CA software vendor as
> > > > the basis for our blacklist.
> > >
> > > I think it's worth getting additional, *very* detailed, information from
> > > your CA software vendor as to where *they* got their Debian weak key list
> > > from. That appears to be the fundamental breakdown here -- you relied on
> > > a
> > > third-party to give you good service, and they didn't. So I think that
> > > digging into your vendor's practices is an important line of enquiry to go
> > > down.
> >
> > As mentioned on our report, we used that list as a basis, and paid
> > attention to augment it with other weak keys from available blacklists,
>
> So presumably if there are other Mozilla-trusted CAs using the same CA
> vendor, who *are* doing the bare minimum and just using the CA vendor's key
> list, they're even more vulnerable to a potential misissuance. As you
> mentioned that your CA software vendor does read this list, I *really* hope
> they speak up soon so we can figure out how they got their key list.
>
> > weak keys from available blacklists, even for the ROCA vulnerability.
>
> Sidenote: my understanding of ROCA is that it is of a different form to the
> Debian weak key problem, in that you can't a priori enumerate ROCA-impacted
> keys, but can only identify them as you find them. As such, my
> understanding is that there isn't, and in fact *cannot*, be a comprehensive
> "blacklist", as such, of keys affected by ROCA.
>
> Is your understanding of the ROCA vulnerability different to my description
> above, and if not, can you explain how a "blacklist"-based approach is a
> suitable mitigation for avoiding issuance of certificates using
> ROCA-impacted private keys?
>
> (Conversely, if it *is* possible get a comprehensive list of ROCA-impacted
> keys, I know what I'm doing this weekend...)
ROCA vulnerability detection is part of our “weak keys detection mechanism”,
not part of a blacklist. Our original language “we do have a weak keys
detection mechanism in place, it does detect Debian weak keys (although it's
not perfect) and it also detects ROCA vulnerable keys” makes that clear.
>
> > For what it's worth, we believe that the current language in the BRs could
> > be less ambiguous as far as the Debian weak keys are concerned. For
> > example, it seems that the community's expectations are for CAs to detect
> > and block weak Debian keys generated by vulnerable RNG using OpenSSL in
> > popular architectures.
>
> The problem with using the argument that "the BRs are ambiguous" to try and
> defend a breach of them is that there are always potential ambiguities in
> all language -- in many ways, "ambiguity is in the eye of the beholder"
> ("ambiguer"?). My understanding of the consensus from past discussions on
> this list is that if a CA believes there is an ambiguity in the BRs, the
> correct action is to raise that in the CA/B Forum *before* they fall foul of
> it.
>
> CAs should be reading the BRs, as I understand it, in a "defensive" mode,
> looking for requirements that could be read multiple ways, and when they are
> found, the CA needs to ensure either that they are complying with the
> strictest possible reading, or else bringing the ambiguity to the attention
> of the CA/B Forum and suggesting less ambiguous wording.
>
> At any rate, it would be helpful to know what, precisely, SSL.com's
> understanding of this requirement of the BRs prior to the commencement of
> this incident. Can you share this with us? Presumably SSL.com did a
> careful analysis of all aspects of the BRs, and came to a conclusion as to
> precisely what was considered "compliant". With regards to this
> requirement, what was SSL.com's position as to what was necessary to be
> compliant with this aspect of the BRs?
We have already described our understanding of the expectations expressed in BR
6.1.1.3 and the steps we took to comply with it. We provide details in bug
1620772, which remains the primary channel for this issue. As always, we
attempt to be as transparent as possible, because we strongly feel that this is
exactly the approach which better serves the ecosystem.
Our implementation did not meet these expectations, as it was missing direct
checks of keys matching the "openssl-blacklist" package. Immediately upon our
coming to this understanding, we initiated development of a fix to meet this
expectation. This fix was tested and pushed to production last Friday. In
parallel, we are conducting an analysis of which
Re: ssl.com: Certificate with Debian weak key
On Wed, Mar 11, 2020 at 10:46:05AM -0700, Chris Kemmerer via
dev-security-policy wrote:
> On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote:
> > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via
> > dev-security-policy wrote:
> > > For the purpose of identifying whether a Private Key is weak, SSL.com uses
> > > a set of Debian weak keys that was provided by our CA software vendor as
> > > the basis for our blacklist.
> >
> > I think it's worth getting additional, *very* detailed, information from
> > your CA software vendor as to where *they* got their Debian weak key list
> > from. That appears to be the fundamental breakdown here -- you relied on a
> > third-party to give you good service, and they didn't. So I think that
> > digging into your vendor's practices is an important line of enquiry to go
> > down.
>
> As mentioned on our report, we used that list as a basis, and paid
> attention to augment it with other weak keys from available blacklists,
So presumably if there are other Mozilla-trusted CAs using the same CA
vendor, who *are* doing the bare minimum and just using the CA vendor's key
list, they're even more vulnerable to a potential misissuance. As you
mentioned that your CA software vendor does read this list, I *really* hope
they speak up soon so we can figure out how they got their key list.
> weak keys from available blacklists, even for the ROCA vulnerability.
Sidenote: my understanding of ROCA is that it is of a different form to the
Debian weak key problem, in that you can't a priori enumerate ROCA-impacted
keys, but can only identify them as you find them. As such, my
understanding is that there isn't, and in fact *cannot*, be a comprehensive
"blacklist", as such, of keys affected by ROCA.
Is your understanding of the ROCA vulnerability different to my description
above, and if not, can you explain how a "blacklist"-based approach is a
suitable mitigation for avoiding issuance of certificates using
ROCA-impacted private keys?
(Conversely, if it *is* possible get a comprehensive list of ROCA-impacted
keys, I know what I'm doing this weekend...)
> For what it's worth, we believe that the current language in the BRs could
> be less ambiguous as far as the Debian weak keys are concerned. For
> example, it seems that the community's expectations are for CAs to detect
> and block weak Debian keys generated by vulnerable RNG using OpenSSL in
> popular architectures.
The problem with using the argument that "the BRs are ambiguous" to try and
defend a breach of them is that there are always potential ambiguities in
all language -- in many ways, "ambiguity is in the eye of the beholder"
("ambiguer"?). My understanding of the consensus from past discussions on
this list is that if a CA believes there is an ambiguity in the BRs, the
correct action is to raise that in the CA/B Forum *before* they fall foul of
it.
CAs should be reading the BRs, as I understand it, in a "defensive" mode,
looking for requirements that could be read multiple ways, and when they are
found, the CA needs to ensure either that they are complying with the
strictest possible reading, or else bringing the ambiguity to the attention
of the CA/B Forum and suggesting less ambiguous wording.
At any rate, it would be helpful to know what, precisely, SSL.com's
understanding of this requirement of the BRs prior to the commencement of
this incident. Can you share this with us? Presumably SSL.com did a
careful analysis of all aspects of the BRs, and came to a conclusion as to
precisely what was considered "compliant". With regards to this
requirement, what was SSL.com's position as to what was necessary to be
compliant with this aspect of the BRs?
> That could be added in the BRs:
>
> Change:
> "The CA SHALL reject a certificate request if the requested Public Key
> does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> it has a known weak Private Key (such as a Debian weak key, see
> http://wiki.debian.org/SSLkeys)"
>
> to something like:
> "The CA SHALL reject a certificate request if the requested Public Key
> does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if
> it has a known weak Private Key using, as a minimum set the Debian weak
> keys produced by OpenSSL in i386 and x64 architectures (see
> http://wiki.debian.org/SSLkeys)"
It would appear that SSL.com is a member in good standing of the CA/B Forum.
Is there any intention on the part of SSL.com to propose this change as a
ballot? While you're at it, if you could include a fix for the issue
described in https://github.com/cabforum/documents/issues/164, that would be
appreciated, since it is the same sentences that need modification, and for
much the same reasons.
> Then, it would be clear that all CAs would need to block "at least" the
> vulnerable keys produced by OpenSSL and could add other keys produced by
> OpenSSH or other applications if they wanted a "more complete"
Re: ssl.com: Certificate with Debian weak key
We regret your impression that we take this issue with anything less than the utmost seriousness. We have opened a ticket and are actively working with our CA software vendor to address the underlying issue. Rather than stopping there, we have been working concurrently to put into place the necessary checks against openssl-blacklist independently of the CA software vendor. Whether through our CA software vendor or independently, we are committed to finding a long-term solution that is effective and efficient. We will provide regular updates of our progress to the bug (to which this message has also been posted). On Wednesday, March 11, 2020 at 1:25:19 PM UTC-5, Ryan Sleevi wrote: > On Wed, Mar 11, 2020 at 1:46 PM Chris Kemmerer via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > You are correct, each compliance violation is considered an incident. > > However in our opinion we have not violated our CP/CPS or the current > > Baseline Requirements. Although this is a complex issue with no definite > > consensus on which authoritative list to use (only suggestions), we do have > > a weak keys detection mechanism in place, it does detect Debian weak keys > > (although it's not perfect) and it also detects ROCA vulnerable keys. > > > I've commented on the bug as much, but I find this response deeply > disappointing and disconcerting. > > This CA ignored a widely known, explicitly circulated list of > known-compromised keys, and is now doubling down that there's nothing wrong > with this. The justification is "This key was not known to be compromised > /by us/", with their rationale of "The BRs explicitly tell us where we > could find a list of known weak/compromised keys, but doesn't say we have > to look at it, and so our elective ignorance is a virtue, not a vice". > > Whatever your view of the correctness [1] of this argument, as a systemic > response from a CA, the entrenchedness here suggests that unless the CA can > be hand-held into being trustworthy, they will do the minimum possible > thing. > > I appreciate the suggestions for improvement, and that's at least slightly > positive, but if the answer is "You have to tell us to read that page or we > won't, even if you tell us /about/ that page", then... meh, that's not a CA > that inspires confidence. > > [1] https://www.youtube.com/watch?v=hou0lU8WMgo ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Wed, Mar 11, 2020 at 1:46 PM Chris Kemmerer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > You are correct, each compliance violation is considered an incident. > However in our opinion we have not violated our CP/CPS or the current > Baseline Requirements. Although this is a complex issue with no definite > consensus on which authoritative list to use (only suggestions), we do have > a weak keys detection mechanism in place, it does detect Debian weak keys > (although it's not perfect) and it also detects ROCA vulnerable keys. I've commented on the bug as much, but I find this response deeply disappointing and disconcerting. This CA ignored a widely known, explicitly circulated list of known-compromised keys, and is now doubling down that there's nothing wrong with this. The justification is "This key was not known to be compromised /by us/", with their rationale of "The BRs explicitly tell us where we could find a list of known weak/compromised keys, but doesn't say we have to look at it, and so our elective ignorance is a virtue, not a vice". Whatever your view of the correctness [1] of this argument, as a systemic response from a CA, the entrenchedness here suggests that unless the CA can be hand-held into being trustworthy, they will do the minimum possible thing. I appreciate the suggestions for improvement, and that's at least slightly positive, but if the answer is "You have to tell us to read that page or we won't, even if you tell us /about/ that page", then... meh, that's not a CA that inspires confidence. [1] https://www.youtube.com/watch?v=hou0lU8WMgo ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Tuesday, March 10, 2020 at 8:44:49 PM UTC-5, Matt Palmer wrote: > On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via > dev-security-policy wrote: > > We have updated https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 with > > the findings of our current investigation. > > Thanks for this update. I have... comments. > > Before I get into the nitty-gritty, though, I'd just like to say that I > found your response to be unnecessarily defensive. So much of it reads -- > to me, at least -- as "please don't blame us, it wasn't our fault!". Fault > isn't really the issue at hand. Discovering what went wrong, and making > sure that the issues are fixed, both for SSL.com and, if appropriate, other > CAs, is what I, at least, am trying to achieve. > > Therefore, while a lot of my responses below address specific points that > you've made in your defence, please understand that I'm not trying to rebut > them in an attempt to say "nee nah nee nah, it *is* your fault!", but rather > to try and provide further information that SSL.com and other CAs could > benefit from considering for improvement in the future, given my experience > dealing with Debian weak keys. We share the same view. Our analysis so far confirms that this is a tricky issue, and it gets trickier if one considers compatibility issues between CA software and lists of weak keys which are publicly available. This is exactly why it would be a useful contribution to the ecosystem, if anyone in possession of such weak key databases publishes them along with the keys. We have included more details for this below. > > For the purpose of identifying whether a Private Key is weak, SSL.com uses > > a set of Debian weak keys that was provided by our CA software vendor as > > the basis for our blacklist. > > I think it's worth getting additional, *very* detailed, information from > your CA software vendor as to where *they* got their Debian weak key list > from. That appears to be the fundamental breakdown here -- you relied on a > third-party to give you good service, and they didn't. So I think that > digging into your vendor's practices is an important line of enquiry to go > down. As mentioned on our report, we used that list as a basis, and paid attention to augment it with other weak keys from available blacklists, even for the ROCA vulnerability. So, instead of just relying on the CA software vendor, we definitely did - and do, on a regular basis - our own homework. > Also, given that there is a non-zero chance that other CAs trusted by > Mozilla may be using the same software, with the same incomplete list of > weak keys, I think it's important that the fact that the vendor is using a > demonstrably incomplete list be circulated to the other customers of this > vendor. How would you suggest this is best accomplished? Our CA software vendor is monitoring this list so they should be able to address this issue independently. For what it's worth, we believe that the current language in the BRs could be less ambiguous as far as the Debian weak keys are concerned. For example, it seems that the community's expectations are for CAs to detect and block weak Debian keys generated by vulnerable RNG using OpenSSL in popular architectures. That could be added in the BRs: Change: "The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)" to something like: "The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key using, as a minimum set the Debian weak keys produced by OpenSSL in i386 and x64 architectures (see http://wiki.debian.org/SSLkeys)" Then, it would be clear that all CAs would need to block "at least" the vulnerable keys produced by OpenSSL and could add other keys produced by OpenSSH or other applications if they wanted a "more complete" list. > > This information was disclosed on 2020-03-07 to the person that submitted > > the Certificate Problem Report. > > I don't see anything from SSL.com that looks relevant in my inbox, but it's > not an important point, just thought I'd mention it in passing. For the record, we replied to mpal...@hezmatt.org at 2020-03-07 8:41 PM (UTC) Email subject: "Problem Report for certificate(s) with compromised private key" Can you please confirm? We did disclose the source of our weak keys in that email. > > The above practices comply with our CP/CPS, version 1.8, section 6.1.1.2, > > which states: > > > > "SSL.com shall reject a certificate request if the request has a known > > weak Private Key". > > Hmm, "known" is doing a lot of work in that sentence. Known to *whom* is > the important, but unanswered, question. It may be a question which should > be answered
Re: ssl.com: Certificate with Debian weak key
On Tue, Mar 10, 2020 at 01:48:49PM -0700, Chris Kemmerer via dev-security-policy wrote: > We have updated https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 with > the findings of our current investigation. Thanks for this update. I have... comments. Before I get into the nitty-gritty, though, I'd just like to say that I found your response to be unnecessarily defensive. So much of it reads -- to me, at least -- as "please don't blame us, it wasn't our fault!". Fault isn't really the issue at hand. Discovering what went wrong, and making sure that the issues are fixed, both for SSL.com and, if appropriate, other CAs, is what I, at least, am trying to achieve. Therefore, while a lot of my responses below address specific points that you've made in your defence, please understand that I'm not trying to rebut them in an attempt to say "nee nah nee nah, it *is* your fault!", but rather to try and provide further information that SSL.com and other CAs could benefit from considering for improvement in the future, given my experience dealing with Debian weak keys. > For the purpose of identifying whether a Private Key is weak, SSL.com uses > a set of Debian weak keys that was provided by our CA software vendor as > the basis for our blacklist. I think it's worth getting additional, *very* detailed, information from your CA software vendor as to where *they* got their Debian weak key list from. That appears to be the fundamental breakdown here -- you relied on a third-party to give you good service, and they didn't. So I think that digging into your vendor's practices is an important line of enquiry to go down. Also, given that there is a non-zero chance that other CAs trusted by Mozilla may be using the same software, with the same incomplete list of weak keys, I think it's important that the fact that the vendor is using a demonstrably incomplete list be circulated to the other customers of this vendor. How would you suggest this is best accomplished? > This information was disclosed on 2020-03-07 to the person that submitted > the Certificate Problem Report. I don't see anything from SSL.com that looks relevant in my inbox, but it's not an important point, just thought I'd mention it in passing. > The above practices comply with our CP/CPS, version 1.8, section 6.1.1.2, > which states: > > "SSL.com shall reject a certificate request if the request has a known > weak Private Key". Hmm, "known" is doing a lot of work in that sentence. Known to *whom* is the important, but unanswered, question. It may be a question which should be answered explicitly in SSL.com's CPS, as well as the CPS of all other CAs (and even, potentially, in the BRs -- although my understanding is that that little bit of the BRs may at some point in the near future get a tidy-up, per https://github.com/cabforum/documents/issues/164). If the appropriate answer is "known to SSL.com", then you could run your CA software with an empty blacklist, issue certs for all manner of known-weak keys, and still be compliant with your CPS. As such, that's probably not an acceptable answer to Mozilla. "Known to SSL.com's CA vendor" is similarly problematic. If, on the other hand, the appropriate interpretation is "known to anyone", then because the key was known to me, and I think I count as "anyone", your CPS was not followed. "Known to the vendor of the software which generated the known-bad key" is also an answer that results in your violating your CPS and the BRs, because the key you issued a certificate for was, as previously mentioned, included in the `openssl-blacklist` Debian package. Do you have another answer to the question "known to whom?" which SSL.com is using in that sentence? > Our understanding is that there is no single or complete list with all > known Debian weak keys, either one that is normative for use by the CAs > included in the Mozilla Root Program, nor one specified in the Baseline > Requirements. That is, I believe, correct, however (at the risk of tooting my own horn) there is quite a comprehensive collection of Debian weak keys in the pwnedkeys.com database. You are welcome to encourage your CA software vendor to perform lookups against the public API if you wish. I don't claim that any possible key you could generate from a buggy Debian system is already in pwnedkeys, but I've accumulated a fair collection of likely candidates, at great cost of (mostly emulated) CPU cycles. > This can be demonstrated by submitting the following CSR That CSR uses a 2048 bit key generated on an i386 system, using OpenSSH with an exponent of 3, with PID 23747. It might not be in certwatch, but it's in pwnedkeys. :grin: > We have strong indications that there are several different lists of > precalculated vulnerable keys whose precise populations depend on > combinations of: > > architecture Yes, different CPU word sizes and endianness produce different keys. That is covered in
Re: ssl.com: Certificate with Debian weak key
We have updated https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 with the findings of our current investigation. We believe all issues raised in this thread are addressed in this update. Our investigation is ongoing and we welcome any positive input by the community as an opportunity to improve our practices and communal PKI operations generally. Our team is evaluating possible improvements for this process, such as acquiring actual public keys created by the weak Debian RNG and using those keys to produce fingerprints compatible with our CA software. We have also contacted our CA software vendor regarding an improved blacklist. csk Chris Kemmerer SSL.com On Monday, March 9, 2020 at 3:41:09 PM UTC-5, Nick Lamb wrote: > On Sun, 8 Mar 2020 10:57:49 +1100 > Matt Palmer via dev-security-policy > wrote: > > > > The fingerpint of the claimed Debian weak key was not included in > > > our database. > > > > I think it's worth determining exactly where SSL.com obtained their > > fingerprint database of weak keys. The private key in my possession, > > which I generated for inclusion in the pwnedkeys.com database, was > > obtained by using the script provided in the `openssl-blacklist` > > source package, with no special options or modifications. > > Yes, I would certainly want SSL.com's report to give me confidence > that > > #1 they've identified why they didn't spot this key, were there (many?) > other keys which would also have been missed? > > #2 they now have a complete and accurate list of such keys > > #3 they went back and did the work to re-check other certificates > they've issued for this (these?) extra weak keys and any matches were > revoked and the subscriber contacted > > > Depending on the circumstances in #1 there may well be a lesson for > other CAs, especially if using a setup which is similar in some way to > SSL.com and so this point is very important. There might also be > further questions about SSL.com's processes which failed to detect this > mistake. > > This sort of incident is also important because of the impact on the > Subscriber. Had this subscriber used a different CA with a complete > list they'd have been informed immediately that their chosen key was a > problem. Because SSL.com didn't do that in fact this subscriber was > potentially vulnerable to active, and in some cases even passive > attacks on their TLS services for the period between issuance and > discovery. > > > Nick. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Sun, 8 Mar 2020 10:57:49 +1100 Matt Palmer via dev-security-policy wrote: > > The fingerpint of the claimed Debian weak key was not included in > > our database. > > I think it's worth determining exactly where SSL.com obtained their > fingerprint database of weak keys. The private key in my possession, > which I generated for inclusion in the pwnedkeys.com database, was > obtained by using the script provided in the `openssl-blacklist` > source package, with no special options or modifications. Yes, I would certainly want SSL.com's report to give me confidence that #1 they've identified why they didn't spot this key, were there (many?) other keys which would also have been missed? #2 they now have a complete and accurate list of such keys #3 they went back and did the work to re-check other certificates they've issued for this (these?) extra weak keys and any matches were revoked and the subscriber contacted Depending on the circumstances in #1 there may well be a lesson for other CAs, especially if using a setup which is similar in some way to SSL.com and so this point is very important. There might also be further questions about SSL.com's processes which failed to detect this mistake. This sort of incident is also important because of the impact on the Subscriber. Had this subscriber used a different CA with a complete list they'd have been informed immediately that their chosen key was a problem. Because SSL.com didn't do that in fact this subscriber was potentially vulnerable to active, and in some cases even passive attacks on their TLS services for the period between issuance and discovery. Nick. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On 07/03/2020 23:57, Matt Palmer via dev-security-policy wrote: As further independent confirmation, the crt.sh page for the certificate shows that crt.sh *also* identifies the certificate as having a Debian weak key. My understanding is that crt.sh uses a database of keys that was independently generated by the operator of the crt.sh service. For the crt.sh check, I augmented Debian's original blacklists with some other blacklists I generated ~12yrs ago for a few less common key sizes [1]. See also [2]. [1] https://secure.sectigo.com/debian_weak_keys/ [2] https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg10060.html - Matt -- Rob Stradling Senior Research & Development Scientist Email: r...@sectigo.com ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
On Sat, Mar 07, 2020 at 09:07:11AM -0500, Ryan Sleevi wrote: > Thanks. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 I'll give points to SSL.com for a speedy initial response, but I'm a bit disconcerted about this: > The fingerpint of the claimed Debian weak key was not included in our > database. I think it's worth determining exactly where SSL.com obtained their fingerprint database of weak keys. The private key in my possession, which I generated for inclusion in the pwnedkeys.com database, was obtained by using the script provided in the `openssl-blacklist` source package, with no special options or modifications. The key used in this certificate is not one for a niche architecture or unusual configuration -- i386 was the dominant architecture at the time of the flaw, and 2048 bits is a standard key size. It would be somewhat more understandable if the key was, say, a 4096 bit key generated on a MIPS machine (it took *ges* to generate all of those), although it would still be a Debian weak key and thus still be a BR violation to issue a certificate for it. On the off-chance that there *was* a mistake in my key generation procedure, *and* a one-in-a-trillion collision of private keys, I've confirmed that the public key of the certificate in question is in the `openssl-blacklist` Debian package (https://packages.debian.org/jessie/openssl-blacklist, uploaded in 2011) with the following command: grep $(wget -O - -q https://crt.sh/?d=2531502044 \ | openssl x509 -noout -pubkey \ | openssl rsa -pubin -noout -modulus \ | sha1sum | cut -d ' ' -f 1 | cut -c 21-) \ /usr/share/openssl-blacklist/blacklist.RSA-2048 As further independent confirmation, the crt.sh page for the certificate shows that crt.sh *also* identifies the certificate as having a Debian weak key. My understanding is that crt.sh uses a database of keys that was independently generated by the operator of the crt.sh service. - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: ssl.com: Certificate with Debian weak key
Thanks. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1620772 On Fri, Mar 6, 2020 at 9:48 PM Matt Palmer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > (Pre) Certificate https://crt.sh/?id=2531502044 has been issued with a > known > weak key, specifically Debian weak key 2048/i386/rnd/pid17691. I believe > this issuance to be in contravention of SSL.com's CPS, version 1.8, section > 6.1.1.2, which states "SSL.com shall reject a certificate request if the > request has a known weak Private Key". > > - Matt > > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
ssl.com: Certificate with Debian weak key
(Pre) Certificate https://crt.sh/?id=2531502044 has been issued with a known weak key, specifically Debian weak key 2048/i386/rnd/pid17691. I believe this issuance to be in contravention of SSL.com's CPS, version 1.8, section 6.1.1.2, which states "SSL.com shall reject a certificate request if the request has a known weak Private Key". - Matt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
