Re: GoDaddy Underscore Revocation Disclosure
On Friday, February 8, 2019 at 7:25:08 PM UTC-8, Jakob Bohm wrote: > On 09/02/2019 01:36, Santhan Raj wrote: > > On Friday, February 8, 2019 at 4:09:32 PM UTC-8, Joanna Fox wrote: > >> I agree on the surface this bug appears to be the same, but the root cause > >> is a different. The issue for bug 1462844 was a specific status not > >> counting as active when it was. To mitigate this issue, we updated the > >> query to include the missing status. However, we are in the process of > >> simplifying the data structures to simplify these types of queries. > >> > >> For the underscore certificates, these were non-active, not even > >> considered as provisioned since they were not delivered to a customer and > >> not publicly used for any encryption. These certificates were effectively > >> abandoned by our system. > > > > Is the term "certificate" accurate in this case? Assuming you embed SCTs > > within the EE cert, what you have is technically a pre-cert that was > > abandoned (not meant to be submitted to CT). Right? I ask because both the > > cert you linked are pre-certs, and I understand signing a pre-cert is > > intent to issue and is treated the same way, but still wanted to clarify. > > > > Or by non-active certificate, are you actually referring to a fully signed > > EE that was just not delivered to the customer? > > > > And in either case, the only reasonable sequences of events within > expectations (and possibly within requirements) would have been: > > Good Scenario A: > 1. Pre-certificate is logged in CT. > 2. Matching certificate is signed by CA key. > 3. Signed certificate is logged in CT. > 4. At this point the customer _might_ retrieve the certificate from CT >without knowledge of CA. > 5. Thus certificate is in reality active, despite any records the CA >may have of not delivering it directly to customer. > > Good Scenario B: > 1. Pre-certificate is logged in CT. > 2. CA decides (for any reason) not to actually sign the actual >certificate. > 3. The serial number listed in the CT pre-certificate is formally >revoked in CRL and OCSP. This is done once the decision not to >sign is final. > 4. If possible, label these revocations differently in CRL and OCSP >responses, to indicate to independent CT auditors that the EE was >never signed and therefore not logged to CT. > > Scenario B should be very rare, as the process from pre-cert logging to > final cert logging should typically be automatic or occur within a > single root key ceremony. Basically, something unusual would have to > happen at the very last moment, such as an incoming report that a > relied-upon external system (Global DNS, Internet routing, reliable > identity database etc.) may have been compromised during the vetting. > > In neither case would a CT-logged serial number be left in a > not-active but not-revoked state beyond a relevant procedural > delay. > > As pointed out in other recent cases, CA software must allow > revoking a certificate without making it publicly valid first, in > case Scenario B happens. > > > > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > This public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded My assumption is, 1) Pre-cert is generated 2) something breaks 3) pre-cert is never submitted to CT (and is termed a non-active certificate) The basis for my assumption is "In rare cases, due to an order of operation problem, non-active certificates were being logged to CT." and in another response "These non-active certificates were revoked and we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks. " There is no reason to "prevent non-active certificates from being logged to CT" if the pre-cert is already in CT. To me, it makes sense to say "prevent non-active pre-cert from being logged to CT". Thanks, Santhan ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: GoDaddy Underscore Revocation Disclosure
On 09/02/2019 01:36, Santhan Raj wrote: On Friday, February 8, 2019 at 4:09:32 PM UTC-8, Joanna Fox wrote: I agree on the surface this bug appears to be the same, but the root cause is a different. The issue for bug 1462844 was a specific status not counting as active when it was. To mitigate this issue, we updated the query to include the missing status. However, we are in the process of simplifying the data structures to simplify these types of queries. For the underscore certificates, these were non-active, not even considered as provisioned since they were not delivered to a customer and not publicly used for any encryption. These certificates were effectively abandoned by our system. Is the term "certificate" accurate in this case? Assuming you embed SCTs within the EE cert, what you have is technically a pre-cert that was abandoned (not meant to be submitted to CT). Right? I ask because both the cert you linked are pre-certs, and I understand signing a pre-cert is intent to issue and is treated the same way, but still wanted to clarify. Or by non-active certificate, are you actually referring to a fully signed EE that was just not delivered to the customer? And in either case, the only reasonable sequences of events within expectations (and possibly within requirements) would have been: Good Scenario A: 1. Pre-certificate is logged in CT. 2. Matching certificate is signed by CA key. 3. Signed certificate is logged in CT. 4. At this point the customer _might_ retrieve the certificate from CT without knowledge of CA. 5. Thus certificate is in reality active, despite any records the CA may have of not delivering it directly to customer. Good Scenario B: 1. Pre-certificate is logged in CT. 2. CA decides (for any reason) not to actually sign the actual certificate. 3. The serial number listed in the CT pre-certificate is formally revoked in CRL and OCSP. This is done once the decision not to sign is final. 4. If possible, label these revocations differently in CRL and OCSP responses, to indicate to independent CT auditors that the EE was never signed and therefore not logged to CT. Scenario B should be very rare, as the process from pre-cert logging to final cert logging should typically be automatic or occur within a single root key ceremony. Basically, something unusual would have to happen at the very last moment, such as an incoming report that a relied-upon external system (Global DNS, Internet routing, reliable identity database etc.) may have been compromised during the vetting. In neither case would a CT-logged serial number be left in a not-active but not-revoked state beyond a relevant procedural delay. As pointed out in other recent cases, CA software must allow revoking a certificate without making it publicly valid first, in case Scenario B happens. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: GoDaddy Underscore Revocation Disclosure
I agree on the surface this bug appears to be the same, but the root cause is a different. The issue for bug 1462844 was a specific status not counting as active when it was. To mitigate this issue, we updated the query to include the missing status. However, we are in the process of simplifying the data structures to simplify these types of queries. For the underscore certificates, these were non-active, not even considered as provisioned since they were not delivered to a customer and not publicly used for any encryption. These certificates were effectively abandoned by our system. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: GoDaddy Underscore Revocation Disclosure
Good catch, thanks Wayne! I also realized it sounds very similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1526154 from DigiCert, which similarly included an overly-restrictive query. It's also similar to Symantec's incident investigation in https://wiki.mozilla.org/CA:Symantec_Issues#Issue_D:_Test_Certificate_Misissuance_.28April_2009_-_September_2015.29 , which saw multiple revisions to the set of misissued certificates - as also captured at https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html . I highlight these to indicate that there is a systemic pattern of CAs having trouble detecting the set of misissued certificates or certificates that require action. When faced with systemic patterns, especially well-publicized ones, I do think it's reasonable to expect a higher bar in both the response and the remediation plans. In particular, each CA that has such an incident should examine past patterns (at both their and other CAs) and come up with comprehensive plans to address. Other CAs - those that have not had such issues - should also consider implementing similar best practices, as the goal of these reports is to highlight gaps and develop and operationalize best practices. Failure to do so will be seen very unfavorably - as the comment on https://bugzilla.mozilla.org/show_bug.cgi?id=1462844#c9 explicitly called out. On Fri, Feb 8, 2019 at 3:53 PM Wayne Thayer wrote: > Perhaps more concerning, this sounds a lot like bug #1462844 in which > misissued certificates were reported that had not been found and revoked > despite GoDaddy having previously scanned their database for the issue. > GoDaddy never identified or described how they would remediate the cause of > that bug. > > On Fri, Feb 8, 2019 at 1:28 PM Ryan Sleevi via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > >> Thanks Joanna, >> >> Just to make sure - this is >> https://bugzilla.mozilla.org/show_bug.cgi?id=1524815 correct? >> >> If so, this sounds remarkably similar to the root cause analysis that >> Entrust shared, in https://bugzilla.mozilla.org/show_bug.cgi?id=1521520 . >> Similar to that issue, please provide an update as to what steps are being >> taken to ensure that future compliance objectives are met - that is, what >> process changes have been made with respect to the data sources you use, >> how tools will be reviewed that use those data sources, and what steps may >> be taken to improve the quality and accuracy of those data sources (e.g. >> disclosing active certificates to CT) >> >> While Mozilla policy does not require or mandate the disclosure to CT, >> those non-active certificates are still relevant to the BR expectations of >> compliance, and should they be discovered (e.g. via auditors or if they >> were to be shared outside GoDaddy), the BR non-compliance may appear >> significantly worse to the community. Rather than not logging non-active >> certificates, it may be more useful to approach this by ensuring your >> tools >> consider both active and non-active certificates - that is, anything that >> has been signed with the CA's private key. >> >> On Fri, Feb 8, 2019 at 2:44 PM Joanna Fox via dev-security-policy < >> dev-security-policy@lists.mozilla.org> wrote: >> >> > GoDaddy received a certificate problem report on 1/29/2019 for 2 >> unrevoked >> > unexpired certificates have underscores in the DNS name that did not >> meet >> > the January 15th deadline for revocation. The certificates reported are >> as >> > follows: >> > https://crt.sh/?opt=zlint=626981823 >> > https://crt.sh/?opt=zlint=637047181 >> > >> > 1. How your CA first became aware of the problem (e.g. via a >> problem >> > report submitted to your Problem Reporting Mechanism, a discussion in >> > mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), >> and >> > the time and date. >> > >> > Certificate Problem Reporting via email address supplied in CP/CPS on >> > Tuesday, January 29, 2019 11:18:42 AM >> > >> > 2. A timeline of the actions your CA took in response. A timeline >> is >> > a date-and-time-stamped sequence of all relevant events. This may >> include >> > events before the incident was reported, such as when a particular >> > requirement became applicable, or a document changed, or a bug was >> > introduced, or an audit was done. >> > >> > 1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy >> > 1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and >> > escalated appropriately. >> > 1/29/2019 to 1/30/2019 Researched as to why these certificates were not >> > caught in the initial data set for underscore revocation. >> > 1/30/2019 Identified root cause as order of an operation problem where >> > certificates could be CT logged but never be delivered to the >> certificate >> > requester. >> > 1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were >> > reported are revoked. >> > >> > 3. Whether your CA
Re: GoDaddy Underscore Revocation Disclosure
Perhaps more concerning, this sounds a lot like bug #1462844 in which misissued certificates were reported that had not been found and revoked despite GoDaddy having previously scanned their database for the issue. GoDaddy never identified or described how they would remediate the cause of that bug. On Fri, Feb 8, 2019 at 1:28 PM Ryan Sleevi via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Thanks Joanna, > > Just to make sure - this is > https://bugzilla.mozilla.org/show_bug.cgi?id=1524815 correct? > > If so, this sounds remarkably similar to the root cause analysis that > Entrust shared, in https://bugzilla.mozilla.org/show_bug.cgi?id=1521520 . > Similar to that issue, please provide an update as to what steps are being > taken to ensure that future compliance objectives are met - that is, what > process changes have been made with respect to the data sources you use, > how tools will be reviewed that use those data sources, and what steps may > be taken to improve the quality and accuracy of those data sources (e.g. > disclosing active certificates to CT) > > While Mozilla policy does not require or mandate the disclosure to CT, > those non-active certificates are still relevant to the BR expectations of > compliance, and should they be discovered (e.g. via auditors or if they > were to be shared outside GoDaddy), the BR non-compliance may appear > significantly worse to the community. Rather than not logging non-active > certificates, it may be more useful to approach this by ensuring your tools > consider both active and non-active certificates - that is, anything that > has been signed with the CA's private key. > > On Fri, Feb 8, 2019 at 2:44 PM Joanna Fox via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > GoDaddy received a certificate problem report on 1/29/2019 for 2 > unrevoked > > unexpired certificates have underscores in the DNS name that did not meet > > the January 15th deadline for revocation. The certificates reported are > as > > follows: > > https://crt.sh/?opt=zlint=626981823 > > https://crt.sh/?opt=zlint=637047181 > > > > 1. How your CA first became aware of the problem (e.g. via a problem > > report submitted to your Problem Reporting Mechanism, a discussion in > > mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and > > the time and date. > > > > Certificate Problem Reporting via email address supplied in CP/CPS on > > Tuesday, January 29, 2019 11:18:42 AM > > > > 2. A timeline of the actions your CA took in response. A timeline is > > a date-and-time-stamped sequence of all relevant events. This may include > > events before the incident was reported, such as when a particular > > requirement became applicable, or a document changed, or a bug was > > introduced, or an audit was done. > > > > 1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy > > 1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and > > escalated appropriately. > > 1/29/2019 to 1/30/2019 Researched as to why these certificates were not > > caught in the initial data set for underscore revocation. > > 1/30/2019 Identified root cause as order of an operation problem where > > certificates could be CT logged but never be delivered to the certificate > > requester. > > 1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were > > reported are revoked. > > > > 3. Whether your CA has stopped, or has not yet stopped, issuing > > certificates with the problem. A statement that you have will be > considered > > a pledge to the community; a statement that you have not requires an > > explanation. > > > > We have prevented new certificates with underscores from being > provisioned > > since November 8, 2018. > > > > 4. The complete certificate data for the problematic certificates. > > The recommended way to provide this is to ensure each certificate is > logged > > to CT and then list the fingerprints or crt.sh IDs, either in the report > or > > as an attached spreadsheet, with one list per distinct problem. > > > > https://crt.sh/?opt=zlint=626981823 > > https://crt.sh/?opt=zlint=637047181 > > > > 5. Explanation about how and why the mistakes were made or bugs > > introduced, and how they avoided detection until now. > > > > Our initial effort to bring underscore certificates into compliance was > > dependent upon an ‘active’ flag in our database. In rare cases, due to an > > order of operation problem, non-active certificates were being logged to > > CT. These non-active certificates were fully vetted meeting the BR > > standards but failed to be delivered to the certificate requester due to > a > > software defect. > > > > 6. List of steps your CA is taking to resolve the situation and > > ensure such issuance will not be repeated in the future, accompanied > with a > > timeline of when your CA expects to accomplish these things. > > > > We have prevented new certificates
Re: GoDaddy Underscore Revocation Disclosure
Thanks Joanna, Just to make sure - this is https://bugzilla.mozilla.org/show_bug.cgi?id=1524815 correct? If so, this sounds remarkably similar to the root cause analysis that Entrust shared, in https://bugzilla.mozilla.org/show_bug.cgi?id=1521520 . Similar to that issue, please provide an update as to what steps are being taken to ensure that future compliance objectives are met - that is, what process changes have been made with respect to the data sources you use, how tools will be reviewed that use those data sources, and what steps may be taken to improve the quality and accuracy of those data sources (e.g. disclosing active certificates to CT) While Mozilla policy does not require or mandate the disclosure to CT, those non-active certificates are still relevant to the BR expectations of compliance, and should they be discovered (e.g. via auditors or if they were to be shared outside GoDaddy), the BR non-compliance may appear significantly worse to the community. Rather than not logging non-active certificates, it may be more useful to approach this by ensuring your tools consider both active and non-active certificates - that is, anything that has been signed with the CA's private key. On Fri, Feb 8, 2019 at 2:44 PM Joanna Fox via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > GoDaddy received a certificate problem report on 1/29/2019 for 2 unrevoked > unexpired certificates have underscores in the DNS name that did not meet > the January 15th deadline for revocation. The certificates reported are as > follows: > https://crt.sh/?opt=zlint=626981823 > https://crt.sh/?opt=zlint=637047181 > > 1. How your CA first became aware of the problem (e.g. via a problem > report submitted to your Problem Reporting Mechanism, a discussion in > mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and > the time and date. > > Certificate Problem Reporting via email address supplied in CP/CPS on > Tuesday, January 29, 2019 11:18:42 AM > > 2. A timeline of the actions your CA took in response. A timeline is > a date-and-time-stamped sequence of all relevant events. This may include > events before the incident was reported, such as when a particular > requirement became applicable, or a document changed, or a bug was > introduced, or an audit was done. > > 1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy > 1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and > escalated appropriately. > 1/29/2019 to 1/30/2019 Researched as to why these certificates were not > caught in the initial data set for underscore revocation. > 1/30/2019 Identified root cause as order of an operation problem where > certificates could be CT logged but never be delivered to the certificate > requester. > 1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were > reported are revoked. > > 3. Whether your CA has stopped, or has not yet stopped, issuing > certificates with the problem. A statement that you have will be considered > a pledge to the community; a statement that you have not requires an > explanation. > > We have prevented new certificates with underscores from being provisioned > since November 8, 2018. > > 4. The complete certificate data for the problematic certificates. > The recommended way to provide this is to ensure each certificate is logged > to CT and then list the fingerprints or crt.sh IDs, either in the report or > as an attached spreadsheet, with one list per distinct problem. > > https://crt.sh/?opt=zlint=626981823 > https://crt.sh/?opt=zlint=637047181 > > 5. Explanation about how and why the mistakes were made or bugs > introduced, and how they avoided detection until now. > > Our initial effort to bring underscore certificates into compliance was > dependent upon an ‘active’ flag in our database. In rare cases, due to an > order of operation problem, non-active certificates were being logged to > CT. These non-active certificates were fully vetted meeting the BR > standards but failed to be delivered to the certificate requester due to a > software defect. > > 6. List of steps your CA is taking to resolve the situation and > ensure such issuance will not be repeated in the future, accompanied with a > timeline of when your CA expects to accomplish these things. > > We have prevented new certificates with underscores from being provisioned > since November 8, 2018. These non-active certificates were revoked and we > are taking steps to prevent non-active certificates from being logged to CT > within the next 2 weeks. > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
GoDaddy Underscore Revocation Disclosure
GoDaddy received a certificate problem report on 1/29/2019 for 2 unrevoked unexpired certificates have underscores in the DNS name that did not meet the January 15th deadline for revocation. The certificates reported are as follows: https://crt.sh/?opt=zlint=626981823 https://crt.sh/?opt=zlint=637047181 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. Certificate Problem Reporting via email address supplied in CP/CPS on Tuesday, January 29, 2019 11:18:42 AM 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy 1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and escalated appropriately. 1/29/2019 to 1/30/2019 Researched as to why these certificates were not caught in the initial data set for underscore revocation. 1/30/2019 Identified root cause as order of an operation problem where certificates could be CT logged but never be delivered to the certificate requester. 1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were reported are revoked. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. We have prevented new certificates with underscores from being provisioned since November 8, 2018. 4. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. https://crt.sh/?opt=zlint=626981823 https://crt.sh/?opt=zlint=637047181 5. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. Our initial effort to bring underscore certificates into compliance was dependent upon an ‘active’ flag in our database. In rare cases, due to an order of operation problem, non-active certificates were being logged to CT. These non-active certificates were fully vetted meeting the BR standards but failed to be delivered to the certificate requester due to a software defect. 6. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. We have prevented new certificates with underscores from being provisioned since November 8, 2018. These non-active certificates were revoked and we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Discrepancy on Address
On Friday, February 8, 2019 at 4:20:14 AM UTC-5, Kurt Roeckx wrote: > On 2019-02-08 1:04, identr...@gmail.com wrote: > > On Thursday, February 7, 2019 at 6:47:03 PM UTC-5, iden...@gmail.com wrote: > >> On 04/04/2018 we found a discrepancy in the address values for some SSL > >> certificates. A formal incident Report was just posted: > >> https://bugzilla.mozilla.org/show_bug.cgi?id=1526099 > > > > CORRECTION: This issue was found last Monday 4/4/2019 and it this date is > > properly reflected in the logged bug.Sorry about this typo. > > I guess the 4th of February ... Yes, the discrepancy was on February 4, 2019. > > > Kurt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Discrepancy on Address
On 2019-02-08 1:04, identr...@gmail.com wrote: On Thursday, February 7, 2019 at 6:47:03 PM UTC-5, iden...@gmail.com wrote: On 04/04/2018 we found a discrepancy in the address values for some SSL certificates. A formal incident Report was just posted: https://bugzilla.mozilla.org/show_bug.cgi?id=1526099 CORRECTION: This issue was found last Monday 4/4/2019 and it this date is properly reflected in the logged big.Sorry about this typo. I guess the 4th of February ... Kurt ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
