As of 9pm AZ on 3/6/2019 GoDaddy started researching the 64bit certificate Serial Number issue. Due to a m.d.s.p.[1] discussion validating an interpretation of BR 7.1 our revised count is approximately 12,152 live certificates not meeting the 64bit serial number requirement. Additionally, we have identified 273,784 “orphaned” certificates meeting the initial interpretation of BR 7.1. Orphaned certificates are certs, which were stopped mid-issuance due to a variety of reasons like requestor cancellation, system errors etc. These certs are most often pre-certificates, but some are leaf-certificates, which were logged to CT, but never received by the certificate requestor.
The initial report stated >1.8 million certificates were impacted. For our initial investigation we checked certs against the first bit being set, which seemed to be the industry interpretation at the time. This lead to more aggressive criteria than necessary. As we started revocations of orphan certificates we continued researching the criteria defined by BR7.1 After re-evaluating the criteria, per m.d.s.p.[1], we adjusted our certificate scope. 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. 9pm 3/6/2019 AZ Time, due to reviewing a discussion in mozilla.dev.security.policy. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 9pm 3/6/2019 AZ Time, identified a hot issue with serial numbers in Mozilla group. 10am 3/7/2019 AZ Time, identified the issue was pervasive, and identified root cause. 6:30pm 3/7/2019 AZ Time, fix deployed to production to correct the serial number issue. 2pm 3/9/2019 AZ Time, we revoked 273,784 orphaned, pre-certs and leaf certificates which met the initial criteria. These certificates were low\no risk as they were never distributed. 11pm 3/11/2019 AZ Time, defined resolution as stated in m.d.s.p., further research revised the quantity of impacted certificates. 3/12 – 3/16 – We will be revoking the before mentioned 12,152 live certificates. Once the revocation is complete we will update this timeline. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. We have deployed a fix to the issue on 3/7/2019, and are no longer issuing certificates with the defect. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 12,134 certificates were affected. Revoking as a precaution, certificates issued on 9/29/2016 9/29/2016 2:41 66269447367104810 9/29/2016 7:44 35092532380173749 9/29/2016 9:46 43324527254073466 9/29/2016 14:16 53640950198707040 9/29/2016 14:31 36562688867169546 The first affected certificate was issued on 9/30/2016 1:29 https://crt.sh/?id=290271291 The last affected certificate was issued on 3/7/2019 15:32 https://crt.sh/?id=1262876175 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. Spreadsheet attached to defect. Older certificates may not yet be CT logged, as the majority of the certs are DV. 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. Ambiguity in language led to different interpretations of BR 7.1 in 2016. It was believed an unsigned 64bit integer was sufficient to satisfy the new requirement. Additionally, industry tools like CABLint/ZLint were not catching this issue, which provided a false sense of compliance. We are submitting CABLint/Zlint updates as part of the fix. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. Defect has been resolved, we are also updating linting tools (CABLint/Zlint) and upstreaming to patch for other peoples usage. Additionally, we are looking to scope and roadmap upgrading our certificate serial number to a minimum of 128-bit, or the max possible. [1] https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/7WuWS_20758 _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
