Sleevi, Thanks you for the links to both the reporting requirements and the underscore issue with DigiCert.
Regarding the statement about the severity of the issue, it was not intended to diminish the non-compliance. Instead it was an attempt to frame the issue with sufficient context to help others who follow this thread answer the question of impact on the community. As for the incident response, we have been making every effort to gather complete and accurate information to enable us to provide a useful and actionable incident report. This has unfortunately taken longer than we had hoped. We now have that report ready and you can find it below. You are right that some affected certificates could not be revoked within the 5 day requirement. In the last section of the report you'll find more information about the reasons for this along with our plan for revoking the remaining certificates. Ryan Hurst Google Trust Services Product Manager Summary --- Some certificates issued by GTS utilize EJBCA and as a result had serial numbers with an effective entropy of 63 bits. These serial numbers were created from a 64 bit CSPRNG output and were believed to be in compliance with Section 7.1 of the Baseline Requirements. Upon closer investigation we learned that EJBCA’s logic for serial number generation only selected output numbers having a leading 0 bit, which reduced their effective entropy to 63 bits. Though GTS agree that the issuance of the certificates based on the above behavior qualifies as missisuance, we also believe that this issue does not represent a material security risk to the community. To ensure that all of our certificates comply with the community’s interpretation of the Baseline Requirements (BR) we have updated the associated EJBCA CAs to mitigate the problematic behaviour. At this time approximately 95% of the affected certificates have been replaced and revoked. The remaining certificates expire over the next 3 months. We are actively working with the subscribers of these remaining certificates to facilitate a replacement with the goal of minimizing disruption of services. Should this not be possible, we will revoke these certificates no later than 2019-03-31.. Certificates issued from non-EJBCA CAs have been checked and are not affected. Incident Report --- 1. How your CA first became aware of the problem We have been following the thread discussing Dark Matter’s root inclusion request. When concerns regarding the EJBCA serial number generation logic were raised, we analyzed the behaviour of our EJBCA installations and found that they were affected as well. 2. A timeline of the actions your CA took in response. 2019-02-22 - A thread on m.d.s.p. mentions serial entropy issue of Dark Matter certificates. 2019-02-26 - GTS begins reviewing the serial number generation behaviour of its CAs. 2019-02-27 - A third-party reports that serial numbers in all certificates issued from a specific GTS CA have a leading bit of 0 and suggests that we may have the same issue as Dark Matter. 2019-02-27 - GTS requests clarification from PrimeKey. It is confirmed that EJBCA serial generation logic causes the issue and that in order to create compliant serial numbers, the logic has to be replaced. 2019-02-27 - The associated CAs used an earlier version of EJBCA where the serial number logic was not configurable. As a result code from a newer version of EJBCA that supports configurable serial number length is backported and configured to use 16 byte serials. 2019-02-28 - The backported code is deployed to production. 2019-02-28 - Ongoing discussion on m.d.s.p. revolves around interpretation of Section 7.1 BR. A consensus emerges that the affected certificates must be considered to have been misissued. 2019-02-28 - We inventory the number of certificates issued since Section 7.1 BR went into effect in September 2016, the number that were currently valid as well as their validity period. The results are provided in the section on remediation actions below. 2019-03-01 - GTS decides to replace and revoke all affected certificates. Customers are contacted to work out revocation plans. 2019-03-01 - Issuance of replacement certificates begins. 2019-03-02 - A first notification is posted to m.d.s.p 2019-03-04 - Certificate revocation begins. 2019-03-05 - An update on progress is posted to m.d.s.p 2019-03-05 - This post mortem is posted to m.d.s.p 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. GTS has stopped using the incorrect serial number generation logic. As of 2019-02-28, all GTS certificates issued from its EJBCA CAs have serial numbers with at least 64 bits of entropy. 4. A summary of the problematic certificates. All certificates issued from GIAG3 (https://crt.sh/?id=109354897, https://crt.sh/?id=158511650) between 2016-09-30 and 2019-02-28 were affected. 5. The complete certificate data for the problematic certificates. Given the large number of certificates (> 100k) they are not enumerated here. All certificates issued from the CAs mentioned above are affected. The CA certificates themselves are not affected. All affected certs were logged to multiple CT logs. 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. GTS relied on the affected EJBCA CA to generate 64 bit long certificate serial numbers as per its configuration and relied on EJBCA to perform this serial number generation in a BR compliant manner. Since the serial numbers had the expected length, the leading bit of 0 was not discovered as limiting the serial number space. Regular internal audits examined the technical profile of samples of issued certificates but did not detect the issue, because the audit was performed on a certificate-by-certificate basis. A comparison of serial numbers across a larger certificate population would have been required to identify it. Such tests were not implemented for the affected EJBCA CA because it is a legacy system scheduled for decommissioning later this year. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future. The issue only affects certificates from one of our legacy EJBCA CAs. Most of the GTS issuance volume has previously been migrated to a bespoke CA that is not affected. Since the affected CA was patched with updated logic, it is issuing with 16 byte serials. At the end of August 2019, it will be decommissioned. An inventory of the affected certificates provided the following: Total number issued since September 2016 - 100,836 Total valid as of 2019-02-28 - 7,171 Expiring within 90 days - 7,137 Revocation Schedule --- Revocation was performed with the objective of meeting the timeline prescribed for cases that fall under the second paragraph of Section 4.9.1 BR (5 days). by 2019-03-04 : 6,199 (86.5%) by 2019-03-05 : 575 (8%) The remaining 397 certificates (5.5%) will expire or be revoked no later than 2019-03-31. For 86.5% of the affected certificates the BR revocation timeline could be met. 954 certificates could not be replaced within the five day target without causing outages or other business issues. Revoking these certificates without providing adequate time for the associated subscribers to replace the certificates certificates without providing adequate time for the associated subscribers to replace the certificates would have caused significant business damage to the respective subscribers. Our goal being to revoke all certificates as quickly as possible while minimizing disruption on the subscribers and the users of their services. We are actively working with remaining subscribers who have affected certs which have not been revoked to replace their certificates in the shortest time window possible. The 397 remaining certificates will expire or be revoked by 2019-03-31. _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
