On March 11, 2019; Apple submitted a followup Incident Report. https://bugzilla.mozilla.org/show_bug.cgi?id=1533655.
Incident Report How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. Apple first became aware that there might be an issue while reviewing the release notes for an updated version of the CA Software used for issuing publicly trusted certificates [1]. After further investigation, including a review of related discussions on the Mozilla Dev Security Policy forum, it was determined that certificates with non-compliant serial numbers were being issued. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 2016-09-30 - CA/Browser Forum requires the use of 64 bit entropy when generating serial numbers [2]. 2017-02-28 - The Mozilla Root Store Policy requires that certificates must have a serial number greater than zero (0) containing at least 64 bits of output from a CSPRNG [3]. 2017-04-10 3:45 PM - Mistakenly suppressed alerts detecting serial numbers suspected to be insufficient in length. 2019-03-05 8:00 AM - We became aware of a potential issue with our understanding and configuration of the CA Software used for issuing publicly trusted certificates, and began investigating. 2019-03-06 5:19 PM - Determined that certificates were being issued with non-compliant serial numbers. 2019-03-06 7:43 PM - Leadership engaged the team to begin gathering data to assess risk and determine next steps. 2019-03-06 9:21 PM - Performed an initial risk assessment and discussed next steps for remediating the issue including revocation of impacted certificates. 2019-03-07 12:13 AM - Generated an initial report on the volume and nature of the impacted certificates. 2019-03-07 8:19 AM - Notified DigiCert (the Root CA) of the incident. 2019-03-07 10:00-11:00 AM - Finalized the action plan to stop issuing non-compliant TLS Server certificates and initial steps taken. 2019-03-07 2:10 PM - Stopped issuance of TLS Server certificates with non-compliant serial numbers. 2019-03-07 2:17 PM - Re-instated alerts for detecting serial numbers suspected to be insufficient in length. 2019-03-07 3:00 PM - Met with DigiCert to further discuss the issue and remediation steps. 2019-03-07 3:55 PM - Determined that S/MIME certificates were also impacted and developed a plan to stop issuance. 2019-03-07 4:32 PM - Stopped issuance of S/MIME certificates with non-compliant serial numbers. 2019-03-07 5:00 PM - Began revocation impact assessment (see below for more details). 2019-03-07 11:19 PM - Posted preliminary incident report to Bugzilla [4]. 2019-03-08 10:00 AM - Notified Ernst & Young (WebTrust assessors). 2019-03-08 5:02 PM - Commenced revocation of impacted certificates. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. Issuance of non-compliant TLS Server certificates stopped on 2019-03-07 at 2:10 PM. Issuance of non-compliant S/MIME certificates stopped on 2019-03-07 at 4:32 PM. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. The number of certificates impacted are as follows: TLS Server Certificates (from Apple IST CA 2 - G1 (https://crt.sh/?id=5250464), Apple IST CA 8 - G1 (https://crt.sh/?id=21760447)) Total number of impacted certificates (issued since 2016-09-30): 877,253 Date the first certificate with the problem was issued: 2016-09-30 Date the last certificate with the problem was issued: 2019-03-07 S/MIME Certificates (from Apple IST CA 5 - G1 (https://crt.sh/?id=12716200)) Total number of impacted certificates (issued since 2017-02-28[3]): 2,224 Date the first certificate with the problem was issued: 2017-02-28 Date the last certificate with the problem was issued: 2019-03-07 The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. Files have been attached with the following data: List of all impacted certificates issued. List of impacted certificates still valid (not expired and not revoked) as of 5 days from when the incident was identified. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. When Ballot 164 [2] was passed, we confirmed that the CA Software was already configured for 64-bit serial numbers, but did not realize the serial number generation algorithm resulted in 63 bits of entropy. In addition to the CA Software, a validator (similar to linting tools) checks each issued certificate for compliance with SSL Baseline Requirements. Alerts were generated that the serial numbers were insufficient, however; because the checks were performed against individual certificates and not collections of certificates, the alerts were mistakenly determined to be incorrect and were suppressed. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. Apple stopped issuance of certificates with non-compliant serial numbers, and is continuing to work with users to revoke impacted certificates. CA Software has been configured to generate serial numbers with 16 octets, ensuring entropy greater than 64 bits. Re-instated alerts for detecting serial numbers suspected to be insufficient in length. The validator software that checks certificates for SSL Baseline compliance will be enhanced to evaluate collections of certificates instead of individual certificates. These enhancements are expected to be implemented by April 30, 2019. Subsequent to suppressing the serial number check alert, and prior to identifying the current issue, a process was implemented to provide more oversight for changes to alerts. This enhanced process may have been sufficient to prevent the incorrect suppression of the serial number alert, but the process will be reviewed again to identify if further enhancements are required. This will be completed by March 31, 2019. Revocation Status More than 355,000 certificates have been revoked since the incident was detected. 23% of total, impacted certificates issued were still valid (not expired and not revoked) as of 5 days from when the incident was identified. Further analysis is being performed to determine an appropriate schedule for revoking these certificates. While the CA had the capability to revoke all impacted certificates within 5 days and provides APIs enabling automated management of certificates, it was determined that for some certificates the operational risk outweighed the security risk and those certificates have not yet been revoked. In conjunction with teams responsible for assessing risk to users, risk assessments were performed. The operational risk was determined to be high because immediate revocation of certain impacted certificates would cause significant interruption to users and services. The security risk was determined to be low because the certificates are SHA256 signed, serial numbers were generated with 63 bits of entropy, and all certificates were issued to Apple entities. We are working with the remaining teams across Apple to determine an appropriate timeline for revoking the remaining certificates that balances the non-compliance risk against the operational impact to users and the ecosystem at large. We expect to provide more details in an update as early as March 21, 2019. [1] Configurable SN Entropy, Default Value Raised to 20 Octets (https://www.ejbca.org/docs/EJBCA_7.0.1_Release_Notes.html) [2] CA/Browser Forum Ballot 164 (https://cabforum.org/pipermail/public/2016-June/007861.html) [3] Mozilla Root Store Policy, version 2.4, section Maintaining Confidence in Included Root Certificates, number 7 (https://wiki.mozilla.org/CA/Root_Store_Policy_Archive) [4] DigiCert: Apple: Non-compliant Serial Numbers Mozilla Bug Report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655) _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
