Re: Apple: Non-Compliant Serial Numbers
On May 3, Apple submitted an update to the original incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655), which is reposted below. Most certificates have been revoked and less than 1% of the total population of impacted certificates remains. As previously noted, it is expected that all impacted certificates will be revoked by July 15. In a previous update, we committed to doing the following: “The validator software that checks certificates for SSL Baseline compliance will be enhanced to evaluate collections of certificates instead of individual certificates. These enhancements are expected to be implemented by April 30, 2019.” The referenced software enhancements were implemented as planned prior to April 30. We expect to provide the next update soon after the July 15 milestone, on or about July 19. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
On April 6, Apple submitted an update to the original incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655), which is reposted below. Over 10,000 additional certificates have been revoked since our last update. In a previous update, we committed to doing the following: “Subsequent to suppressing the serial number check alert, and prior to identifying the current issue, a process was implemented to provide more oversight for changes to alerts. This enhanced process may have been sufficient to prevent the incorrect suppression of the serial number alert, but the process will be reviewed again to identify if further enhancements are required. This will be completed by March 31, 2019.” This review was completed. As a result, additional approval is required before alerts are modified and alerts will be tested on a quarterly basis to ensure they continue to function as intended. We expect to provide the next update soon after the April 30 milestone, on or about May 3 ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
On Monday, April 1, 2019 at 5:21:14 AM UTC-6, Jakob Bohm wrote: [Apple Responses] ___ > For the benefit of the community (including possible future creation of > policies for mass revocation scenarios), could you detail: > > 1. How many of the 54,583 certificates are issued to Apple owned and > operated servers and services and how many not. [Apple] All impacted certificates were issued to Apple entities. > 2. What kinds of practical issues are delaying the replacement of > certificates on any such Apple operated servers and services, > perhaps with approximate percentages. [Apple] A number of factors have impacted replacement timelines for the large number of disparate teams affected: - work required to migrate systems to an internal, enterprise PKI - QA and change control processes - prioritization of tasks - level of automation - freeze periods associated with existing operational schedules ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
> 1. How many of the 54,583 certificates are issued to Apple owned and > operated servers and services and how many not. All impacted certificates were issued to Apple entities > 2. What kinds of practical issues are delaying the replacement of > certificates on any such Apple operated servers and services, > perhaps with approximate percentages. A number of factors have impacted replacement timelines for the large number of disparate teams affected: - work required to migrate systems to an internal, enterprise PKI - QA and change control processes - prioritization of tasks - level of automation - freeze periods associated with existing operational schedules ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
On March 30, Apple submitted an update to the original incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655), which is reposted below. ___ We've been working our plan to revoke impacted certificates. Thus far over 500,000 certificates have been revoked since the issue was identified and 54,853 remain (file attached with remaining certificates [in teh Bugzilla post]). Our plan will result in all impacted certificates being revoked. Our approach to resolving this incident has been to strike a balance between a compliance incident with low associated security risk and impact to critical services. We have established a timeline to address the remaining certificates that minimizes service impact and allows standard QA and change control processes to ensure uptime is not affected. As part of the remediation plan, a number of certificates will be migrated to an internal, enterprise PKI, which will take more time. Based on these factors, it is expected that most certificates will be revoked by April 30 with less than 2% extending until July 15. Another update will be provided next week. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
On March 22, Apple submitted an update to the original incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1533655), which is reposted below. Over 115,000 additional certificates have been revoked since our last update leaving less than 10% of the total population of impacted certificates remaining (file attached with remaining certificates). We continue to work with teams across Apple to revoke the remaining certificates on a timeline that balances the non-compliance risk against the operational impact to users and the ecosystem at large. Another update will be provided next week. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Apple: Non-Compliant Serial Numbers
On March 11, 2019; Apple submitted a followup Incident Report. https://bugzilla.mozilla.org/show_bug.cgi?id=1533655. Incident Report How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. Apple first became aware that there might be an issue while reviewing the release notes for an updated version of the CA Software used for issuing publicly trusted certificates [1]. After further investigation, including a review of related discussions on the Mozilla Dev Security Policy forum, it was determined that certificates with non-compliant serial numbers were being issued. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 2016-09-30 - CA/Browser Forum requires the use of 64 bit entropy when generating serial numbers [2]. 2017-02-28 - The Mozilla Root Store Policy requires that certificates must have a serial number greater than zero (0) containing at least 64 bits of output from a CSPRNG [3]. 2017-04-10 3:45 PM - Mistakenly suppressed alerts detecting serial numbers suspected to be insufficient in length. 2019-03-05 8:00 AM - We became aware of a potential issue with our understanding and configuration of the CA Software used for issuing publicly trusted certificates, and began investigating. 2019-03-06 5:19 PM - Determined that certificates were being issued with non-compliant serial numbers. 2019-03-06 7:43 PM - Leadership engaged the team to begin gathering data to assess risk and determine next steps. 2019-03-06 9:21 PM - Performed an initial risk assessment and discussed next steps for remediating the issue including revocation of impacted certificates. 2019-03-07 12:13 AM - Generated an initial report on the volume and nature of the impacted certificates. 2019-03-07 8:19 AM - Notified DigiCert (the Root CA) of the incident. 2019-03-07 10:00-11:00 AM - Finalized the action plan to stop issuing non-compliant TLS Server certificates and initial steps taken. 2019-03-07 2:10 PM - Stopped issuance of TLS Server certificates with non-compliant serial numbers. 2019-03-07 2:17 PM - Re-instated alerts for detecting serial numbers suspected to be insufficient in length. 2019-03-07 3:00 PM - Met with DigiCert to further discuss the issue and remediation steps. 2019-03-07 3:55 PM - Determined that S/MIME certificates were also impacted and developed a plan to stop issuance. 2019-03-07 4:32 PM - Stopped issuance of S/MIME certificates with non-compliant serial numbers. 2019-03-07 5:00 PM - Began revocation impact assessment (see below for more details). 2019-03-07 11:19 PM - Posted preliminary incident report to Bugzilla [4]. 2019-03-08 10:00 AM - Notified Ernst & Young (WebTrust assessors). 2019-03-08 5:02 PM - Commenced revocation of impacted certificates. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. Issuance of non-compliant TLS Server certificates stopped on 2019-03-07 at 2:10 PM. Issuance of non-compliant S/MIME certificates stopped on 2019-03-07 at 4:32 PM. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. The number of certificates impacted are as follows: TLS Server Certificates (from Apple IST CA 2 - G1 (https://crt.sh/?id=5250464), Apple IST CA 8 - G1 (https://crt.sh/?id=21760447)) Total number of impacted certificates (issued since 2016-09-30): 877,253 Date the first certificate with the problem was issued: 2016-09-30 Date the last certificate with the problem was issued: 2019-03-07 S/MIME Certificates (from Apple IST CA 5 - G1 (https://crt.sh/?id=12716200)) Total number of impacted certificates (issued since 2017-02-28[3]): 2,224 Date the first certificate with the problem was issued: 2017-02-28 Date the last certificate with the problem was issued: 2019-03-07 The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. Files have been attached with the following data: List of all impacted certificates issued. List of impacted certificates still valid (not expired and not revoked) as of 5 days from when the incident was identified. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. When
Re: Apple: Non-Compliant Serial Numbers
Apple just submitted an updated report: Incident Report How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. Apple first became aware that there might be an issue while reviewing the release notes for an updated version of the CA Software used for issuing publicly trusted certificates [1]. After further investigation, including a review of related discussions on the Mozilla Dev Security Policy forum, it was determined that certificates with non-compliant serial numbers were being issued. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 2016-09-30 - CA/Browser Forum requires the use of 64 bit entropy when generating serial numbers [2]. 2017-02-28 - The Mozilla Root Store Policy requires that certificates must have a serial number greater than zero (0) containing at least 64 bits of output from a CSPRNG [3]. 2017-04-10 3:45 PM - Mistakenly suppressed alerts detecting serial numbers suspected to be insufficient in length. 2019-03-05 8:00 AM - We became aware of a potential issue with our understanding and configuration of the CA Software used for issuing publicly trusted certificates, and began investigating. 2019-03-06 5:19 PM - Determined that certificates were being issued with non-compliant serial numbers. 2019-03-06 7:43 PM - Leadership engaged the team to begin gathering data to assess risk and determine next steps. 2019-03-06 9:21 PM - Performed an initial risk assessment and discussed next steps for remediating the issue including revocation of impacted certificates. 2019-03-07 12:13 AM - Generated an initial report on the volume and nature of the impacted certificates. 2019-03-07 8:19 AM - Notified DigiCert (the Root CA) of the incident. 2019-03-07 10:00-11:00 AM - Finalized the action plan to stop issuing non-compliant TLS Server certificates and initial steps taken. 2019-03-07 2:10 PM - Stopped issuance of TLS Server certificates with non-compliant serial numbers. 2019-03-07 2:17 PM - Re-instated alerts for detecting serial numbers suspected to be insufficient in length. 2019-03-07 3:00 PM - Met with DigiCert to further discuss the issue and remediation steps. 2019-03-07 3:55 PM - Determined that S/MIME certificates were also impacted and developed a plan to stop issuance. 2019-03-07 4:32 PM - Stopped issuance of S/MIME certificates with non-compliant serial numbers. 2019-03-07 5:00 PM - Began revocation impact assessment (see below for more details). 2019-03-07 11:19 PM - Posted preliminary incident report to Bugzilla [4]. 2019-03-08 10:00 AM - Notified Ernst & Young (WebTrust assessors). 2019-03-08 5:02 PM - Commenced revocation of impacted certificates. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. Issuance of non-compliant TLS Server certificates stopped on 2019-03-07 at 2:10 PM. Issuance of non-compliant S/MIME certificates stopped on 2019-03-07 at 4:32 PM. summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. The number of certificates impacted are as follows: TLS Server Certificates (from Apple IST CA 2 - G1 (https://crt.sh/?id=5250464), Apple IST CA 8 - G1 (https://crt.sh/?id=21760447)) Total number of impacted certificates (issued since 2016-09-30): 877,253 Date the first certificate with the problem was issued: 2016-09-30 Date the last certificate with the problem was issued: 2019-03-07 S/MIME Certificates (from Apple IST CA 5 - G1 (https://crt.sh/?id=12716200)) Total number of impacted certificates (issued since 2017-02-28[3]): 2,224 Date the first certificate with the problem was issued: 2017-02-28 Date the last certificate with the problem was issued: 2019-03-07 The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. Files have been attached with the following data: List of all impacted certificates issued. List of impacted certificates still valid (not expired and not revoked) as of 5 days from when the incident was identified. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. When Ballot 164 [2] was passed, we confirmed that the CA Software was already
Apple: Non-Compliant Serial Numbers
Yesterday, Apple submitted this preliminary incident report: https://bugzilla.mozilla.org/show_bug.cgi?id=1533655, which is reposted below. On 2019-03-06 we determined that we were issuing certificates with non-compliant serial numbers because of the EJBCA issue [1]. We fixed the problem within 24 hours and stopped issuing non-compliant certificates as of the afternoon of 2019-03-07 and are working to address previously issued certificates. All impacted certificates were issued to Apple entities. To minimize impact to our users, we do not expect to revoke all impacted certificates within the 5-day requirement. We expect to provide a timeline for revoking all impacted certificates in a forthcoming update. Certificates issued by the following CA's were impacted: * Apple IST CA 2 - G1 (https://crt.sh/?id=5250464): TLS Server * Apple IST CA 8 - G1 (https://crt.sh/?id=21760447): TLS Server * Apple IST CA 5 - G1 (https://crt.sh/?id=12716200): S/MIME Based on our initial analysis, the number of certificates impacted are as follows: * TLS Server Certificates (from Apple IST CA 2 - G1, Apple IST CA 8 - G1) --Total number of impacted certificates (issued since 2016-09-30): ~878,000 --Total number of impacted certificates that are still valid (not expired and not revoked) as of 2019-03-07 3:20 PST: ~558,000 * S/MIME Certificates (from Apple IST CA 5 - G1) -- Total number of impacted certificates (issued since 2016-09-30): ~2,400 -- Total number of impacted certificates that are still valid (not expired and not revoked) as of 2019-03-07 3:20 PST: ~2,000 We expect to reply back with more details and a full incident report in a forthcoming update. [1] Configurable SN Entropy, Default Value Raised to 20 Octets (https://www.ejbca.org/docs/EJBCA_7.0.1_Release_Notes.html) ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
