Summary During a signing ceremony in October 2018, Google Trust Services generated OCSP responses for five of its subordinate CAs and published them afterwards. On 11 January 2019 it was discovered that one of these responses was not created accurately.
The incorrect OCSP response did not have an impact on subscribers or relying parties because the concerned CA GTSY1 is not in operation yet. By publishing this report we nevertheless want to share our lessons learned and hope that they will help other CAs improve their systems and processes. Cause and Detection A key ceremony tool was used to generate the OCSP responses based on a config file which specifies all relevant parameters of the output files to be created. The config file for the October ceremony was prepared, reviewed and tested following established procedures to ensure quality and conformity with the Baseline Requirements. Subsequently, it was submitted to a version control system. In a later review, a CA engineer discovered that some of the serial numbers in the file were in decimal and others in hex format. To make the number formats consistent across the file, they submitted a change list replacing the decimal numbers with their hex representation. The change list contained a paste error, which assigned the GTSY1 OCSP response a serial number which does not correspond to the serial number of the CA certificate. The change list was reviewed before it was submitted but the review did not catch the mismatch. On 11 January 2019 a CA engineer identified it while doing work to prepare for use of the previously generated certificate. Timeline 2018-10-18 Config file is created and submitted to version control system 2018-10-18 Change list is submitted against config file (convert decimal to hex serial numbers) 2018-10-29 OCSP responses are created and signed during a ceremony 2018-11-03 OCSP responses are published 2019-01-11 Serial mismatch is discovered and the root cause investigated. An impact assessment is performed and it yields that relying parties and subscribers are not affected. 2019-01-14 Generation of new OCSP response is planned and tested 2019-01-17 New OCSP response is generated 2019-01-18 New OCSP response is published Analysis and Findings The serial number mismatch was the result of a human error made when composing the change list. As a control against such errors we enforce the performance of code reviews before change lists can be submitted. The reviews are supported by a set of linters for various purposes. Tests for serial number consistency were not implemented at the time of the error, but have been added since that time. Additionally, OCSP responders are monitored and alerts are generated if certain conditions are met. CA specific alerting for GTSY1 was not configured when the OCSP response was published because the CA was not in an issuing state. Remediation -Correct OCSP response for GTSY1 has been prepared and published. -CA specific alerting was configured on the OCSP responder for all newly created subCAs regardless of their operating status. -Instructions for generating subCAs were improved to update the OCSP monitoring configuration without waiting for the subCA to be productionized. -Additional linters are being implemented which test the accuracy and plausibility of ceremony config files. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
