Please note, Google wrote this report for internal use immediately after the issue. We intended to post it to m.d.s.p at that time, but securing internal approvals took a while and the posting ended-up on the back burner for a bit. It was a minor issue, but we want the community to be aware of it.
Summary: May 21st 2018, a new tool for issuing certificates within Google was made available to internal customers. Within hours we started to receive reports that Chrome Canary (v67) with Certificate Transparency checks enabled was showing warnings. A coding error led to the new tool providing Signed Certificate Timestamps (SCTs) from 2 Google CT logs instead of one Google and one non-Google log. * NOTE: Affected certs were logged at issuance to at least 2 Google CT logs and 2 non-Google CT logs. The embedded SCTs for affected certs only provided proofs from Google logs instead of Google and non-Google logs as required by Chrome. * NOTE: The bug was due to an 'if/else' chain fall through. The code in question has been refactored to be simpler and more readable. The issue was fully resolved ~14 hours after initial notification. The issue was mitigated within 4 hours. Triage and code fixes happened within 11 hours and it took ~3 hours to deploy the fixed code and confirm the fixed behavior in production. The new code was running in relatively few locations, so deployment was quick compared to some changes in our infrastructure. Most affected customers responded quickly to communications that they should replace their certificates and revoke the old ones before a given deadline. All certificates that were issued with an SCT set that was not fully compliant were revoked on 2018-06-19 if they had not already been revoked by the customer previously. Most users replaced certificates shortly after notification. Timeline: 2018-03-22 Bug introduced to codebase. 2018-05-21 Push including bug became available to clients. 2018-05-22 08:05 UTC First user reports that Chrome Canary presents a CT warning for a certificate. 2018-05-22 09:25 UTC Bug filed with initial assessment. 2018-05-22 12:01 UTC Frontend jobs with the bug are taken offline following standard CA procedures. 2018-05-22 15:59 UTC Issue conclusively identified. 2018-05-22 19:07 UTC Fix is submitted. 2018-05-22 21:48 UTC Fix starts to be rolled out. 2018-05-22 22:16 UTC Fix fully deployed and tested on test instances followed by deployment to production. Access to frontends restored. 2018-05-24 Customer communication sent to affected users to ask them to renew their certificates and revoke the old ones. 2018-06-19 The final handful of certificates that had not already been revoked and replaced by users were revoked by the CA. Findings: * The operational plan to halt issuance worked as expected and was implemented quickly. * The problem was quickly found, fully understood and easy to remedy. * Tests existed, but did not cover this failure case. Remediation Plan * Completed ** Message of the Day (MOTD) functionality was added or improved for all issuance systems to make it easier to communicate issues to users when issuance is intentionally paused. ** Test coverage was expanded to ensure that both the quantity and type of SCTs are checked. _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
