Hi Andy, Just so I follow, this is something you're proactively sharing, right? As far as I can tell, there's no violation of any Mozilla Root Program rules here, just an issue that caused interstitials in Chrome.
Either way, I appreciate your sharing. You mentioned the issue was do to some overly complex control flow. In order to help other CAs out, do you think there are testing methodologies that could have helped catch this earlier? Alex On Thu, Aug 23, 2018 at 8:50 AM Andy Warner via dev-security-policy < [email protected]> wrote: > Please note, Google wrote this report for internal use immediately after > the issue. We intended to post it to m.d.s.p at that time, but securing > internal approvals took a while and the posting ended-up on the back burner > for a bit. It was a minor issue, but we want the community to be aware of > it. > > Summary: > > May 21st 2018, a new tool for issuing certificates within Google was made > available to internal customers. Within hours we started to receive reports > that Chrome Canary (v67) with Certificate Transparency checks enabled was > showing warnings. A coding error led to the new tool providing Signed > Certificate Timestamps (SCTs) from 2 Google CT logs instead of one Google > and one non-Google log. > > * NOTE: Affected certs were logged at issuance to at least 2 Google CT > logs and 2 non-Google CT logs. The embedded SCTs for affected certs only > provided proofs from Google logs instead of Google and non-Google logs as > required by Chrome. > > * NOTE: The bug was due to an 'if/else' chain fall through. The code in > question has been refactored to be simpler and more readable. > > The issue was fully resolved ~14 hours after initial notification. The > issue was mitigated within 4 hours. Triage and code fixes happened within > 11 hours and it took ~3 hours to deploy the fixed code and confirm the > fixed behavior in production. The new code was running in relatively few > locations, so deployment was quick compared to some changes in our > infrastructure. > > Most affected customers responded quickly to communications that they > should replace their certificates and revoke the old ones before a given > deadline. All certificates that were issued with an SCT set that was not > fully compliant were revoked on 2018-06-19 if they had not already been > revoked by the customer previously. Most users replaced certificates > shortly after notification. > > Timeline: > > 2018-03-22 Bug introduced to codebase. > 2018-05-21 Push including bug became available to clients. > 2018-05-22 08:05 UTC First user reports that Chrome Canary presents a CT > warning for a certificate. > 2018-05-22 09:25 UTC Bug filed with initial assessment. > 2018-05-22 12:01 UTC Frontend jobs with the bug are taken offline > following standard CA procedures. > 2018-05-22 15:59 UTC Issue conclusively identified. > 2018-05-22 19:07 UTC Fix is submitted. > 2018-05-22 21:48 UTC Fix starts to be rolled out. > 2018-05-22 22:16 UTC Fix fully deployed and tested on test instances > followed by deployment to production. Access to frontends restored. > 2018-05-24 Customer communication sent to affected users to ask them to > renew their certificates and revoke the old ones. > 2018-06-19 The final handful of certificates that had not already been > revoked and replaced by users were revoked by the CA. > > Findings: > > * The operational plan to halt issuance worked as expected and was > implemented quickly. > * The problem was quickly found, fully understood and easy to remedy. > * Tests existed, but did not cover this failure case. > > Remediation Plan > * Completed > ** Message of the Day (MOTD) functionality was added or improved for all > issuance systems to make it easier to communicate issues to users when > issuance is intentionally paused. > ** Test coverage was expanded to ensure that both the quantity and type of > SCTs are checked. > _______________________________________________ > dev-security-policy mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-security-policy > _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
