Correct, we do not believe there was a policy violation, we're proactively sharing in the interest of transparency and knowledge sharing.
I believe there is additional information we could share about how we've modified testing to ensure compliance with Chrome and Safari's SCT inclusion rules and have more flexible tests. I want to discuss this with the engineer who implemented the changes to ensure they agree with how I would summarize the changes. Update to follow. On Thu, Aug 23, 2018 at 8:57 AM Alex Gaynor <[email protected]> wrote: > Hi Andy, > > Just so I follow, this is something you're proactively sharing, right? As > far as I can tell, there's no violation of any Mozilla Root Program rules > here, just an issue that caused interstitials in Chrome. > > Either way, I appreciate your sharing. > > You mentioned the issue was do to some overly complex control flow. In > order to help other CAs out, do you think there are testing methodologies > that could have helped catch this earlier? > > Alex > > On Thu, Aug 23, 2018 at 8:50 AM Andy Warner via dev-security-policy < > [email protected]> wrote: > >> Please note, Google wrote this report for internal use immediately after >> the issue. We intended to post it to m.d.s.p at that time, but securing >> internal approvals took a while and the posting ended-up on the back burner >> for a bit. It was a minor issue, but we want the community to be aware of >> it. >> >> Summary: >> >> May 21st 2018, a new tool for issuing certificates within Google was made >> available to internal customers. Within hours we started to receive reports >> that Chrome Canary (v67) with Certificate Transparency checks enabled was >> showing warnings. A coding error led to the new tool providing Signed >> Certificate Timestamps (SCTs) from 2 Google CT logs instead of one Google >> and one non-Google log. >> >> * NOTE: Affected certs were logged at issuance to at least 2 Google CT >> logs and 2 non-Google CT logs. The embedded SCTs for affected certs only >> provided proofs from Google logs instead of Google and non-Google logs as >> required by Chrome. >> >> * NOTE: The bug was due to an 'if/else' chain fall through. The code in >> question has been refactored to be simpler and more readable. >> >> The issue was fully resolved ~14 hours after initial notification. The >> issue was mitigated within 4 hours. Triage and code fixes happened within >> 11 hours and it took ~3 hours to deploy the fixed code and confirm the >> fixed behavior in production. The new code was running in relatively few >> locations, so deployment was quick compared to some changes in our >> infrastructure. >> >> Most affected customers responded quickly to communications that they >> should replace their certificates and revoke the old ones before a given >> deadline. All certificates that were issued with an SCT set that was not >> fully compliant were revoked on 2018-06-19 if they had not already been >> revoked by the customer previously. Most users replaced certificates >> shortly after notification. >> >> Timeline: >> >> 2018-03-22 Bug introduced to codebase. >> 2018-05-21 Push including bug became available to clients. >> 2018-05-22 08:05 UTC First user reports that Chrome Canary presents a CT >> warning for a certificate. >> 2018-05-22 09:25 UTC Bug filed with initial assessment. >> 2018-05-22 12:01 UTC Frontend jobs with the bug are taken offline >> following standard CA procedures. >> 2018-05-22 15:59 UTC Issue conclusively identified. >> 2018-05-22 19:07 UTC Fix is submitted. >> 2018-05-22 21:48 UTC Fix starts to be rolled out. >> 2018-05-22 22:16 UTC Fix fully deployed and tested on test instances >> followed by deployment to production. Access to frontends restored. >> 2018-05-24 Customer communication sent to affected users to ask them to >> renew their certificates and revoke the old ones. >> 2018-06-19 The final handful of certificates that had not already been >> revoked and replaced by users were revoked by the CA. >> >> Findings: >> >> * The operational plan to halt issuance worked as expected and was >> implemented quickly. >> * The problem was quickly found, fully understood and easy to remedy. >> * Tests existed, but did not cover this failure case. >> >> Remediation Plan >> * Completed >> ** Message of the Day (MOTD) functionality was added or improved for all >> issuance systems to make it easier to communicate issues to users when >> issuance is intentionally paused. >> ** Test coverage was expanded to ensure that both the quantity and type >> of SCTs are checked. >> _______________________________________________ >> dev-security-policy mailing list >> [email protected] >> https://lists.mozilla.org/listinfo/dev-security-policy >> >
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
