A quick follow-up to close this out.

The push to fully address the issue was completed globally shortly before 16:00 
UTC on 2019-09-02.

After additional review, we're confident the only certificates affected were 
these two:
https://crt.sh/?id=760396354
https://crt.sh/?id=759833603

Google Trust Services considers this matter fully addressed. We will of course 
continue our ongoing internal review program, but no other work or information 
is outstanding at this point.

--
Andy Warner
Google Trust Services

On Friday, August 30, 2019 at 2:39:51 PM UTC-4, Andy Warner wrote:
> This is an initial report and we expect to provide some additional details 
> and the completion timeline after a bit more verification and full deployment 
> of in-flight mitigations. We are posting the most complete information we 
> have currently to comply with Mozilla reporting timelines and will follow-up 
> with additional details soon.
> 
> 1. How your CA first became aware of the problem and the time and date.
> 
> While performing an internal review and assessment of the CRL generation 
> system for Google Trust Services' GTS CA 1O1 on August 16, 2019, it was 
> discovered that the CRL generation service did not include CRL entries of 
> expired certificates. The periodic job only considered certificates with 
> valid lifetimes. This does not conform to RFC 5280 Section 3.3 which states 
> that “An entry MUST NOT be removed from the CRL until it appears on one 
> regularly scheduled CRL issued beyond the revoked certificate's validity 
> period.”  We expect that few, if any, clients have been impacted.  For a 
> client to be impacted they would have to: clock skewed to a time before the 
> not-after field of the certificate; and have a CRL published after expiration 
> dropping the revoked certificate.
> 
> 
> 2. A timeline of the actions your CA took in response. A timeline is a 
> date-and-time-stamped sequence of all relevant events. This may include 
> events before the incident was reported, such as when a particular 
> requirement became applicable, or a document changed, or a bug was 
> introduced, or an audit was done.
> 
> August 16, 2019 15:00 UTC - Reviewer realizes that CRL will not publish for 
> one update past expiration
> August 16, 2019 16:00 UTC - Reviewer checks for other issues & talks to peers 
> to confirm problem
> August 16, 2019 17:00 UTC - Bug is filed to fix the issue with a proposed 
> design fix
> August 16, 2019 23:30 UTC - Fix is sent for review
> August 20, 2019 16:00 UTC - Remediation work is discussed & assigned
> August  20, 2019 18:00 UTC - Query to inspect revoked certificates is created 
> and sent to be run by production team for initial analysis.
> August 21, 2019 10:40 UTC - Production team runs query and returns result
> August 21, 2019 15:00 UTC - Reviewer analyzes data
> August 21, 2019 20:30 UTC - Reviewer asks for a follow up query to ascertain 
> if any certificates did not make it onto the CRL 
> August 22, 2019 07:00 UTC - Initial attempt at updating test systems with fix.
> August 22, 2019 09:00 UTC - Updating of test systems aborted due to 
> (unrelated) issues.
> August 22, 2019 07:00 UTC - Production team runs query for CRLs that may have 
> missed a certificate
> August 22, 2019 15:00 UTC - Reviewer ascertains that certificates under 
> question were on a CRL
> August 26, 2019 11:00 UTC - Second attempt at updating test systems with fix.
> August 26, 2019 13:00 UTC - Test systems updated, confirmed integrity of 
> fixed software.
> August 27, 2019 09:00 UTC - Confirmed fix is effective on test systems.
> August 27, 2019 10:00 UTC - present: Ongoing staged deployment to production 
> systems. Should complete fully by September 3, 2019 17:00 UTC (slightly 
> extended window due to push policies around holiday weekends. The rollout was 
> staged in accordance with Google's standard rollout procedures.)
> 
> 
> 3. Whether your CA has stopped, or has not yet stopped, issuing certificates 
> with the problem. 
> 
> The affected CA software has been patched.  It now populates expired 
> certificates in the CRL for 7 days after their expiration to ensure they 
> appear in at least one regularly issued CRL update.  Automated testing was 
> added as part of the same patch to check that revoked certificates are kept 
> in the CRL.  The patch was developed, tested, reviewed and landed within the 
> codebase by August 19, 2019.  The CRL entry removal bug has been fully 
> remediated.
> 
> 
> 4. A summary of the problematic certificates. For each problem: number of 
> certs, and the date the first and last certs with that problem were issued.
> 
> Investigation began on August 20, 2019 to discover the potential impact of 
> the logic bug. The CRL generation had contained the bug since its inception, 
> affecting all issuance under GTS 1O1 since March 2018. There were 200,263 
> revoked certificates during that time window. Almost all certificates were 
> for internal monitoring specific to checking revocation. The few 
> non-monitoring certificates were all revocations by clients following 
> rotation of certificates and not due to compromises.
> 
> 
> 5. The complete certificate data for the problematic certificates. The 
> recommended way to provide this is to ensure each certificate is logged to CT 
> and then list the fingerprints or crt.sh IDs, either in the report or as an 
> attached spreadsheet, with one list per distinct problem.
> 
> crt.sh IDs to follow, waiting on confirmation that the 2 test certificates 
> mentioned below are the only cases where the issue was surfaced.
> 
> The team looked for revoked certificates from first issuance that never 
> appeared within a published CRL from operation of CA until August 21, 2019.  
> It was detected that 2 test certificates which were revoked within 2 standard 
> CRL update windows; but both were present in at least one CRL before 
> expiration. 
> 
> 
> 6. Explanation about how and why the mistakes were made or bugs introduced, 
> and how they avoided detection until now.
> 
> It is believed that this went unnoticed for so long due to the majority of 
> requirements being located in CA/B BR 4.10.1 & RFC 5280 section 5.  The extra 
> requirements inside RFC 5280 3.3 were most likely overlooked due to the 
> explicit wording of the BR - Revocation entries on a CRL or OCSP Response 
> MUST NOT be removed until after the Expiry Date of the revoked Certificate - 
> during initial development and subsequent reviews.
> 
> 
> 7. List of steps your CA is taking to resolve the situation and ensure such 
> issuance will not be repeated in the future, accompanied with a timeline of 
> when your CA expects to accomplish these things.
> 
> 
> Our existing internal review program has proven effective in discovering 
> issues related to certificates. The program is already in place and will 
> continue.

_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Reply via email to