This is an initial report and we expect to provide some additional details and 
the completion timeline after a bit more verification and full deployment of 
in-flight mitigations. We are posting the most complete information we have 
currently to comply with Mozilla reporting timelines and will follow-up with 
additional details soon.

1. How your CA first became aware of the problem and the time and date.

While performing an internal review and assessment of the CRL generation system 
for Google Trust Services' GTS CA 1O1 on August 16, 2019, it was discovered 
that the CRL generation service did not include CRL entries of expired 
certificates. The periodic job only considered certificates with valid 
lifetimes. This does not conform to RFC 5280 Section 3.3 which states that “An 
entry MUST NOT be removed from the CRL until it appears on one regularly 
scheduled CRL issued beyond the revoked certificate's validity period.”  We 
expect that few, if any, clients have been impacted.  For a client to be 
impacted they would have to: clock skewed to a time before the not-after field 
of the certificate; and have a CRL published after expiration dropping the 
revoked certificate.

2. A timeline of the actions your CA took in response. A timeline is a 
date-and-time-stamped sequence of all relevant events. This may include events 
before the incident was reported, such as when a particular requirement became 
applicable, or a document changed, or a bug was introduced, or an audit was 

August 16, 2019 15:00 UTC - Reviewer realizes that CRL will not publish for one 
update past expiration
August 16, 2019 16:00 UTC - Reviewer checks for other issues & talks to peers 
to confirm problem
August 16, 2019 17:00 UTC - Bug is filed to fix the issue with a proposed 
design fix
August 16, 2019 23:30 UTC - Fix is sent for review
August 20, 2019 16:00 UTC - Remediation work is discussed & assigned
August  20, 2019 18:00 UTC - Query to inspect revoked certificates is created 
and sent to be run by production team for initial analysis.
August 21, 2019 10:40 UTC - Production team runs query and returns result
August 21, 2019 15:00 UTC - Reviewer analyzes data
August 21, 2019 20:30 UTC - Reviewer asks for a follow up query to ascertain if 
any certificates did not make it onto the CRL 
August 22, 2019 07:00 UTC - Initial attempt at updating test systems with fix.
August 22, 2019 09:00 UTC - Updating of test systems aborted due to (unrelated) 
August 22, 2019 07:00 UTC - Production team runs query for CRLs that may have 
missed a certificate
August 22, 2019 15:00 UTC - Reviewer ascertains that certificates under 
question were on a CRL
August 26, 2019 11:00 UTC - Second attempt at updating test systems with fix.
August 26, 2019 13:00 UTC - Test systems updated, confirmed integrity of fixed 
August 27, 2019 09:00 UTC - Confirmed fix is effective on test systems.
August 27, 2019 10:00 UTC - present: Ongoing staged deployment to production 
systems. Should complete fully by September 3, 2019 17:00 UTC (slightly 
extended window due to push policies around holiday weekends. The rollout was 
staged in accordance with Google's standard rollout procedures.)

3. Whether your CA has stopped, or has not yet stopped, issuing certificates 
with the problem. 

The affected CA software has been patched.  It now populates expired 
certificates in the CRL for 7 days after their expiration to ensure they appear 
in at least one regularly issued CRL update.  Automated testing was added as 
part of the same patch to check that revoked certificates are kept in the CRL.  
The patch was developed, tested, reviewed and landed within the codebase by 
August 19, 2019.  The CRL entry removal bug has been fully remediated.

4. A summary of the problematic certificates. For each problem: number of 
certs, and the date the first and last certs with that problem were issued.

Investigation began on August 20, 2019 to discover the potential impact of the 
logic bug. The CRL generation had contained the bug since its inception, 
affecting all issuance under GTS 1O1 since March 2018. There were 200,263 
revoked certificates during that time window. Almost all certificates were for 
internal monitoring specific to checking revocation. The few non-monitoring 
certificates were all revocations by clients following rotation of certificates 
and not due to compromises.

5. The complete certificate data for the problematic certificates. The 
recommended way to provide this is to ensure each certificate is logged to CT 
and then list the fingerprints or IDs, either in the report or as an 
attached spreadsheet, with one list per distinct problem. IDs to follow, waiting on confirmation that the 2 test certificates 
mentioned below are the only cases where the issue was surfaced.

The team looked for revoked certificates from first issuance that never 
appeared within a published CRL from operation of CA until August 21, 2019.  It 
was detected that 2 test certificates which were revoked within 2 standard CRL 
update windows; but both were present in at least one CRL before expiration. 

6. Explanation about how and why the mistakes were made or bugs introduced, and 
how they avoided detection until now.

It is believed that this went unnoticed for so long due to the majority of 
requirements being located in CA/B BR 4.10.1 & RFC 5280 section 5.  The extra 
requirements inside RFC 5280 3.3 were most likely overlooked due to the 
explicit wording of the BR - Revocation entries on a CRL or OCSP Response MUST 
NOT be removed until after the Expiry Date of the revoked Certificate - during 
initial development and subsequent reviews.

7. List of steps your CA is taking to resolve the situation and ensure such 
issuance will not be repeated in the future, accompanied with a timeline of 
when your CA expects to accomplish these things.

Our existing internal review program has proven effective in discovering issues 
related to certificates. The program is already in place and will continue.
dev-security-policy mailing list

Reply via email to