>From 2020-09-07 at 05:44:35 UTC to 2020-09-08 at 17:48:28 UTC, we
served OCSP responses older than 3.5 days for 268 certificate serial
numbers. From 2020-09-12 at 09:40:31 UTC to 2020-09-13 at 07:22:13
UTC, we served OCSP responses older than 3.5 days for an additional 34
certificate serial numbers. None of the OCSP responses were served
beyond their validity period (nextUpdate). The maximum age an OCSP
response ever reached was 5 days. For OCSP responses with a 7-day
validity period, the Microsoft Root Program specifies that updated
responses be available within 3.5 days and the CA/B Forum Baseline
Requirements specify 4 days.

We were notified of the problem by an alert on elevated error-level
logs. We found that the errors were caused by a recent change to our
RPC system that, in a certain error case, caused a particular column
in our certificate status table to have a value of "0" for a specific
empty field rather than either the expected value or NULL. We
collected serials and last-update timestamp information for affected
entries, and enacted a manual plan for continued remediation of these
entries.

A Boulder CA software release was deployed to production on 2020-09-10
concluding at 17:59 UTC (
https://github.com/letsencrypt/boulder/releases/tag/release-2020-09-09
) ensuring no future erroneous values would be added to the database,
but remediation queries within regular intervals was still required
for existing entries.

On 2020-09-12, the manual plan for recurring remediation steps was not
executed in time causing OCSP responses for the aforementioned
additional 34 certificate serial numbers to age beyond the Microsoft
Root Program and CA/B Forum Baseline Requirements mandates.

On 2020-09-13 at 17:22 UTC, the final manual remediation query was
executed on the database and we verified that all potentially-affected
Certificate Status entries had been remediated.

We have filed the following bug regarding this issue:
https://bugzilla.mozilla.org/show_bug.cgi?id=1666047
-- 
Kiel C
SRE - Let's Encrypt
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Reply via email to