We use "sharding" and "partitioning" as synonyms. Our issuing intermediates (e.g. E5 and R10) each issue 128 different CRLs. Each subscriber certificate is uniquely mapped to one of those 128 shards/partitions. We've recently been changing how that mapping works (from a computation based on the cert's notAfter date to a computation based on the cert's serial number), and this bug was introduced as part of that changeover. The key phrase in that technical paragraph is "explicitly sharded", which is how we internally refer to certificates using the new serial-based mapping scheme.
Aaron On Fri, Apr 18, 2025 at 8:43 AM Jeremy Rowley <[email protected]> wrote: > This is super interesting. What does it mean when OCSP and CRL entries > don't match? You're right that this isn't a compliance issue that I can > tell (and 6960 only says they should match). > > Can I ask a tangential question? How are you defining sharted certs? I'm > familiar with sharded CT logs but what is a CRL shard? Are you just using > standard CRL partitioning or is it something more? > > On Thu, Apr 17, 2025 at 1:00 PM 'Kiel Christofferson' via CCADB Public < > [email protected]> wrote: > >> Let's Encrypt likes to learn about the ecosystem from the experiences of >> the community. Today we want to bring some information about an event that >> may be interesting or useful to other CAs in precisely that way. >> >> One of our engineers discovered a bug in Boulder >> <https://github.com/letsencrypt/boulder/pull/8096> (our Certificate >> Authority software) which caused revoked certificates included in one of >> our CRL shards to retain their original revocation reason code despite a >> subsequent ACME revoke-certificate request which "upgraded" the reason code >> to keyCompromise. >> >> In succinct technical detail, our engineer described the bug in this way: >> >>> For explicitly sharded certificates, CRL status is read from the >>> revokedCertificates table. This table gets written at revocation time. At >>> re-revocation time (for key compromise), it only gets written by the SA if >>> the caller passes a nonzero ShardIdx to UpdateRevokedCertificate. The RA >>> was never passing a nonzero ShardIdx to UpdateRevokedCertificate. >> >> >> Note: “SA” and “RA” here are referencing our Boulder components, and “RA” >> is not a typical “Registration Authority”. >> >> This prompted an investigation covering two primary points: >> 1) What language in the Baseline Requirements and Root Programs covers >> the "upgrade" of reason codes in CRLs, or the expectations around >> re-revocation? >> 2) Since this bug did not affect our OCSP infrastructure, what are the >> implications of CRLs and OCSP responses containing different reason codes >> for a single revoked certificate? >> >> For the first point, we found what we believe to be clear language in >> Mozilla's Root Store Policy and the BRs: >> - First paragraph of Mozilla's Root Store Policy 6.1.1 >> <https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#611-end-entity-tls-certificate-crlrevocation-reasons> >> >>> When an end entity TLS certificate (i.e. a certificate capable of being >>> used for TLS-enabled servers) is revoked for one of the reasons below, the >>> specified CRLReason MUST be included in the reasonCode extension of the CRL >>> entry corresponding to the end entity TLS certificate, as described in >>> sections 4.9.1 and 7.2.2 of the TLS BRs. >> >> >> - Final paragraph of Baseline Requirements 7.2.2 >> <https://cabforum.org/working-groups/server/baseline-requirements/documents/CA-Browser-Forum-TLS-BR-2.1.4.pdf#51> >> >>> When a CA obtains verifiable evidence of Key Compromise for a >>> Certificate whose CRL entry does not contain a reasonCode extension or has >>> a reasonCode extension with a non‐keyCompromise reason, the CA SHOULD >>> update the CRL entry to enter keyCompromise as the CRLReason in the >>> reasonCode extension. >> >> >> For the second point, language specifically discussing the possibility of >> a mismatch between CRL and OCSP revocation reasons was harder to find. What >> we did find seemed to represent a design of eventual consistency, which >> makes operational sense given the very different nextUpdate requirements >> for CRLs vs OCSP, and possible variability for software that is publishing >> CRLs vs software responding to OCSP requests. >> >> Even if "SHOULD update the CRL entry" appears to leave room for this sort >> of thing, it did not match our expectations on our system. We fixed the >> bug, deployed the new version, and we tracked-down three certificates in >> our database which would have shown as Revoked in both CRLs and OCSP, but >> which would have shown reason code "keyCompromise" in OCSP while retaining >> their original reason code in CRLs (all three happened to be >> "unspecified"). Re-revocation is pretty rare, and re-revocation with the >> new reason being keyCompromise is even more rare. We started using the >> previously-mentioned revokedCertificates table in early March, and only >> accumulated these three. We updated the affected rows, and all is >> consistent. >> >> Thank you for reading, we welcome questions and discussion. >> -- >> Kiel C >> SRE at Let's Encrypt >> >> -- >> You received this message because you are subscribed to the Google Groups >> "CCADB Public" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion visit >> https://groups.google.com/a/ccadb.org/d/msgid/public/CABK0L0038%2Bo235fz7oM2YkQDNhdBVdF8DQmKsQ%2B--bq%3DHbtC-Q%40mail.gmail.com >> <https://groups.google.com/a/ccadb.org/d/msgid/public/CABK0L0038%2Bo235fz7oM2YkQDNhdBVdF8DQmKsQ%2B--bq%3DHbtC-Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "CCADB Public" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/a/ccadb.org/d/msgid/public/CAFK%3DoS9skMUBynCUjgfRFO9Y4w8uzQ_bLbvhMrZzxDdVzW%3DO%3DQ%40mail.gmail.com > <https://groups.google.com/a/ccadb.org/d/msgid/public/CAFK%3DoS9skMUBynCUjgfRFO9Y4w8uzQ_bLbvhMrZzxDdVzW%3DO%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "CCADB Public" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/ccadb.org/d/msgid/public/CAEmnErc2Mb1jojciUHStX7uZbiF6yKb2aU4%2BuYO%2BkSbJOTARyQ%40mail.gmail.com.
