This incident report relates to the 64-bit serial numbers in all certificates
that DarkMatter CAs have issued since their inception. The dialog surrounding
CABF Ballot 164 “Certificate Serial Number Entropy” was unknown to DarkMatter
until shared with us recently by Ryan Sleevi of Google, and demonstrated to DM
that the industry expected at least 64-bits of entropy in resulting
serialNumbers despite the actual wording of the BRs Section 7.1.
1. How your CA first became aware of the problem (e.g. via a problem report
submitted to your Problem Reporting Mechanism, a discussion in
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the
time and date.
A post by Corey Bonnell to the mozilla.dev.security.policy group on February
23, 2019 at 1:28am UAE time. Corey referenced Section 7.1 of the Baseline
Requirements and the specific sentence he believed DM was violating: “Effective
September 30, 2016, CAs SHALL generate non-sequential Certificate serial
numbers greater than zero (0) containing at least 64 bits of output from a
CSPRNG”. Corey also listed 23 certificates (CAs & EE’s) and their serialNumbers
as evidence. Corey followed this up 2 days later with a more comprehensive list
that included DMs pre-certs and final certs published to various CT Logs.
Corey also pointed out a second issue in respect to nameConstrained
intermediates issued by Quo Vadis to DM as also a potential violation, but this
shall be dealt with in a separate incident report, since DM was not the issuer
of these ICAs. Corey’s second post was received on the list at 12:50am UAE time
February 25, 2019.
2. A timeline of the actions your CA took in response. A timeline is a
date-and-time-stamped sequence of all relevant events. This may include events
before the incident was reported, such as when a particular requirement became
applicable, or a document changed, or a bug was introduced, or an audit was
done.
29-Apr-16: DM having previously engaged with QuoVadis for the establishment of
a National PKI utilizing a platform and hardware that would initially be
located in QV DCs, held key ceremonies for Private PKIs on EJBCA platform.
Platform was configured for random 64-bit serialNumbers which at the time was
considered far in excess of the then current BR requirement of 20-bit entropy.
30-Apr-16: QV creates DM intermediates intended for issuance of EV and OV
public trust TLS. These intermediates are instantiated in separate partitions
to the DM Private Trust PKIs, but issuance from them is still subject to the
same platform setting of 64-bit random serialNumbers.
08-Jul-16: CABF Ballot 164 closes and passes. The phrase Corey highlighted
above is added in place of the previous 20-bit entropy statement.
28-Mar-17: Transition of DM private PKIs and platform to DM DC’s is complete
under supervision of EY as WebTrust Auditors. Public Trust CAs continue to
operate out of QV DCs.
08-Jun-17: DM Point In Time Audit is complete. NOTE: DM is now responsible for
platform configuration in compliance with BRs. DM considers current
serialNumber setting of platform (64-bit SNs) as compliant with BRs Section 7.1
27-Oct-17: DM successfully completes WebTrust Period In Time.
04-Nov-17: QV & DM complete transfer of DM public trust intermediates from QV
to DM under EY & KMPG audit observation
18-Dec-17: DM & UAE Roots submitted to Mozilla
02-Oct-18: DM successfully completes 2nd WebTrust Audit
06-Nov-18: DM submits new WebTrust Audits to Mozilla
23-Feb-19: 1:28am: Corey posts to MDSP claiming DM has mis-issuance of certs
due to BRs Section 7.1
23-Feb-19: 3:36am: Post is observed by DM, internal investigation requested
immediately.
23-Feb-19: 6:14pm: Internal investigation confirms 64-bit setting for all
certificates, certlint/cablint/zlint do not complain about certs submitted with
this setting. Confirm that settings appear to be compliant with BRs. Further
validation sought from platform provider, support ticket opened.
25-Feb-19: 12:50am: Second email from Corey with expanded list of pre-certs and
issued certs is posted to MDSP
25-Feb-19: 5:08am: Response posted to Corey on MDSP regarding DM’s
investigation and the conclusion that DM is in compliance with BRs Section 7.1.
NOTE: over next two days several folks weigh in on the 64-bit entropy vs 64-bit
CSPRNG output wording of BRs and what that means.
26-Feb-19: 7:41pm: Wayne Thayer posts to MSDP that 64-bit serialNumber with
63-bit entropy are mis-issued under the BRs.
27-Feb-19:10:55am: DM responds to Wayne reiterating our position of compliance
since BRs only say 64-bit output from a CSPRNG and do not say anything about
entropy. However, DM commits to make a move to 128-bit serialNumbers from a
CSPRNG in next change control to seek to alleviate concerns. More discussion
continues on the list.
27-Feb-19: 11:55am: Ryan Sleevi posts historical data/discussions links to MDSP
regarding the change resulting from Ballet 164 that detailed where subsequent
discussion was had regarding some CAs that included serials of exactly 64 bits,
and how those were considered that they could not be compliant, and they made
the necessary changes.
27-Feb-19: 5:00pm: DM, following a review of data/links provided by Ryan, stops
issuance of public trust certificates. Plans are begun for an immediate
migration to 128-bit serial numbers with far exceeding the required entropy. DM
announces decision on MDSP at 5:09pm
28-Feb-19: 500pm: DM action is past 24 hours: Testing of manual upgrade of
platform without waiting for patch from vendor, roadmap of patch application so
that other profiles i.e. private trust (not TLS public trust), can later be
downgraded back to 64-bit – issuance is stopped for those profiles until patch
can be applied. Change scripts created, tested and QA on pre-production,
validated. Production change control scheduled and executed successfully. TLS
issuance restored in production with confirmed 128-bit serialNumber from
CSPRNG. Full list of still active certificates with 64-bit serialNumber is
generated. Notices to respective customers are sent, and where possible phone
calls made. Revocation and re-issuance of affected certificates begins for
customers who are responsive. NOTE: Thursday is last day of work week in UAE,
Friday & Saturday are week-end, most businesses open again on Sunday.
28-Feb-19: 9:30pm: Progress to date: 157 total certificates identified as still
active that will be revoked during this exercise. Currently 45 have been
re-issued. 18 revoked.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates
with the problem. A statement that you have will be considered a pledge to the
community; a statement that you have not requires an explanation.
Issuance was stopped at 5:00pm 27 Feb 2019. Platform is now upgraded to 128-bit
serialNumbers, and issuance now with larger bit sized serialNumbers is active.
4. A summary of the problematic certificates. For each problem: number of
certs, and the date the first and last certs with that problem were issued.
all TLS certificates up until 5:00pm yesterday (27 Feb 2019) were issued with
64-bit serialNumber from 29 April 2016 until 27- Feb 2019. 157 TLS certificates
still active as of yesterday. 45 have now been reissued, 18 revoked.
5. The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged to CT
and then list the fingerprints or crt.sh IDs, either in the report or as an
attached spreadsheet, with one list per distinct problem.
This post will be updated with this info shortly, but please reference Corey
Bonnel’s earlier post of the same
6. Explanation about how and why the mistakes were made or bugs introduced, and
how they avoided detection until now.
CA platform was originally provisioned with 64-bit serialNumber during period
this was explicitly allowed under BRs. When DM migrated private trust platform
and began preparing for public trust issuance through WebTrust Audit
preparations, this was after Ballot 164 had updated the BRs. DM was not privy
to the discussions around Ballet 164 and the subsequent community discussions.
DM analyzed the BRs post Ballot 164 and concluded based on the definition in
Section 7.1, that the platform was complaint with the BRs. This continued to be
our position until Ryan Sleevi posted historical data regarding the
interpretations provided to other CAs after the adoption of Ballet 164. At this
point DM realized that despite what the BRs said, the general community
interpretation of Section 7.1 was different, and it had been consistently
applied to all other CAs in the community. DM is not seeking an exception, and
only wishes to be held to the same standard as all other CAs in the community,
therefore it became necessary to make an immediate change. It was not bug
originally, it was complaint at the time it was introduced. It avoided
detection until now because the update to the BRs appears to have lost some
specificity in it’s re-wording which existing CAs knew about at the time, but
any new CAs coming after the change may not make the same conclusion without
the benefit of the background information.
7. List of steps your CA is taking to resolve the situation and ensure such
issuance will not be repeated in the future, accompanied with a timeline of
when your CA expects to accomplish these things.
Change is already active – CA now using 128-bit serialNumbers for any new
certificates. There are 157 active TLS certificates at the time that were
impacted and will be revoked. Expected completion for revocations is next 24 to
48 hours.
Regards,
--
Scott Rea
Scott Rea
Senior Vice President - Trust Services
[cid:[email protected]]<http://www.darkmatter.ae>
Level 15, Aldar HQ
Abu Dhabi, United Arab Emirates
T +971 2 417 1417<tel:+971%202%20417%201417>
M +971 52 847 5093<tel:+971%2052%20847%205093>
E [email protected]<mailto:[email protected]>
darkmatter.ae<http://darkmatter.ae>
[Linkedin]<https://www.linkedin.com/company/dark-matter-llc> [Twitter]
<https://twitter.com/GuardedbyGenius>
[Year of Zayed] [expo]
The information in this email is intended only for the person(s) or entity to
whom it is addressed and may contain confidential or privileged information. If
you receive this email by error, please notify us immediately, delete the
original message and do not disclose the contents to any other person, use or
store or copy the information in any medium and for whatever purpose. Any
unauthorized use is strictly prohibited.
_______________________________________________
dev-security-policy mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-security-policy
