Apologies for the delayed response; it took longer than I expected to go through the many similar incidents and find the references I wanted, and indeed in the end I omitted many others. Thanks to Ben and the Mozilla community for their patience.
Entrust Report Comments First, I just have to say that given Ben's very explicit expectations in the request for a response, the contents of Entrust's report are shockingly poor. They failed to address many of the requirements, and the entire exercise reads like a rushed homework assignment--not a credible plan by one of the most experienced CAs on the web to restore their operations to the level of quality and compliance expected by the Mozilla Root Program. Shallow Action Item Specifications The Executive Summary claims that the report provides a “detailed overview of concrete, measurable steps”, but the Action Items included in the report are often neither detailed, nor measurable. A single example, from "2.1.4 Improvement Plan": “Establish cross-functional change control board: Complete”. There is no detail as to what this board will decide, how they will be selected, or how their effectiveness can be measured. This Action Item is, as described, basically just "we made a list of some people". Inadequate Response to Delayed Revocation Incidents The Mozilla Root Store Policy itself does not itself admit to any option for delayed revocation. It references the BRs, which require (SHALL and MUST language) revocation within 5 days. Provisions for delayed revocation only come from Mozilla's "Responding To An Incident" page at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation . Mozilla grants generous latitude to CAs in that CAs are permitted to weigh the impact of revocation against the impact of further delaying revocation, but it also makes clear what is expected from a CA when it decides that the circumstances are "exceptional", such as when used in "critical infrastructure". In https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c35 , Ngook Kong clarifies Entrust's position on these expectations: Our interpretation is any delayed revocation will comply with the Mozilla revocation policy at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation Unfortunately, Entrust has systematically failed to comply with the policy to which they refer. Requirement: "The decision and rationale for delaying revocation will be disclosed in the form of a preliminary incident report immediately; preferably before the BR-mandated revocation deadline. The rationale must include detailed and substantiated explanations for why the situation is exceptional. Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable. When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis." Entrust's delayed revocation incidents have consistently failed to meet this expectation. In two of the three delayed revocation incidents that Entrust chose to include in their report, no per-subscriber information whatsoever was provided, with the exception of a comment that listed four out of more than 100 affected subscribers. When rationales are provided ( https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c32), they are insufficiently detailed: they do not contain enough information to determine whether the proposed action items are likely to meaningfully affect the risk of future delayed-revocation incidents. This item has been present, in substantially identical form, since 2019, so it was well known to Entrust when they made their commitments in 2020 to avoid delayed revocation and adhere to the BRs and root program expectations. It is alarming that a CA that boasts of its long experience in the web PKI and involvement in the community is still consistently unable or unwilling to adhere to those requirements. Requirement: "Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable." To the best of my ability to determine, no Root Stores were consulted with regards to the risk analysis or plan of remediation; Ben Wilson's comment at https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c19 seems to indicate that Mozilla's root program representatives were not consulted. There has also not been any indication of auditor involvement. If indeed Entrust worked with their auditor with respect to the decisions not to revoke captured in bugs 1890898 and 1890685, it is difficult to understand how their later analysis came to a different conclusion. Investment in Incident Response Capacity Ngook Kong states in https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c51: "Yes, we are equipped to perform a wide scale revocation if needed and necessary. [...] We have the technical capability to revoke within the required timelines." But note also https://bugzilla.mozilla.org/show_bug.cgi?id=1898848#c1 in which Entrust pleads that they lack the resources to meet their commitment to the BRs: "In addition, the authority to launch and conduct formal investigations, confirm incidents, initiate incident reporting processes, and trigger revocation events is held by a small number of individuals within the compliance team. These same individuals were responsible for helping to respond to incidents, communicating with impacted subscribers, responding to questions from the Bugzilla community, and drafting and submitting incident reports, in addition to other day-to-day responsibilities." The former comment comes after the latter one, so perhaps the resources available have been increased. However, there is no mention in section 2.5 of the report of any action items related to increasing the resources available for investigating or responding appropriately to misissuance events, or how it can be measured that the level of investment is appropriate on an ongoing basis. I don't feel that CAs should generally have to share staffing levels or budgeting processes with root programs, but if those staffing levels are the cause of incidents, then I think they become relevant to evaluation of the CA's ability to operate correctly (and therefore the risk posed by continuing to trust certificates that they issue). Failure to Meet Previous Commitments In 2020, Entrust made the following commitments to Ryan Sleevi (then a peer of the Mozilla root program module): https://bugzilla.mozilla.org/show_bug.cgi?id=1651481#c6 - [1] We will not the make the decision not to revoke. - [2] We will plan to revoke within the 24 hours or 5 days as applicable for the incident. - [3] We will provide notice to our customers of our obligations to revoke and recommend action within 24 hours or 5 days based on the BR requirements. - [4] We will recommend to our customers to implement automation of certificate management. - [5] We will increase our ability for correct implementation and testing to ensure that certificate profiles will meet the latest CA/Browser Forum or root program requirements. - [6] We will monitor the Mozilla incidents and the discussion list to discover problems which other CAs have experienced and how they were resolved. This will allow us to review and react if required to our own implementation. This will also help to minimize the number of miss-issued certificates, which will reduce the risk of late revocation. - [7] We will manage and update our pre-issuance and post-issuance linting to discover or prevent the problem early. (I have added the numbers for easier reference.) These commitments have been repeatedly broken, and in some cases appear repeated again in the report 4 years later: In 2.5.4 (Improvement Plan), we see We intend to revoke and replace certificates that do not meet TLS Baseline Requirements or certificate-specific guidelines.We will plan to do so within the prescribed revocation periods [2] We will work with our subscribers to ensure awareness and minimize delayed revocation requests [3] Driving customer adoption of automation [4] Of specific note, there is an action item "Launch communication and education to subscribers on requirements for public trust certificates" with a target completion of 2024-07-31. Why is that only being undertaken now, four years after Entrust made commitment [3]? In 2.4.4 (Improvement Plan), we have as action items: Run pkilint against all CRLs [7] Update automated test to cover the added requirement [5] which are a near-identical repetition of the indicated commitments, and are indicated in the report as "completed". How can we believe that Entrust has made this improvement, when they committed to doing so already 4 years ago? In the same bug 1651481, Bruce Morton states: "We have put in practices to close out non-conformance and late revocation issues." If they indeed kept commitments [5] and [7], then they should provide substantial detail on why they believed that the previous efforts were appropriate, why it was reasonable for the incidents in 2.4.2 to have occurred with good-faith implementations of [5] and [7], and how their approach is different in 2024 such that the community can have faith that there has been a systematic remedy for this systematic fault. Failure to Address Conflict Between Subscriber Limitations and BR Requirements More than 8000 of the affected certificates in bug 1886532 are indicated to be delayed due to limitations of subscriber process, but none of the action items in the report provide measurable concrete ways to eliminate those limitations. Indeed, I don't think it is reasonable to hold Entrust accountable for making its subscribers improve internal processes, and I am surprised that Entrust has proposed that they be evaluated on efforts to that end. I think it is very reasonable to hold Entrust accountable for ensuring that the BRs are upheld regardless of whether a given subscriber has chosen to invest in changes to help them replace certificates more promptly; that is to say, hold Entrust accountable for revoking, as they have said explicitly that they have the technical capability to do. In https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c1, Paul van Brouwsershaven says: The revocation for customers is in most cases manual and complex, involving multiple internal parties to ensure that the change does not create an adverse impact on web services and back-end processing. As a result, we have moved these customers into delayed revocation, particularly where their operations are critical to the web ecosystem Entrust has stated that in 2020 (and presumably for the intervening four years), they felt that their processes were sufficient to meet the commitments made in 1651481. This is an assertion that they were not aware that the "revocation for customers is in most cases manual and complex", because those Subscriber limitations are still used as an excuse for delayed revocation today. Entrust has made no commitments that would limit the extent to which one of their Subscribers' operational limitations would be allowed to inflict harm on the integrity of the web PKI. Indeed, they continue to knowingly issue web PKI certificates to Subscribers who have regulatory or policy limitations that require as much as 90 days for a certificate replacement to be completed, which is entirely inconsistent with their 2020 commitments, and with a belief that the aforementioned (but unspecified) processes were an appropriate means for meeting those commitments–even over a four year time span. Deceptive and Contradictory Statements Regarding Subscriber Communication In https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c36 , Ngook Kong says No delayed revocation options were offered. Later in that bug, Ngook posts a sample email that was sent to a subscriber regarding a certificate that was misissued, but part of it is redacted: Subject:URGENT ACTION REQUIRED: EV Revocation May be Required Message Group: System Interruptions Message Expiry Date: 3/30/24 We are writing to inform you about a recent issue that affected some of the EV digital certificates issued by Entrust. We apologize for any inconvenience this may have caused you and we are committed to ensuring the security and integrity of your online transactions. Summary: Entrust discovered that some Extended Validation (EV) TLS certificates (EV Multi-Domain SSL, QWAC eIDAS, QWAC PSD2) were missing a specific component required by the EV Guidelines. This component links the certificate to the Certificate Policy (CP) and the Certification Practice Statement (CPS) of the issuer. Entrust has taken steps to address the issue and prevent it from happening again. Any of the specified certificate types issued after 21:40 UTC today, Mar 18, 2024 are not affected. Entrust is required to revoke affected certificates as soon as possible, which will occur on Saturday, March 23, 2024 at 21:00 UTC ACTION REQUIRED: Certificates issued after September 11th, 17:40 UTC, 2023, to March 18th, 21:40 UTC, 2024, will need to be replaced by issuing a new certificate and revoking the old certificate(s) shortly thereafter. We will assist you with this process and ensure a smooth transition. Steps: [REDACTED] Thank you, Entrust Certificates Services Later, it was revealed by a member of the community that what was redacted was an instruction to the Subscriber to revoke along a 30-day timeline, and Entrust was asked why they redacted that portion: https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c40 Perform a REISSUE, and select "Revoke within 30 days" so your production certificate maintains validity, providing sufficient time to perform the replacement Ngook Kong responded The letter we shared is an example of what was sent from us directly to a subscriber and was not posted in the public domain. We were being transparent by sharing the message. The redacted section provides specific instructions to our subscribers on how to revoke and reissue certificates. and uploaded a PDF copy of the email ( https://bug1886532.bmoattachments.org/attachment.cgi?id=9406229). It is very difficult to believe that this section was omitted due to any sensitivity of the material contained; the PDF was provided promptly when it was revealed that the contents were known to the community member, and Entrust's website contains related instructions on renewal for their product already ( https://www.entrust.com/knowledgebase/ssl/how-to-renew-your-entrust-certificate-using-self-service as an example). It is also very difficult to believe that this was anything but an attempt to conceal elements of Entrust's subscriber communication that contradict Entrust's position that they prioritized prompt revocation and conveyed proper urgency to their subscribers. I actually don't think the email is that bad, because their software seems to be such that 30 days is the appropriate window to be selected here, but I think that the fact that they tried to conceal it is very concerning. Similarly concerning is their repeated uncooperative responses to requests for per-subscriber detail and rationale, which is a crystal-clear requirement of the Mozilla incident response policy for delayed revocation. Conclusion In conclusion, I do not feel that Entrust has demonstrated satisfactory commitment or investment in addressing systemic issues with Subscriber management and revocation policy/process. Neither their historical behaviour nor their inadequate response to the community are appropriate for a CA that chooses to issue public web PKI certificates. I strongly recommend that Entrust-issued certificates issued after June 7, 2024, not be trusted by Mozilla products. Mike On Fri, 7 Jun 2024 at 15:53, 'Bruce Morton' via [email protected] <[email protected]> wrote: > Please respond to comments you may have on our report or action items > here. We will track our progress against the action items list in Bugzilla > under bug 1901270. > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/ebea97dc-1871-4588-93b7-8fd97f50ec0cn%40mozilla.org > <https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/ebea97dc-1871-4588-93b7-8fd97f50ec0cn%40mozilla.org?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CADQzZqsKSw-3ho8%3Deh373K9iJF9-Vq-Ww4agkwk-vLeucZZMpg%40mail.gmail.com.
