Part of me wants to commend Entrust for this response. If we can believe its sincerity—and this is a if given their recent history and how this has played out—it took 13 compliance incidents and 107 days for their leadership to recognize, at least publicly, the systemic issues that have happened under their watch, and that does not even count the fact that this has been a problem since at least 2020.
The disappointing thing is that here we are, 30% into the year, and what we have is a commitment to restructure a part of Entrust and fund it to do better without concrete actions to address the specific issues. Meanwhile, they are still trusted and exposing the internet to their continued management challenges. I can’t help but think this response is too little, too late. With that said, it does indicate some level of recognition of how bad things have gotten, which is a step in the right direction. With all that said, it’s difficult to imagine ISRG or Sectigo, for example, showing the same level of disregard for the processes at play or taking this long to get to this point. While this organizational change might help address that, at the same time as of three days ago, it appears that Entrust was still suggesting that EV certs issued in violation of their CPS weren’t actually misissued. This raises questions about whether they have truly internalized the gravity of the situation or if this public gesture is just that—a gesture. Beyond that, the thing that I can’t help but ask myself is how long is too long. I’ve not yet gone back and looked at the average response time for other incidents, but just from looking at this thread and the associated bugs since March 6th, there are still missing responses/updates that were promised, and those that were provided were shallow. In my experience as an engineering leader, my first priority in a situation like this would be to ensure that we never missed a promised or obligated response, the second would be to make sure we had done everything possible to address the identified issues immediately. It’s unfortunate that at this point, we are not even there yet. Entrust has had every opportunity to do the right thing, but even with the world watching, they didn’t seem to prioritize it. As a result, today’s response might be better categorized as performative. Ryan Hurst On Friday, June 21, 2024 at 2:17:30 PM UTC-7 Wayne wrote: > This has been written without checking prior replies - there may be > overlap. > > First off, good work on the new report addressing more matters however > this should have been your original report at a minimum. Before I even > start I will outright state that I hope that Entrust actually improves > throughout this and while this comment will be cleaned up it reflects an > ongoing opinion as the report is read. > > First looking at the letter I will only note this paragraph: > "We are disappointed as this does not represent Entrust values and falls > short of the standards we set for ourselves. We also want to make sure it > is understood that none of these lapses have been malicious or done with > ill-intent to make the internet less secure. As a global CA we must walk a > tightrope in balancing the requirements of the root programs and subscriber > needs, especially for critical infrastructure. In some cases, we did not > strike the right balance." > > It does trouble me that compliance is seen as a balancing point against > issuance for critical infrastructure. There has been a common talking point > of Entrust's delayed revocation incidents of the concept of irresponsible > revocation. I point it to Entrust that such a scenario only presents itself > when a CA is culpable of irresponsible issuance. > > As I read through this I keep seeing a repeating pattern of changing the > organizational structure and creating committees and board of > cross-discipline personnel. While this is all good in theory, I am > concerned that this is not addressing the actual root causes of internal > decision making and that the outputs will be just the same with a different > label on the team providing it. > > Before I delve into any minutiae of the report itself I do find it > noteworthy that in incident #1890898 (Entrust: Failure to revoke OV TLS - > CPS typographical (text placement) error)) > <https://bugzilla.mozilla.org/show_bug.cgi?id=1890898> we have a > functional example of the new cross-functional team evaluating compliance > and coming to a decision. Now this could be a third unspecified team but > given the report I presume this is the template going forward, for brevity > I'll keep this to the broad strokes: > > 2024-04-11: Issue opens, mis-issuance confirmed but no intent to revoke. A > long conversation ensues, nothing changes until a day before the June 7th > report appears. > > 2024-06-06 <https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c28>: > "We reviewed and consulted with independent external experts on this > revised analysis, and based on this broader consultation, we now believe > there was no mis-issuance and thus no need to revoke the affected > certificates. A detailed analysis is below." > > Following that analysis Mozilla and Chrome's Root Programs give a > different opinion. > > 2024-06-18 <https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c42>: > "On this basis, we will treat this as a mis-issuance, and intend to > complete revocation by end of day Saturday, June 22." > > 2024-06-19 <https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c49>: > "On the last question, our position is that there was no mis-issuance—not > that there was a mis-issuance and we decided not to revoke which is the > situation that recommends discussion with affected root stores." > > Now, using the 06-06 opinion as a basis we have an example of this new > cross-functional team. They reviewed the original incident and came to a > conclusion that a) was not the same as Entrust in April, but crucially b) > was not compatible with the viewpoints of the Root Programs who spoke up. I > am of the strong belief of evaluating institutional changes not on their > stated internal changes, but on their outputs. The decisions are all we > will see, they are all that will matter in practice. > > I will not detail line by line but I do notice that some factual > discrepancies in the original report have been addressed. It would be good > to find out how those came to be in the first case. There are still > outstanding ones that I already stated previously. > > >>Note: During our investigation of this issue, we noted that a subset of > 1,975 EV certificates were also issued without the Entrust EV policy > identifier (OID), based on our interpretation of the ballot update. > >This is also a miscount, presumably due to the original figure being 1963 > + 6 certs on a test site that are being double-counted. > > On reading further in 2.1.1 Entrust have outright stated they still stand > by their incorrect analysis as previously noted in this reply. This speaks > volumes as to the decisions that will occur going forward. Within 2.1.3 > there is a mention of Entrust continuing to issue certificates and advocate > their position, but I am seeing no reflection as to the root cause of what > causes them to advocate for their incorrect positions to this day. Not a > single line of 2.1.4 addresses this either. > > Oddly 2.2.3 does not mention that on April 3rd "The issue was escalated to > our verification team for further investigation.". Instead it purports a > subtly different timeline where nothing happened until the 15th. The April > 4th issue as stated in the bugzilla timeline is also absent. > > It is at this point in the report that my original reply must have gotten > lost as I still have outstanding issues. I am quoting my original reply > below: > > >>2.3.4 Improvement Plan > >>... > >>Automate CPR form to collect all required information at the outset from > the reporter rather than relying solely on email > >This goes back to policy issues discussed for years now, see: > >https://github.com/mozilla/pkipolicy/issues/98 > >https://github.com/cabforum/servercert/issues/201 > >https://bugzilla.mozilla.org/show_bug.cgi?id=1650234 > > Now, moving on. In 2.4.1 I am again mis-identified as a reporter of the EV > cert issue. This does not factually matter but is amusing as the initial > factual corrections show that part of my response was read and applied. > > The only significant change I can see in 2.5.1 is the insistence that the > analysis Entrust performed on the mis-issuance not existing on the OV TLS > Typo issue must still be correct. As previously stated, I do not see how > this is compatible with multiple Root Programs stating otherwise. > > I am confused about 2.5.3 though, it is about delayed revocation but the > RCA is focused on the technical issue in the original incidents. 2.5.4 > contradicts itself from paragraph to paragraph. A commitment to revocation > and replacement, and then statements that delays will be managed on a > case-by-case basis. > > It is a bit troubling to see the conclusion state the following: > "The mis-issuances we experienced were technical non-conformities and, had > any one of them happened in isolation, they would not have resulted in us > taking such a hard look at our program and finding the opportunities that > we did." > > Regarding ACME, I previously stated this question and will repeat it now: > Can you make any guarantees that ACME will be a requirement for subscribers > going forward, and that they will not be charged extra for using these > systems? > > Looking into 4.3 Appendix 3: Success Measures I won't address each > individually. I am curious how you intend to get the WebTrust annual audit > results to result in 0 qualifications in the space of a year. I would > suggest an element for Communication is added to address how often a > question has to be restated or followed up on due to a lack of clarity and > transparency. Otherwise the list presents a minimal standard for any > complying CA, if this is not kept by any CA it would be further cause for > concern. > > Once again in evaluating against what was requested I am struck at how the > systemic failures are not being addressed. We have commitments to > committees and boards, but the decisions are what truly matter. There is no > mention of what policies caused these initial issues and how they were not > adhered to. The 2020 commitments are only highlighted due to every comment > noting it specifically, no attempt seems to exist to evaluate against > historical issues. > > On the 2020 commitments I am deeply troubled about this statement in > particular: > "Knowledge of 2020 commitments was similarly confined to a small number of > business unit employees, without broader leadership team/organizational > awareness." > This should have came up in audits which cover incidents on bugzilla. What > happened? Did the auditor only address this with the same small number of > business unit employees and somehow no note of these commitments made it > into any report that went further up the chain? What confidence can we have > in any bugzilla-specific commitments outside of this report going forward? > > As a final note I will highlight this section: > "As part of our response process to the Mozilla community, Entrust > assigned a group of three senior leaders, as well as an external > consultant, to review each incident to validate and expand root cause > analysis." > > Can we please have a breakdown on Entrust's end of what their original > opinion was at the start of each incident, and how these personnel would > evaluate the situation if it were to happen today? I sincerely hope that > #1890898 is not an example going forward. > > The point of incident reports and action items is to ensure things do not > repeat, knowing that the decision-making process is repaired would be one > small step. > > - Wayne > > On Tuesday, June 18, 2024 at 6:35:48 PM UTC+1 Amir Omidi (aaomidi) wrote: > >> I am not going to say with certainty that Entrust is definitely putting >> Chrome over Mozilla. However, I hope they know that most Linux systems out >> there use the Mozilla root store directly. >> On Tuesday, June 18, 2024 at 1:12:19 PM UTC-4 Mike Shaver wrote: >> >>> On Tue, Jun 18, 2024 at 12:49 PM Walt <[email protected]> wrote: >>> >>>> I'd just like to point out that we now have a situation where Entrust >>>> is in the position of seemingly valuing the opinion of other Root Programs >>>> over Mozilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c42 >>>> >>>> In Comment #37, it was hinted at (and made slightly more explicit in >>>> #39) that the opinion of the Mozilla RP is that the attempt to >>>> re-characterize these certs was not going to be looked kindly upon, and >>>> only once a Google RP member explicitly said that it was the Google RP >>>> opinion that the certs remained mis-issued was any movement made on >>>> re-confirming the mis-issuance and taking action to revoke them. >>>> >>>> Also, if we're in a position where Entrust is finally able to commit to >>>> revoking certs within a 5 day period (setting aside that these certs >>>> technically need a delayed revocation bug as the mis-issuance was known as >>>> far back as 2024-04-10), why are other incidents not able to be resolved >>>> in >>>> this amount of time? Is it because Google showed up? >>>> >>> >>> We’ve seen this behaviour in other incidents as well, I believe >>> including the cpsURI one that has turned into a magnet for evidence of poor >>> operation and lack of transparency and responsiveness. I remarked on it in >>> my initial snarky reply to the Entrust Report, in fact. >>> >>> From a realpolitik perspective their behaviour could indeed be rational, >>> especially when the only tool root programs have is distrust. Firefox would >>> suffer substantial market disadvantage if it stopped trusting Entrust >>> certificates when other browsers didn’t. I think people generally >>> underestimate how much Mozilla would be willing to take near-term pain to >>> protect users, but it’s also possible that I am overestimating it. >>> >>> Related to that, I think Chrome’s root program representatives have >>> generally been more willing to take a concrete position quickly, so Mozilla >>> might be waiting for more explanation when Chrome decides that there’s no >>> explanation that could suffice, or similar. The root programs tend to be in >>> agreement more often than not (virtually always with Chrome and Mozilla, I >>> would say, excepting some slightly different root store populations), so it >>> may be somewhat irrelevant whose opinion spurs motion. >>> >>> Realpolitik analysis aside, I do agree that Entrust has created the >>> impression that they care much more about Chrome’s opinion than Mozilla’s, >>> which IMO might not be the best posture to take given that Mozilla and its >>> community are the locus for the processing and evaluation of the incidents >>> in question. >>> >>> Mike >>> >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/8fc828df-d8a5-4526-a8b9-9ad4a43cc96fn%40mozilla.org.
