Hi Matt, I appreciate your interest in getting to the root causes of this issue, and the polite and professional manner in which you are asking questions. However, I am concerned that this line of questioning seem to have reached the limits of Certigna's analysis capabilities, and is thus unlikely to uncover much more information that helps others in our community to improve. I think it is time that we draw conclusions from the information we have rather than continuing a press for answers that can be misinterpreted as an attack on the CA. If you would like recommend specific actions from Certigna or Mozilla, please do so.
I have included more specific comments inline, and I would welcome a meaningful response from Certigna to any of Matt's questions or my comments. - Wayne On Mon, Oct 1, 2018 at 4:49 AM Matt Palmer via dev-security-policy < [email protected]> wrote: > On Wed, Sep 26, 2018 at 07:36:57AM -0700, josselin.allemandou--- via > dev-security-policy wrote: > > Thank you for your exchanges. We hope that the additions below will > answer your questions. > > It appears that your response has removed most indications of what parts of > your message are my questions, compared to your responses. For clarity, > I've re-added appropriate formatting, but it would be appreciated if you > could use standard e-mail formatting in the future. > > > > Was the action required to manually override the CAA validation failure > > > different from what would be required if the CAA validation had > succeeded? > > > Could an operator have just "clicked the same button they always > clicked", > > > and have the CAA validation failure overridden? Or was there a > separate > > > sequence of UI interactions required to override a CAA validation > failure? > > > > > > This answer does not address the controls in place at the time of the > > > misissuance. Any action which an operator can manually take should be > able > > > to be audited for correctness, but I don't see any mention of that > being a > > > possibility in your response. I take that to mean that there are no > > > functional audit logs, or at least that there are no effective audits > > > (sampling or otherwise) of those logs, for decisions and actions > undertaken > > > by registration agents. > > > > The check on the authorization to be issued for the organization was the > > only one that combined both the verification of the consent form and the > > verification of DNS CAA alerts. The operator could view the alert but > the > > same validation button was used to proceed to the next step, whether the > > DNS CAA is valid or not. > > Thank you for this clear statement of your validation interface design. > Unfortunately, this sounds like a design which is extremely risky, from an > unintentional errors perspective. What form(s) of review for UX, including > but not limited to human factors of safety, were applied to this interface > prior to it being deployed? > > If the implication here is that CAs should apply a high level of UX expertise to their internal validation interfaces, I would wager that very few CAs meet your standards. > Each operation performed by an operator is traced so that it can be > > audited. The periodic audits of registration requests are also intended > > to ensure the conduct of controls by RA and the conformity of their > > results. > > Based on your initial report, I got the impression that the misissuance > we're discussing was not picked up by an ordinary operational audit, but > was > instead identified by some sort of extraordinary review. Is that an > accurate impression? If so, can you provide more detail around the > criteria you use for selecting operations for auditing, and the frequency > of > those audits, with particular reference to how such an unusual event > (overriding a CAA validation failure) wasn't picked up by ordinary > auditing? > > Agreed - the misissuance was caught by an audit that was only performed after Certigna's CAA validation practices were questioned. CAs are required to audit 3% of issued certificates (BR section 8.7), and I would be surprised if there are many that exceed that requirement, so this seems like an industry-wide concern. > > This sounds like there may have been a misunderstanding in the BR > > > description of CAA validation. > > > > > > What specific language in the BRs surrounding CAA validation caused > you to > > > believe that successful CAA validation was optional in the presence of > other > > > forms of consent? > > > > > > What changes do you propose in the language of the CAA validation > > > requirements in the BRs to ensure that no other CA could possibly > > > misunderstand the CAA validation requirements? > > > > We have no improvement to issue regarding BRs. We understood the > > requirement and implemented the DNS CAA control but we made two errors: > > > > - The first is to have assessed that the existing control having the same > > purpose, made it possible to cover the risks and requirements related > to > > the control of the DNS CAA. This decision led to wrongly allow RA > > operators to validate this checkpoint despite DNS CAA alerts. > > > > - The second is to have accumulated these two controls (consent + DNS > CAA) > > in the same validation step because they had the same objective, and > > this without imposing that the DNS CAA negative result be blocking. > > I'm unable to reconcile your two statements. "We understood the > requirements" cannot be true at the same time as you made errors in > understanding the requirements. It's not logically possible for both of > those to be true at the same time. > > That the misunderstand occured is not at issue. The important question > that > needs answering is: how did the misunderstanding occur? Perhaps the BRs > are > insufficiently clear, either on the particulars of the need for CAA > validation, or in the larger sense that the prescriptive controls laid out > in the BRs cannot be substituted for controls which the CA happens to feel > are equivalent. If that is the case, then the BRs need to be clarified in > some way, so that other CAs cannot suffer from the same misunderstanding in > the future. > > Alternately, if the BRs *are*, in fact, sufficiently clear in all respects, > the only other possibility that comes to my mind is that Certigna failed to > correctly interpret the BRs, which is far more concerning -- for Certigna, > at least. It would mean that there could be any number of other, as yet > unidentified, misunderstandings in Certigna's procedures. I would imagine > there would need to be a very comprehensive review of Certigna's processes > and procedures, with a detailed public report of the findings of that > review, for confidence in Certigna to be restored. > > I think we have established that the problem was with Certigna's chosen interpretation of the BRs. I am not clear on how you are proposing to have a "comprehensive review of Certigna's processes and procedures, with a detailed public report of the findings of that review" performed. This sounds like an audit to me? Certigna's audit period ends on 20-October, so their next assessment reports are due relatively soon. > > Have you proposed those changes to the CA/B Forum, to improve the BRs > for > > > all participants? If not, why not? > > > > We would recommend to the other CAs to segment each control even if their > > objective is the same, in order to be sure that the result of each > control > > is taken into account and can not be circumvented under the pretext that > a > > control, having the same purpose, is positive. > > At the risk of re-iterating the previous point, I'm still at a loss to > understand what led Certigna management to believe that substitution of > controls that were deemed equivalent was permitted by the BRs or Mozilla > policy? > > A key point that I hope has been made here is that compensating controls are not a substitute for meeting BR requirements. I don't know what more there is to say on this. > Regarding the control on obtaining the consent of the legal > > representative, it is imposed by a national standard that is gradually > > aligned with the guidelines of the CA\B Forum, but this standard > > unfortunately does not evolve quickly due to administrative constraints. > > > > We did not perceive the interest of suggesting the use of this method > > knowing that it is imposed only on the scale of France and not for other > > international CA and that the control of the DNS CAA had in our opinion > > the same purpose . > > > > We want as much as possible that the different national, European and > > international standards are harmonized in order to avoid the > > implementation of different controls but for the same purposes. > > Fascinating though this is, including this aside does not really improve > the > impression you're giving of Certigna's commitment to adhere to the > requirements of the Mozilla root store policy. That it may be > inconvenient, > or expensive, to comply with the requirements of multiple regulatory > regimes, does not excuse you from doing so. Certigna's choices in > operating > its business are exactly that -- Certigna's choices -- and expense or > inconvenience do not justify failure to adhere to the requirements. > > If there are fundamental incompatibilities between regulatory regimes, the > BRs provide a mechanism for drawing attention to those incompatibilities, > and if Certigna has made use of that mechanism, I would appreciate a > reference to that. > > I agree. I don't think a conflict exists, but if it does, Certigna needs to describe it in detail both here and to the CA/Browser Forum. Otherwise, this line of reasoning is misguided. > > That the BRs have been misunderstood in this fashion is somewhat > concerning. > > > it suggests that there may be other misunderstandings latent in your > systems > > > and processes, which haven't surfaced because they haven't (yet) led > to an > > > identified misissuance or other incident. > > > > > > What steps have you taken, or are you planning on taking, to address > > > possible misunderstandings of the BRs or Mozilla policy that may cause > > > Certigna to unintentionally misissue a certificate or suffer from some > other > > > incident? > > > > > > In what ways have you modified your monitoring, internal audits, and > annual > > > audits by a certification body in light of the fact that the previous > > > editions of those were manifestly insufficient to identify the > > > non-compliance of your CAA-related procedures ? > > > > To address the two errors mentioned above, we have updated the RA > > Validation Portal to separate the DNS CAA control and make it blocking > > automatically. The other validation steps are well separated. > > Can you expand on how changing the RA portal around the DNS CAA control > addresses any *other* potential vectors for misissuance due to Certigna's > misunderstanding of the BRs or Mozilla policy? Given that you've made this > response in regards to my questions surrounding these other possible > misunderstandings, I assume there is a link, but I'm afraid I can't see it. > > It seems more likely that Certigna simply doesn't have an answer to your question. > The certification body in charge of our audit includes in these audit > > criteria the ETSI standards but also the BR and EV guidelines as we > > requested. As said above, new auditors are selected for our next > > certification audit. > > This statement suggests that Certigna believes there was some degree of > blame to be attached to Certigna's auditors regarding this misissuance > (otherwise why raise it as part of this discussion?). If Certigna believes > that their auditors are partially at fault, can you expand on why that is? > Were they simply incompetent in their duties? If so, have you reported > this > incompetence to either Mozilla or the CA/B Forum, so that appropriate steps > can be taken? If you do not believe the auditor to have been incompetent, > what leads you to believe that selecting alternate auditors will > necessarily > lead to a more satisfactory outcome? Are their changes or improvements to > the audit criteria that auditors are required to follow that you can > suggest, to prevent future occurrences of this kind? > > I think that Certigna is just arguing that audits will uncover any other problems that they might have. Frankly, blaming the auditor smacks of scapegoating to me; it's like that > sequence in Monty Python's Quest for the Holy Grail: "the people > responsible > for sacking the people responsible have themselves been sacked". It > doesn't > solve any fundamental problem within the organisation to get new auditors. > > I think we are both in agreement here that Certigna needs to take responsibility for strict adherence to Mozilla's policies, and they need to implement policies and processes that ensure compliance. > > I'm also concerned by your use of the word "blocking" in this context. > You > > > say that "all controls results [...] are now blocking", which I would > > > normally take to mean that there is no way that a registration agent > would > > > be able to influence the issuance of a certificate in contravention of > your > > > policies. However, in your answer to Q23 (which I've included below) > you > > > state that there are a number of validation checks which are performed > > > manually, which (I presume, and please provide a detailed correction > if I am > > > wrong) would allow for a certificate to be misissued as a result of > > > misunderstanding, mistake, or malfeasance on the part of a registration > > > agent. > > > > > > Could you please explain further what exactly you mean by "blocking" > in the > > > context of your above answer? > > > > Controls are automatic such as DCV or DNS CAA and others are not and are > > manually performed by the operators as mentioned above. Whether the > > control is done automatically or manually, if the result of only one of > > these checks is negative, the request is "blocked" and can not be > > validated by a RA operator. No validation and authorization to be issued > > is possible by a RA operator if the result of one of the manual or > > automatic control points is negative. The request is in an "Invalid" > > state while waiting for corrections or proofs to be made to the request. > > Thanks for that clarification. > > - Matt > > _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
