On Thu, Nov 12, 2020 at 7:27 PM Kathleen Wilson via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote:
> I am very much in favor of increasing transparency about the > qualifications of the auditors providing audit statements for CAs in our > program. However, I think that we need to spend more time figuring out a > few things before adding such a requirement to our policy. Therefore, I > think we should add this to our list of things to spend some focused > time to figure out in early 2021, and move this item to the next version > of Mozilla’s root store policy. > I think that's a reasonable place, but I do worry that there's a risk that either progress won't be made until then, and we'll be in the similar situation of needing more time. Given that this relates to audits and audit statements, does it make sense to actually go forward with trying to find a narrowly banded approach, with a forward dated requirement, such as 12 months after 2.7.1? The benefit to this approach is that it allows the expectation to be factored into contractual agreements with auditors, which may be done several years in advance, as well as provides the necessary signals to audit firms, that this is a thing of real and concrete value that Mozilla is committed to. It also allows Mozilla to assess progress, throughout the year, on the full set of requirements. If we think about other changes that have required some degree of details to be worked out (e.g. the SHA-1 deprecation, RSA-1024 deprecation, certificate lifetimes), the benefit of setting a clear expectation ahead of time helps move the conversation from debating "if" to discussing "how", which can result in more productive engagement. Equally, I think there's a lot that can be borrowed from how approaches to transparency have been done in other regards. For example, with the introduction of CAA, there was "merely" a requirement to disclose, which later turned into more concrete criteria and objectives. Similarly, with respect to organizational validations, an approach being taken right now for the EV Guidelines is to disclose, with the intent of using such disclosures to better inform, analyze, and develop concrete requirements, based on those collective disclosures and interpretations. In this regard, the principles from Mozilla's 1.0 Certificate Policy provide a small minimum, along with some of the language from, say, the FPKI, regarding technical competencies. The basis here is simply for the auditor to *disclose* why they believe they meet the criteria or objectives set. This avoids the need to address part of your questions (e.g. "How do auditors apply to be considered qualified auditors"), because it leaves the current policies and presumptions in place, but introduces the disclosure requirement for why the auditor is relevant and reliable for the report. This is why I took exception to ETSI's approach, which I worry your questions jump to as well, which is the first step to answering those questions is understanding the existing set of qualifications and how those relate to Mozilla's goals and principles, without seeking to establish a full "qualification regime" out of the gate. By focusing on disclosure, it allows us to evaluate both "is this what we expect/want", as well as better address some of the issues (e.g. "We see auditors with skill X provide much better reports than skill Y; we should consider making X a required skill") As it relates to Ben's proposal, I think the language he's proposed largely works, and we can avoid the set of questions you're worried about, by removing the language "Mozilla will review each individual auditor's credentials and ensure that any Qualified Auditor has the collective set of skills required by Section 8.2 of the Baseline Requirements". Additionally, introducing a forward-dated requirement (e.g. 12 months) helps work through any of the delivery issues Jeff highlighted, in a way that's mutually workable. This ensures transparency, without adding a hurdle here. Issues related to auditors that Mozilla may lose trust in are already covered in Section 2.3 of the policy, with https://github.com/mozilla/pkipolicy/issues/146 existing to provide greater clarity, but I think this can be seen as a contingency. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
