On Tue, Oct 31, 2017 at 8:34 AM, Dimitris Zacharopoulos via dev-security-policy <dev-security-policy@lists.mozilla.org> wrote: > > Do you believe that the requirements stated in the policy are unclear? That >> is, as Kathleen mentioned, the Mozilla policy states all the information >> that must be present, as a template of what needs to be there. Perhaps >> this >> is just confusion as to expecting, say, Mozilla to provide a PDF of a >> cover >> sheet? >> > > I do not believe the requirements are unclear which is why we have seen > this information included properly in many ETSI audit reports. > If Mozilla finds this problem repeating for some ETSI reports, perhaps a > guidance on the expected audit template would be a good place to start. > Webtrust had different-looking reports in the past until the Webtrust > Committee issued templates as guidance for practitioners.
Alternatively, might it also suggest that those CABs and NABs are not aligned with the (long-established) community norms, and as such, is indicative of both quality and comprehensiveness (or lack thereof)? > I think you are looking at this from the opposite side. Auditors have > their own scheme to follow which is governed under ISO 17065 and ETSI EN > 319 403. These schemes provide guidance on how to conduct an effective > audit for ETSI EN 319 401, 411-1, 411-2, 421 and so on. In addition to > these, there are National Accreditation Body schemes for specific audits > which provide additional guidance. When the audit covers operations for one > year (mandated by the Baseline Requirements and which finds it's way in the > contract between the CA and the CAB), the sampling must include evidence > from the entire year. > I don't believe your statement is supported by the evidence - which is why I'm pushing you to provide precise references. Consider from the perspective as a consumer of such audits - there is zero awareness of the contract as to whether or not the BRs were in scope - after all, 319 411-1 is meant to be inclusive of the normative requirements with respect to audit supervision. But stepping back further from the contract, the claim that "the audit covers operations for one year" is also not part of the 17065, 17021, or 319 403 oversight. That is, the certification is forward looking (as evidenced by the expiration), and while it involves historic review, it is not, in and of itself, a statement of assurance of the historic activities. This is the core difference between the 17021/17065 evaluation of processes and products versus, say, the ISAE3000 assurance evaluation. > The eIDAS Regulation mandates for 2-year audits (not the ETSI EN 319 > 411-1). This has been reflected in the ETSI EN 319 403 audit scheme, under > 7.4.6 (Audit Frequency), which states: > > "There shall be a period of no greater than two years for a full > (re-)assessment audit unless otherwise required by the applicable > legislation or commercial scheme applying the present document. > > NOTE: A surveillance audit can be required by an entitled party at any > time or by the conformity assessment > body as defined by the surveillance programme according to clause 7.9." > I'm patently aware of that, but I'm trying to highlight to you that this statement itself lacks the specificity to give confidence of the operations. We've seen CAs try to present surveillance audits as full audits. We've seen CAs try to present point in time assessments as periods of time. > Also, as we discussed at F2F 38 in Bilbao and is covered in the minutes < > https://cabforum.org/2016/05/25/2016-05/#ETSI-and-eIDAS-Update>, even > though the general guidance for ETSI EN 319 403 is full re-assessment audit > every two years, it is up to the CA/TSP to request that a full audit is > conducted yearly and that this information is declared in the audit report. > If you see ETSI audit reports that don't specifically state that a > full-audit took place, then this information probably wasn't clear to the > specific TSP. This is not related to the audit scheme. > Respectfully, I disagree. If the audit scheme routinely encourages audits that are insufficient, then one MUST look at the root cause for that. It would appear your argument - and apologies if I misrepresent, but I rephrase it here in the hopes of highlighting our disagreement - is that "319 401, 319 401, and 319 411-1 provide frameworks for audits, but it's up the the CAB, NAB, and TSP to establish a procedure that will suitably satisfy the user agents". My argument is that, especially when combined with the fact that Regulation No 910/2014 establishes a process that is unquestionably insufficient, and TSPs and CABs are using that as the framework for their procedures and agreements, this is not sufficient. Or, put differently, your argument seems to be that 319 411-1 "could" be used to satisfy the requirements of the Mozilla Root Program (if supplanted with additional contractual and procedural requirements, documented privately to the TSP as part of Phase 1), while my argument is that 319 411-1 "alone" does not satisfy the requirements (and that, given the confidentiality aspects of the reporting, as required under 319 403 and 17065, cannot) > Is this requirement to opine on the past year of activities described in > WebTrust for CAs? Yes. As noted in the relevant professional standards I cited (with respect to AT101 or ISAE3000), and the underlying professional standards (for which we discussed during "Period of Time" definition), the very foundation of the reporting is based on this degree of analysis. > If an auditor allowed for this scenario, then this audit is not > "effective" and the auditor should not be trusted. With a very quick search > in ETSI EN 319 403, Section 7.4.3.2 describes the Requirements of Sample > Based Approach. If a CAB does not effectively sample the audit period and > capture evidence only for the last 3 months, they jeopardize their > accreditation. Again, this is not related to the audit scheme. > I appreciate you highlighting the text, but this again highlights the deficiency. You've highlighted the text with respect to sampling, but as you can see, it does not note the period of sampling. More precisely, you're referencing "audit period", but that itself is missing from the requirements. Further, I don't think you can argue 7.4.3.2 is relevant with respect to the historic process review - it's instead normative for multi-site sampling. This is precisely because it's designed around a process/site overview, not a historic overview - as evidenced throughout Section 7 (and further emphasized in 7.4.5.2) It would appear, and again, apologies if I'm mistating, that your view is that the 'audit period' is negotiated with the CAB as part of the contractual negotiations, and that the documentation review of Phase 1 (which includes the historic analysis) would include a sampling analysis over the period. I don't believe this is actually supported under 17021, 17065, or 319 403. Indeed, the only sampling of records covered within 319 403 is in Section 7.9.1 with respect to surveillance audits > I really hope that some ETSI auditors monitoring this list will be able to > confirm or correct me if my statements are incorrect. In the past years, I > had to go through the documentation of both sides (auditors and CAs) to > better understand the ETSI scheme but I believe the same basic concepts > exist in all audit schemes, including the Webtrust scheme. > I really appreciate you engaging on the list - please don't take my criticisms personally. But I am trying to highlight the specific parts of the texts that describe (insufficient) processes and levels of assurance. Hopefully I've provided sufficient evidence of that, but I can always work to clarify (with a specific section reference) why I'm asserting something specific. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
