On Monday, October 30, 2017 at 2:19:31 PM UTC-7, Ryan Sleevi wrote: > > If we put the layer cake together and simplify: > * ISO/IEC 17065 - Common requirements for conformity assessment bodies > looking at products/services (e.g. "What should all auditors do") > * ISO/IEC 17021 - Common requirements for conformity assessment bodies > looking at management systems > * ISO/IEC 27006 - Common requirements for conformity assessment bodies > looking at information security management systems > * EN 319 403 - Common requirements for conformity assessment bodies > evaluating TSPs (e.g. "What makes an auditor qualified to be a CA auditor") > * EN 319 411-1 - Common requirements on the TSP for websites (combined with > EN 319 401, which 319 411-1 incorporates-and-builds-on) > > In trying to understand why the reports are what they are, we need to look > in particular at 17021 and 17065, and the framework they use for both audit > engagements and reporting. 17065 describes a certification scheme. > Reproducing a paragraph from the introduction: > > """ > Certification of products, processes or services is a means of providing > assurance that they comply with > specified requirements in standards and other normative documents. Some > product, process or service > certification schemes may include initial testing or inspection and > assessment of its suppliers' quality > management systems, followed by surveillance that takes into account the > quality management system and > the testing or inspection of samples from the production and the open > market. Other schemes rely on initial > testing and surveillance testing, while still others comprise type testing > only. > """ > > 319 411-1 certification describes a system that is based on an initial > testing or inspection, along with periodic surveillance.
Thanks for the explanation. It's very helpful. Up to this point, it makes sense to me. > Importantly, > within 17065 and 17021, the way of ensuring continued compliance is done > based on contracts and reporting - that is, the client is responsible for > reporting changes to their conformity assessment body, and the CAB may > determine to revoke the certification or indicate corrective actions should > be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E) 8.6.3). > ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the > name of the CAB, the date the certification is granted, the name and > address of the client, the scope of the certification, the validity > period/expiration date of the certificate, and anything else required by > the certification scheme (319 411-1). If I am understanding correctly... an ETSI auditor performs a point-in-time audit and then relies on the CA to notify the auditor of material changes. Wish I could say that I believe that would work. Unfortunately, based on my experience with CAs I do not believe the CAs will be pro-active in this way. And I think the only way to really know what a CA is doing is to look at their data -- look at the actual certs they are issuing, and their documentation regarding such certs/issuance. > > Within the WebTrust scheme, the reports are based on either US (AICPA) > Standards of AT101, Canadian (CPA Canada) Standards of Section 5025, or > IFAC's ISAE3000 standards. What is important and notable about these is > that they are reports over information, with a scope and norm (e.g. > WebTrust for CAs) applied. This is why there's a consistent period of time > - information is collected and the auditor evaluates, on the basis of that > information, the level of assurance that is being met. That makes sense to me. > > I'm still working to have a better understanding here (and this is all in a > personal capacity), but my conclusion is this: > - "ETSI" audits reflect an engagement at a particular point in time, where > a series of system controls are evaluated, and the result is a certificate > indicating the process and products comply with the relevant criteria (319 > 411-1). This is similar to a "Point in Time Readiness Assessment". After > the certification is granted, the TSP is responsible for notifying the CAB > of changes (in the future), and the CAB evaluates whether those changes > should cause it to revoke certification or institute some corrective action > (which may not be noted in the certification). An ETSI audit is thus meant > to indicate a CA "is" or "will be" conforming. > - "WebTrust" audits reflect an engagement looking over the past, by the CA > providing documentation and evidence that their policies were consistent > with the requirements. That is, a WebTrust audit is meant to assess whether > or not a CA is being accurate when it says it "was" conforming, but makes > no guarantees about whether or not a CA "is" conforming, nor whether or not > the CA disclosed all relevant information. This is a very important distinction. Thank you for clarifying. > The challenge is in determining whether this is a correct understanding of > the respective differences, How do we verify that this is correct? > and then determining whether or not the > certification-based approach provides the desired or sufficient level of > assurance - in particular, its 'forward-looking' approach with > self-reporting, rather than the WebTrust-style assurance engagement that is > more 'backward-looking' without future promises. Based on what I've seen with CAs over the past several years, I do not believe the 'forward-looking' approach with self-reporting is sufficient. I think we have to (and we do!) require the backward-looking approach. CAs can make all the "future promises" they want, but the proof is in the resulting data -- the certs that they issue. Based on the above information, I do not think the ETSI audits meet the requirements of Mozilla's Root Store Policy or the CA/Browser Forum Baseline Requirements. Maybe that's the real problem here. Am I missing something? Kathleen _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
