Re: ETSI auditors still not performing full annual audits?

[cornelia.enke66--- via dev-security-policy]

Am Montag, 19. Juni 2017 21:15:09 UTC+2 schrieb Kathleen Wilson:
> I just filed https://bugzilla.mozilla.org/show_bug.cgi?id=1374381 about an 
> audit statement that I received for SwissSign. I have copied the bug 
> description below, because I am concerned that there still may be ETSI 
> auditors (and CAs?) who do not understand the audit requirements, see below.
> 
> ~~~
> SwissSign provided their annual audit statement:
> https://bug1142323.bmoattachments.org/attachment.cgi?id=8853299
> 
> Problems noted in it:
> -- "Agreed-upon procedures engagement" - special words for audits - does not 
> necessarily encompass the full scope
> -- "surveillance certification audits" - does not necessarily mean a full 
> audit (which the BRs require annually)
> -- "point in time audit" -- this means that the auditor's evaluation only 
> covered that point in time (note a period in time)
> -- "only intended for the client" -- Doesn't meet Mozilla's requirement for 
> public-facing audit statement.
> -- "We were not engaged to and did not conduct an examination, the objective 
> of which would be the expression of an opinion on the Application for 
> Extended Validation (EV) Certificate. Accordingly, we do not express such an 
> opinion. Had we performed additional procedures, other matters might have 
> come to our attention that would have been reported to you." -- some of the 
> included root certs are enabled for EV treatment, so need an EV audit as well.
> 
> 
> According to section 8.1 of the CA/Browser Forum's Baseline Requirements: 
> "Certificates that are capable of being used to issue new certificates MUST 
> ... be ... fully audited in line with all remaining requirements from this 
> section. 
> ...
> The period during which the CA issues Certificates SHALL be divided into an 
> unbroken sequence of audit periods. An audit period MUST NOT exceed one year 
> in duration."
> 
> So, a full period-in-time audit is required every year.
> 
> After I voiced concern 
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1142323#c27) the CA provided an 
> updated audit statement to address the concerns I had raised in the bug:
> https://bugzilla.mozilla.org/attachment.cgi?id=8867948
> I do not understand how the audit statement can magically change from 
> point-in-time to a period-in-time.
> ~~~
> 
> I will greatly appreciate thoughtful and constructive input into this 
> discussion about what to do about this SwissSign audit situation, and if this 
> is an indicator that ETSI auditors are still not performing full annual 
> audits that satisfy the CA/Browser Forum's Baseline Requirements.
> 
> Thanks,
> Kathleen

Hello togehter

the Report is the annual Attestation letter. I agree htat the Format was not 
the best, I also agree that it would be worth to have a confirmed Audit 
Statement what is understandable and readable by the relevant responsible 
persons. 

Thanks Conny
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: ETSI auditors still not performing full annual audits?

[Ryan Sleevi via dev-security-policy]

Thanks, Kathleen, for raising these issues.

At a high level, this highlights an interesting concern. If we, as the
broader community, lack the expertise to appropriate review and consume the
audit reports as intended, it may signal a question about whether or not we
should consider consuming ETSI reports. Thus, to ensure ETSI reports
continue to be viable for CAs to provide, it would behove those supporters
and professionals to ensure there is a robust understanding about how to
consume such reports, much as there is similar ongoing discussion (and much
better expertise) towards the consumption of WebTrust reports.

The text you've described seems to align with the process outlined in
https://assets.kpmg.com/content/dam/kpmg/ch/pdf/kpmg-certification-compliance-and-methodology-en.pdf

However, that process notably differs from the expectations of the Mozilla
Root Certificate Policy and expectations.

As you've noted, the letter calls out that this is a main certification
audit (good!), but then notes surveillance audits are planned. While
permissible under the appropriate ETSI guidelines, for purposes of
inclusion in the Mozilla store, the expectation is that a full assessment
audit is performed each year. That is,
http://www.etsi.org/deliver/etsi_en/319400_319499/319403/02.02.02_60/en_319403v020202p.pdf
Section 7.9 permits (limited scope) assessments, but Mozilla does not.

Thus, despite being a main certification audit (which is expected), the
fact that it's called out that other items are planned is concerning.

I think this also aligns with why "agreed upon procedures" is concerning.
An AUP could indicate that the CA has gone above and beyond the set of
controls, or it may indicate that the CA has self-limited the scope of the
engagement to a subset of activities. It may be that the activities are not
performed, as we've seen similarly in WebTrust audits (for example, with
respect to certificate suspension). Unfortunately, this is not clear - and
there's not enough information in the report, as best I can tell, to
distinguish this case.

I am very concerned with respect to the "point in time" audit. There should
not be any ambiguity here - both with respect to Mozilla expectations and
to the general understanding, as previously discussed in the CA/Browser
Forum with the respective auditor communities. To an extent, this aligns
with my understanding of a "main certification audit" - that is, that they
may be inclined to be "point in time" audit, with the surveillance audit
being a review of the CA's adherence to that.

I agree there's some ambiguity with the statement of "Note that the
corresponding certification report was written in German and is only
intended for the client". On one sense, we want to ensure that the report
provided is consistent with and 'binding' of the auditor (conformance
assessment body) and a statement they'll stand by, hence the concern. That
said, the report also asserts KPMG's SCESm registration, so presumably,
this is an official statement. The scope is noted as discussed further at
https://www.seco.admin.ch/sas/PKI , except that URL isn't valid.

Similarly, I agree, it's concerning that it's noted as conforming to the
EVCP profile, but similarly notes "We were not engaged to and did not
conduct an examination, the objective of which would be the expression of
an opinion on the Application for Extended Validation (EV) Certificate."
It's not clear how to reconcile this difference.

Another element of uncertainty is that the policies are listed as
compliance to "DVCP and PTC-BR", except that with respect to EN 319 411-1,
it states that "NOTE: Within the context of the present document PTC is
used synonymously with EVC, DVC and OVC as per
CAB Forum documents. ". So is this indicative that they evaluated DVCP,
OVCP, and EVCP? Or something else? As best I can tell, "PTC-BR" is an
artifact from the predecessor document, TS 102 042, and not relevant in the
context of 411-1.

One other thing that I'm unclear with - it's notable that in audits
performed by other CABs provide certificates of compliance. For example,
TUVIT provides
https://www.tuvit.de/en/services/certification/certification-authorities-according-to-etsi/
. My understanding from the ETSI representatives to the CA/B Forum is that
such certifications are standard - but it's unclear that this represents
such a certification. Perhaps this is only unique to TUVIT?


On Mon, Jun 19, 2017 at 4:57 PM, Kathleen Wilson via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On Monday, June 19, 2017 at 12:21:46 PM UTC-7, Peter Bowen wrote:
> > It seems there is some confusion. The document presented would appear
> > to be a Verified Accountant Letter (as defined in the EV Guidelines)
> > and can used as part of the process to validate a request for an EV
> > certificate.  It is not an audit report and is not something normally
> > submitted to browsers.
>
> Yet, it is the document that was provided to root 

Re: ETSI auditors still not performing full annual audits?

[Kathleen Wilson via dev-security-policy]

On Monday, June 19, 2017 at 12:21:46 PM UTC-7, Peter Bowen wrote:
> It seems there is some confusion. The document presented would appear
> to be a Verified Accountant Letter (as defined in the EV Guidelines)
> and can used as part of the process to validate a request for an EV
> certificate.  It is not an audit report and is not something normally
> submitted to browsers.

Yet, it is the document that was provided to root store operators as the annual 
audit statement. And there has been plenty of time in Bug #1142323 for that to 
have been rectified. 

As reference, here is the audit statement that was provided in 2016:
https://bug343756.bmoattachments.org/attachment.cgi?id=8781268
It says: "KPMG has executed a main certification audit in year 2013, and 
surveillance certification audits in 2014 and 2015..."
"We were engaged to conduct the annual examinations, with the objective of 
which would be the expression of an opinion on the application for Extended 
Validation (EV) Certificates. Accordingly we do express our positive opinion 
and provide you confirmation that the requirements were fulfilled during the 
annual certification audits... "


In the audit statement in question 
(https://bug1142323.bmoattachments.org/attachment.cgi?id=8853299) it says:
"KPMG has executed a main certification audit in year 2017..." So I took that 
to mean that this was intended to be their annual audit statement, and the 
format is very similar to the audit statement from the previous year. But as I 
read through it I noticed phrases like "point in time audit". And then it said:
"We were not engaged to and did not conduct an examination, the objective of 
which would be the expression of an opinion on the Application for Extended 
Validation (EV) Certificate. Accordingly, we do not express such an opinion. 
Had we performed additional procedures, other matters might have come to our 
attention that would have been reported to you." 
This is very different from the statement the previous year.

Thanks,
Kathleen

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: ETSI auditors still not performing full annual audits?

[Peter Bowen via dev-security-policy]

On Mon, Jun 19, 2017 at 12:14 PM, Kathleen Wilson via
dev-security-policy  wrote:
> I just filed https://bugzilla.mozilla.org/show_bug.cgi?id=1374381 about an 
> audit statement that I received for SwissSign. I have copied the bug 
> description below, because I am concerned that there still may be ETSI 
> auditors (and CAs?) who do not understand the audit requirements, see below.
>
> ~~~
> SwissSign provided their annual audit statement:
> https://bug1142323.bmoattachments.org/attachment.cgi?id=8853299
>
> Problems noted in it:
> -- "Agreed-upon procedures engagement" - special words for audits - does not 
> necessarily encompass the full scope
> -- "surveillance certification audits" - does not necessarily mean a full 
> audit (which the BRs require annually)
> -- "point in time audit" -- this means that the auditor's evaluation only 
> covered that point in time (note a period in time)
> -- "only intended for the client" -- Doesn't meet Mozilla's requirement for 
> public-facing audit statement.
> -- "We were not engaged to and did not conduct an examination, the objective 
> of which would be the expression of an opinion on the Application for 
> Extended Validation (EV) Certificate. Accordingly, we do not express such an 
> opinion. Had we performed additional procedures, other matters might have 
> come to our attention that would have been reported to you." -- some of the 
> included root certs are enabled for EV treatment, so need an EV audit as well.
>
>
> According to section 8.1 of the CA/Browser Forum's Baseline Requirements:
> "Certificates that are capable of being used to issue new certificates MUST 
> ... be ... fully audited in line with all remaining requirements from this 
> section.
> ...
> The period during which the CA issues Certificates SHALL be divided into an 
> unbroken sequence of audit periods. An audit period MUST NOT exceed one year 
> in duration."
>
> So, a full period-in-time audit is required every year.
>
> After I voiced concern 
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1142323#c27) the CA provided an 
> updated audit statement to address the concerns I had raised in the bug:
> https://bugzilla.mozilla.org/attachment.cgi?id=8867948
> I do not understand how the audit statement can magically change from 
> point-in-time to a period-in-time.
> ~~~
>
> I will greatly appreciate thoughtful and constructive input into this 
> discussion about what to do about this SwissSign audit situation, and if this 
> is an indicator that ETSI auditors are still not performing full annual 
> audits that satisfy the CA/Browser Forum's Baseline Requirements.

Kathleen,

It seems there is some confusion. The document presented would appear
to be a Verified Accountant Letter (as defined in the EV Guidelines)
and can used as part of the process to validate a request for an EV
certificate.  It is not an audit report and is not something normally
submitted to browsers.

I suspect someone simply attached the wrong document to an email or
uploaded the wrong document.  This makes no sense to be part of an
audit report.

Thanks,
Peter
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy