Re: Auditor letters and incident reports
Thanks for the response Jeff. On Fri, Sep 6, 2019 at 4:17 PM jeffwardpki--- via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Wednesday, August 21, 2019 at 11:46:37 PM UTC-5, Jeremy Rowley wrote: > > Hey all, > > > > An interesting issue came up recently with audits. Because the Mozilla > policy includes some requirements that diverge from the BRs, the audit > criteria don't necessarily cover everything Mozilla cares about. Thus, it's > possible to have an incident that doesn't show up on an audit. It's also > possible that the auditor determines the incident is not sufficiently > important/risky(?) to include it in an audit. For example: > https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't > controlled by the CA and operate independently which means the CA can't > dictate what goes into the opinion. One solution is to require CAs to list > all of the incidents that occur during their audit in the management > assertion letter. I posted an addendum to the management assertion on that > thread. Going forward, we'll just include it as part of the main body. I > need to look into whether I can get our existing audit reissued to the > appendix is part of the seal as well. > > > > What do you think about just requiring that as part of the Mozilla > policy? Ie - the management assertion letter must include a list of the > incidents active/opened during the audit period. Something like that could > ensure transparency and make sure all incidents are disclosed to the > auditor, distinguishing the CA's disclosures from the auditors. > > > > Jeremy > > Reposting as I don't see my original response to this thread: > > Sorry for the delay in responding, but I first wanted to gather feedback > from the members of the WebTrust Task Force. Keep in mind any > audit/assurance engagement entails professional judgment, which of course > may vary from firm to firm. We are certainly seeing a trend in audit > reports whereby “Other Matters” are described that do not modify/qualify > the auditor’s opinion but do allow the auditor to make mention of the > issues. Some firms use this section to list items noted in Bugzilla, for > instance, to demonstrate that these types of issues were addressed in the > course of the audit. > > Depending on the firm, some firms have been listing all issues noted in > Bugzilla, while others are listing only those that are significant. As a > Task Force, it is not our position, or even a possibility, for us to > mandate how these should be handled as professional judgment will dictate > their treatment based on the respective firms’ risk management policies. > That being said, we have as part of our draft practitioner guidance > commentary that the browser community prefers seeing all these types of > issues listed as either modifications/qualifications, or described in > “Other Matters” and encourage practitioners to use this approach. > > I would go beyond "all issues noted in Bugzilla" to "all issues, including those noted in Bugzilla". In most cases, management’s Assertion accompanies the audit report, and > there is greater flexibility for what goes into them. Our professional > standards require certain items be present in the Assertion, but other > items that management wants to add are permissible to include. Except in > the rare reporting scenario when an assertion by management is not > provided, most of the Task Force members agree there would not be any issue > with your proposed requirement to require a CA's management to detail > Bugzilla or similar issues relevant to the current audit period in their > assertion to the auditors. > > By doing this, a CA can eliminate any question about disclosure of incidents to auditors, which is terrific and encouraged. As a Mozilla requirement it has the problem - as Ryan pointed out - that ETSI audits don't include management assertions, at least not today. > As far as the management representation letter, which is a required > written communication to the auditors at the conclusion to the audit, > regulatory non-compliance would normally form part of the management > representation that is needed for each audit, and the assertion can form > part of that representation. Keep in mind that the management > representation letter is not part of the deliverable that is viewed > publicly as is the case with the auditor’s opinion and management assertion. > > To demonstrate that the auditor has addressed the significance of the > relevant issues, we can propose that the standard WebTrust reports have an > Exhibit that sets out the issues identified by reference to Bugzilla, as an > example. The audit report can then reflect the relevant significance > either as a report qualification or through an "Other Matters" paragraph > that provides further commentary. The main issue is that, even by > identifying and addressing them in the audit report, there is still the > potential for disagreements t
Re: Auditor letters and incident reports
On Wednesday, August 21, 2019 at 11:46:37 PM UTC-5, Jeremy Rowley wrote: > Hey all, > > An interesting issue came up recently with audits. Because the Mozilla policy > includes some requirements that diverge from the BRs, the audit criteria > don't necessarily cover everything Mozilla cares about. Thus, it's possible > to have an incident that doesn't show up on an audit. It's also possible that > the auditor determines the incident is not sufficiently important/risky(?) to > include it in an audit. For example: > https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't > controlled by the CA and operate independently which means the CA can't > dictate what goes into the opinion. One solution is to require CAs to list > all of the incidents that occur during their audit in the management > assertion letter. I posted an addendum to the management assertion on that > thread. Going forward, we'll just include it as part of the main body. I need > to look into whether I can get our existing audit reissued to the appendix is > part of the seal as well. > > What do you think about just requiring that as part of the Mozilla policy? Ie > - the management assertion letter must include a list of the incidents > active/opened during the audit period. Something like that could ensure > transparency and make sure all incidents are disclosed to the auditor, > distinguishing the CA's disclosures from the auditors. > > Jeremy Reposting as I don't see my original response to this thread: Sorry for the delay in responding, but I first wanted to gather feedback from the members of the WebTrust Task Force. Keep in mind any audit/assurance engagement entails professional judgment, which of course may vary from firm to firm. We are certainly seeing a trend in audit reports whereby “Other Matters” are described that do not modify/qualify the auditor’s opinion but do allow the auditor to make mention of the issues. Some firms use this section to list items noted in Bugzilla, for instance, to demonstrate that these types of issues were addressed in the course of the audit. Depending on the firm, some firms have been listing all issues noted in Bugzilla, while others are listing only those that are significant. As a Task Force, it is not our position, or even a possibility, for us to mandate how these should be handled as professional judgment will dictate their treatment based on the respective firms’ risk management policies. That being said, we have as part of our draft practitioner guidance commentary that the browser community prefers seeing all these types of issues listed as either modifications/qualifications, or described in “Other Matters” and encourage practitioners to use this approach. In most cases, management’s Assertion accompanies the audit report, and there is greater flexibility for what goes into them. Our professional standards require certain items be present in the Assertion, but other items that management wants to add are permissible to include. Except in the rare reporting scenario when an assertion by management is not provided, most of the Task Force members agree there would not be any issue with your proposed requirement to require a CA's management to detail Bugzilla or similar issues relevant to the current audit period in their assertion to the auditors. As far as the management representation letter, which is a required written communication to the auditors at the conclusion to the audit, regulatory non-compliance would normally form part of the management representation that is needed for each audit, and the assertion can form part of that representation. Keep in mind that the management representation letter is not part of the deliverable that is viewed publicly as is the case with the auditor’s opinion and management assertion. To demonstrate that the auditor has addressed the significance of the relevant issues, we can propose that the standard WebTrust reports have an Exhibit that sets out the issues identified by reference to Bugzilla, as an example. The audit report can then reflect the relevant significance either as a report qualification or through an "Other Matters" paragraph that provides further commentary. The main issue is that, even by identifying and addressing them in the audit report, there is still the potential for disagreements to exist as to significance due to auditor’s professional judgment. Of course, in our new SOC-like report, these issues will be discussed in the system description, audit report and potentially an unaudited management comment section. Thanks, Jeff Ward ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Auditor letters and incident reports
Dear all, just a short note on that with regard to auditing and Audit Attestations based upon ETSI: throughout the audit we check the incidents of the current audit period as documented by the CA (have they been addressed at a sufficient level, have the measures taken proven that they are sufficient in depth and coverage, etc.). We are going to write a finding in case incidents seem to be not or not fully covered. We do have a corresponding section 7.9 in ETSI EN 319 401 dealing with that. Furthermore we agreed with the browsers to have a specific section in the Audit Attestation: any buzilla bugs reported in the audit period are going to be listed in the Attestation including their status (closed, open/reason). Hence, from my perspective we would not really need an additional requirement in the Mozilla policy for that. And please let's keep in mind that the best way to treat such things is to add them to a CA/B Forum requirements document rather than to a browsers root store policy as that affects only CAs being part of that Root Store rather than the whole community. Clemens ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Auditor letters and incident reports
On Thu, Aug 22, 2019 at 12:46 AM Jeremy Rowley via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Hey all, > > An interesting issue came up recently with audits. Because the Mozilla > policy includes some requirements that diverge from the BRs, the audit > criteria don't necessarily cover everything Mozilla cares about. Thus, it's > possible to have an incident that doesn't show up on an audit. It's also > possible that the auditor determines the incident is not sufficiently > important/risky(?) to include it in an audit. For example: > https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't > controlled by the CA and operate independently which means the CA can't > dictate what goes into the opinion. One solution is to require CAs to list > all of the incidents that occur during their audit in the management > assertion letter. I posted an addendum to the management assertion on that > thread. Going forward, we'll just include it as part of the main body. I > need to look into whether I can get our existing audit reissued to the > appendix is part of the seal as well. > > What do you think about just requiring that as part of the Mozilla policy? > Ie - the management assertion letter must include a list of the incidents > active/opened during the audit period. Something like that could ensure > transparency and make sure all incidents are disclosed to the auditor, > distinguishing the CA's disclosures from the auditors. > > Jeremy > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > Hey Jeremy, Huge thanks for raising this. Sadly, you’re about to be eaten by a grue [1] Browsers have definitely had conversations with both WebTrust and ETSI on this, and I recall some conversations to that effect during the CA/Browser Forum F2F discussions. One of the challenges is that ETSI and WebTrust are highly divergent in this area, in that an ETSI audit is not based on Management’s Assertions in the way that WebTrust is, but against a set of criteria (largely) CAs (via ETSI) developed for themselves, in order to meet certain European legal obligations. To date, the policy has left it flexible for that reason. Instead, we’ve tried to work very closely to explain the concerns to both WebTrust and to ETSI. To address these concerns, the WebTrust TF went back and examined their relevant professional obligations (e.g. under ISAE 3000 or AICPA’s AT-C standards) and developed illustrative reports that WebTrust-licensed auditors could use to opine on and disclose other relevant matters, such as incidents. While it’s certainly true that the WebTrust TF does not presently place an obligation on the auditor to disclose these matters, either via the (“externally” maintained) aforementioned professional standards or via the (TF-maintained) audit criteria, a competent and capable auditor should be familiar both with the illustrative reports and the policy requirements of the relevant report recipients. CAs contracting with their auditors can certainly ensure that the auditor understands the requirements placed upon the CA by its report recipients, and thus ensure the auditor will disclose information they are professionally empowered to disclose, and as demonstrated via the illustrative report. There are several subtle differences in the implications of the reporting, as to whether it’s disclosed via Management’s Assertion or disclosed as other findings, and much more stark differences with respect to ETSI reporting, so I’m not sure a one-size-fits-all approach works here. That said, I absolutely entirely welcome any and all additional disclosures that are made by CAs in their management letters, and that’s something commonly discussed with WebTrust in trying to find angles for formalizing various reporting and disclosure. This community is wholly open, so perhaps the best thing CAs can do is ensure their auditors actively follow, and perhaps even engage with, this community. We’ve definitely had folks from the audit community involved and seeking insight and input (without violating their professional obligations re: confidentiality or their professional opinion forming requirements), so perhaps it’s worth making sure your auditor does or commits to do so. I know Scott Perry was one of the original firms involved here, before the CPA licensing, as they were recognized by Mozilla as a competent auditor independent from Mozilla when they first proposed becoming an auditor many years ago (15+?) Perhaps they should revisit and be more closely following? I realize that answer is a bit “No, but yes, but no” - but it’s exactly the sort of discussion worth having, and it’s been a conversations browsers have been having for years now with the auditors, in trying to address auditor quality control issues. Hopefully, some of the auditors, and particularly the WTTF memb
RE: Auditor letters and incident reports
Full disclosure - this was not my idea, but I thought it was a really good one and worth bringing up here. -Original Message- From: dev-security-policy On Behalf Of Jeremy Rowley via dev-security-policy Sent: Wednesday, August 21, 2019 10:46 PM To: mozilla-dev-security-policy Subject: Auditor letters and incident reports Hey all, An interesting issue came up recently with audits. Because the Mozilla policy includes some requirements that diverge from the BRs, the audit criteria don't necessarily cover everything Mozilla cares about. Thus, it's possible to have an incident that doesn't show up on an audit. It's also possible that the auditor determines the incident is not sufficiently important/risky(?) to include it in an audit. For example: https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't controlled by the CA and operate independently which means the CA can't dictate what goes into the opinion. One solution is to require CAs to list all of the incidents that occur during their audit in the management assertion letter. I posted an addendum to the management assertion on that thread. Going forward, we'll just include it as part of the main body. I need to look into whether I can get our existing audit reissued to the appendix is part of the seal as well. What do you think about just requiring that as part of the Mozilla policy? Ie - the management assertion letter must include a list of the incidents active/opened during the audit period. Something like that could ensure transparency and make sure all incidents are disclosed to the auditor, distinguishing the CA's disclosures from the auditors. Jeremy ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
