Re: Misissuance and BR Audit Statements
Comodo responded to my question about disclosure of incidents to their auditor with the following statement [1]: It turns out that we did not disclose these to EY. That was down to Comodo CA not offering the evidence of these events during the audit evidence gathering phase. It was not our intention to mislead and we regret this short-coming on our part. We have amended our own internal audit process to make sure these are gathered up as part of the evidence pack to go to the auditors in future audit cycles. I think we should make it a Mozilla program requirement that CAs disclose incidents and misissuance to their auditors [2]. - Wayne [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993#c15 [2] https://github.com/mozilla/pkipolicy/issues/154 On Wed, Aug 15, 2018 at 12:09 PM Wayne Thayer wrote: > I went ahead and noted these DigiCert audits as a concern on the CCADB > record for Scott S. Perry CPA, PLLC. > > I do think it's important for CAs to disclose these issues to their > auditors, but I also expect auditors to discover them. > > - Wayne > > On Wed, Aug 15, 2018 at 8:21 AM Ben Wilson > wrote: > >> Re-sending >> >> -Original Message- >> From: Ben Wilson >> Sent: Wednesday, August 15, 2018 8:34 AM >> To: 'r...@sleevi.com' ; Wayne Thayer < >> wtha...@mozilla.com> >> Cc: mozilla-dev-security-policy < >> mozilla-dev-security-pol...@lists.mozilla.org> >> Subject: RE: Misissuance and BR Audit Statements >> >> Thanks, Ryan and Wayne, >> >> Going forward we'll work to improve our management letter disclosures to >> include reported mis-issuances during the audit period. >> >> Sincerely yours, >> >> Ben >> >> -Original Message- >> From: dev-security-policy >> On Behalf Of Ryan Sleevi via dev-security-policy >> Sent: Monday, August 13, 2018 3:57 PM >> To: Wayne Thayer >> Cc: mozilla-dev-security-policy < >> mozilla-dev-security-pol...@lists.mozilla.org> >> Subject: Re: Misissuance and BR Audit Statements >> >> Wayne, >> >> Thanks for raising this. I definitely find it surprising to see nothing >> noted on Comodo's report, as you call out. >> >> As another datapoint, consider this recent audit that is reported to be >> from DigiCert, by way of Amazon Trust Services' providing the audits for >> their externally operated sub-CAs in [A]. The scope of the WebTrust BR >> audit report in [B] contains in its scope "DigiCert ECC Extended Validation >> Server CA" of hash >> FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, >> which [C]. During that time, this CA issued a cert [D] as part of their >> improperly configured Onion issuance in [E], which was remediated in early >> March, within the audit period for [B]. I couldn't find it listed in the >> report. >> >> Looking over that period, there were two other (resolved) DigiCert >> issues, [F] and [G], which affect the CAs listed in scope of [B]. >> >> I was a bit surprised by this, as like you, I would have expected these >> to be called out by both Management's Assertion and the auditor. >> http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf >> provides some of the illustrative reports, but it appears to only provide >> templates for management on the result of obtaining a qualified report. >> >> [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 >> [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 >> [C] https://crt.sh/?id=23432431 >> [D] https://crt.sh/?id=351449246 >> [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 >> [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 >> [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 >> >> On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < >> dev-security-policy@lists.mozilla.org> wrote: >> >> > Given the number of incidents documented over the past year [1][2] for >> > misissuance and other nonconformities, I would expect many of the 2018 >> > period-of-time WebTrust audit statements being submitted by CAs to >> > include qualifications describing these matters. In some cases, that >> > is exactly what we’re seeing. One of many positive examples is >> > Deloitte’s report on Entrust [3] that includes 2 of the 3 issues >> documented in Bugzilla. >> > >> > Unfortunately, we are also beginning to see some reports that don’t >> > meet my expectations. I was surprised by GlobalSign’s clean reports >> > [4] from Ernst & Young, but after examining their i
Re: Misissuance and BR Audit Statements
Thank you for responding on behalf of ETSI ESI and ACABc! I believe that this is an important topic and I hope that ETSI ESI and ACABc members will continue to participate in the discussion. On Thu, Aug 16, 2018 at 11:11 AM clemens.wanko--- via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Dear all, > this is a joint response from ETSI ESI and ACABc: > > ETSI have published a supplement to its audit requirements specifically to > address specific requirements of Mozilla, and other CA/Browser Forum > members, for auditing Trust Service Providers that issue Publicly-Trusted > Certificates TS 119 403-2. This is available for download at: > https://www.etsi.org/standards-search#search=TS119403-2 > This is extremely helpful in normalizing ETSI Audit Attestations and ensuring that they contain the basic information that Mozilla requires, but it provides no guidance on documenting non-conformities. > > > With regard to the treatment of non-conformities it says in PTA-4.3-08: > The Audit Attestation shall be issued only if no critical non-conformities > are identified. > > > Yes, as I mentioned in my original email, this is the fundamental concern I have with ETSI audits: qualified Audit Attestation reports cannot be issued. I do not even understand how it is possible for ETSI auditors to provide an unbroken sequence of period-of-time Audit Attestations if a "critical non-conformity" occurs. Apparently once the non-conformity is remediated it is no longer "critical"? This process gives relying parties no visibility into the problems identified by ETSI audits. > > ETSI audits do cover the CA incident management. That includes the whole > process including the timely treatment of incidents as well how to > guarantee proper and comprehensive responses to incidents. In ETSI EN 310 > 401 corresponding requirements are not only provided directly by section > 7.9 Incident Management but also through the requirement for a ISMS as > stated in section 5. Assessing that, the ETSI auditor will look at the CA > incident treatment during the Stage 1 as well as during the Stage 2 onsite > part of the audit. This includes the treatment of Bugzilla and cert.sh > listed issues. Incidents closed by the CA may have resulted in a change in > the CA operations. In such cases the auditor checks that the changes are > functioning correctly as defined by the CA. In that way the auditor is > assessing the incident management as such including possible measures taken > to avoid such incidents in the future and at the implemented measures > itself. > > > Good. But if, for example, a CA were to repeatedly and systematically fail to respond to incident reports within 24 hours. Would that non-conformity appear on the Audit Attestation? This has appeared as a qualification on a number of recently completed WebTrust audit statements. > > Another matter is the question of how to handle security related incidents > and the counter measures taken by a CA in audit reports. In order to keep > the security issue confidential as well as the details of the measures > taken by the CA, the accredited CABS (ETSI auditors) decided to document > such findings in their detailed audit reports. These detailed reports list > all relevant non conformities and the counter measures taken by the CA. It > is handed over to the CA in addition to the audit attestation. Based upon > that detailed report, the ETSI auditor will compile the Audit Attestation > as the browsers have it in their hand. The contents of the Audit > Attestation as summary document was agreed upon between ACAB’c and the CA/B > Browser Forum. If you regard it helpful to add information about the audit > results gained in the area of the CA incident treatment, we can certainly > discuss that. We then should reach a common agreement on what exactly we > add. > > Yes, this would be helpful information, and I agree that there should be a discussion resulting in auditor guidance on what to report to ensure that we get consistent information from all auditors. > > > We certainly believe however, that it is not advisable to publish detected > weak points. E.g. there might be findings in the way that the CA has NOT > correctly treated an incident. In that case the ETSI auditor will document > such findings and will not issue a positive report as well as no Audit > Attestation. The CA is then obliged to immediately install appropriate > counter measures which again will be judged by the auditor. Only if the > counter measures are rated sufficient in coverage and suitability by the > auditor, he will issue a positive report and an Audit Attestation. > > > Any sufficiently serious vulnerability should of course be fixed immediately. Once it has been fixed, there is value in disclosing the problem. It provides relying parties with information on the risks they are taking in trusting the CA and can result in the development of better practices across the industry. This is
RE: Misissuance and BR Audit Statements
What about all of the other audit firms? From: Wayne Thayer Sent: Wednesday, August 15, 2018 1:09 PM To: Ben Wilson Cc: Ryan Sleevi ; mozilla-dev-security-policy Subject: Re: Misissuance and BR Audit Statements I went ahead and noted these DigiCert audits as a concern on the CCADB record for Scott S. Perry CPA, PLLC. I do think it's important for CAs to disclose these issues to their auditors, but I also expect auditors to discover them. - Wayne On Wed, Aug 15, 2018 at 8:21 AM Ben Wilson mailto:ben.wil...@digicert.com> > wrote: Re-sending -Original Message- From: Ben Wilson Sent: Wednesday, August 15, 2018 8:34 AM To: 'r...@sleevi.com <mailto:r...@sleevi.com> ' mailto:r...@sleevi.com> >; Wayne Thayer mailto:wtha...@mozilla.com> > Cc: mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: RE: Misissuance and BR Audit Statements Thanks, Ryan and Wayne, Going forward we'll work to improve our management letter disclosures to include reported mis-issuances during the audit period. Sincerely yours, Ben -Original Message- From: dev-security-policy mailto:dev-security-policy-boun...@lists.mozilla.org> > On Behalf Of Ryan Sleevi via dev-security-policy Sent: Monday, August 13, 2018 3:57 PM To: Wayne Thayer mailto:wtha...@mozilla.com> > Cc: mozilla-dev-security-policy mailto:mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Misissuance and BR Audit Statements Wayne, Thanks for raising this. I definitely find it surprising to see nothing noted on Comodo's report, as you call out. As another datapoint, consider this recent audit that is reported to be from DigiCert, by way of Amazon Trust Services' providing the audits for their externally operated sub-CAs in [A]. The scope of the WebTrust BR audit report in [B] contains in its scope "DigiCert ECC Extended Validation Server CA" of hash FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, which [C]. During that time, this CA issued a cert [D] as part of their improperly configured Onion issuance in [E], which was remediated in early March, within the audit period for [B]. I couldn't find it listed in the report. Looking over that period, there were two other (resolved) DigiCert issues, [F] and [G], which affect the CAs listed in scope of [B]. I was a bit surprised by this, as like you, I would have expected these to be called out by both Management's Assertion and the auditor. http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf provides some of the illustrative reports, but it appears to only provide templates for management on the result of obtaining a qualified report. [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 [C] https://crt.sh/?id=23432431 [D] https://crt.sh/?id=351449246 [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org <mailto:dev-security-policy@lists.mozilla.org> > wrote: > Given the number of incidents documented over the past year [1][2] for > misissuance and other nonconformities, I would expect many of the 2018 > period-of-time WebTrust audit statements being submitted by CAs to > include qualifications describing these matters. In some cases, that > is exactly what we’re seeing. One of many positive examples is > Deloitte’s report on Entrust [3] that includes 2 of the 3 issues documented > in Bugzilla. > > Unfortunately, we are also beginning to see some reports that don’t > meet my expectations. I was surprised by GlobalSign’s clean reports > [4] from Ernst & Young, but after examining their incident bugs, it > appears that the only documented misissuance that occurred during > their audit period was placing metadata in Subject fields. I can > understand how this could be regarded as a minor nonconformity rather > than a qualification, but I would have liked to at least see the issue noted > in the reports. > > Ernst & Young’s clean reports on Comodo CA [5] is the example that > prompted this message. We have documented the following issues that > occurred during Comodo’s last audit period: > * Misissuance using "CNAME CSR Hash 2" method of domain control > validation (bug 1461391) > * Assorted misissuances and failure to respond to an incident report > within > 24 hours (bug 1390981) > * CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 ) > > I would like to know if Comodo reported these issues to EY. I asked > Comodo this question four weeks ago [6] but have not received a
Re: Misissuance and BR Audit Statements
Dear all, this is a joint response from ETSI ESI and ACABc: ETSI have published a supplement to its audit requirements specifically to address specific requirements of Mozilla, and other CA/Browser Forum members, for auditing Trust Service Providers that issue Publicly-Trusted Certificates TS 119 403-2. This is available for download at: https://www.etsi.org/standards-search#search=TS119403-2 With regard to the treatment of non-conformities it says in PTA-4.3-08: The Audit Attestation shall be issued only if no critical non-conformities are identified. ETSI audits do cover the CA incident management. That includes the whole process including the timely treatment of incidents as well how to guarantee proper and comprehensive responses to incidents. In ETSI EN 310 401 corresponding requirements are not only provided directly by section 7.9 Incident Management but also through the requirement for a ISMS as stated in section 5. Assessing that, the ETSI auditor will look at the CA incident treatment during the Stage 1 as well as during the Stage 2 onsite part of the audit. This includes the treatment of Bugzilla and cert.sh listed issues. Incidents closed by the CA may have resulted in a change in the CA operations. In such cases the auditor checks that the changes are functioning correctly as defined by the CA. In that way the auditor is assessing the incident management as such including possible measures taken to avoid such incidents in the future and at the implemented measures itself. Another matter is the question of how to handle security related incidents and the counter measures taken by a CA in audit reports. In order to keep the security issue confidential as well as the details of the measures taken by the CA, the accredited CABS (ETSI auditors) decided to document such findings in their detailed audit reports. These detailed reports list all relevant non conformities and the counter measures taken by the CA. It is handed over to the CA in addition to the audit attestation. Based upon that detailed report, the ETSI auditor will compile the Audit Attestation as the browsers have it in their hand. The contents of the Audit Attestation as summary document was agreed upon between ACAB’c and the CA/B Browser Forum. If you regard it helpful to add information about the audit results gained in the area of the CA incident treatment, we can certainly discuss that. We then should reach a common agreement on what exactly we add. We certainly believe however, that it is not advisable to publish detected weak points. E.g. there might be findings in the way that the CA has NOT correctly treated an incident. In that case the ETSI auditor will document such findings and will not issue a positive report as well as no Audit Attestation. The CA is then obliged to immediately install appropriate counter measures which again will be judged by the auditor. Only if the counter measures are rated sufficient in coverage and suitability by the auditor, he will issue a positive report and an Audit Attestation. Regards Clemens ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Misissuance and BR Audit Statements
I went ahead and noted these DigiCert audits as a concern on the CCADB record for Scott S. Perry CPA, PLLC. I do think it's important for CAs to disclose these issues to their auditors, but I also expect auditors to discover them. - Wayne On Wed, Aug 15, 2018 at 8:21 AM Ben Wilson wrote: > Re-sending > > -Original Message- > From: Ben Wilson > Sent: Wednesday, August 15, 2018 8:34 AM > To: 'r...@sleevi.com' ; Wayne Thayer > > Cc: mozilla-dev-security-policy < > mozilla-dev-security-pol...@lists.mozilla.org> > Subject: RE: Misissuance and BR Audit Statements > > Thanks, Ryan and Wayne, > > Going forward we'll work to improve our management letter disclosures to > include reported mis-issuances during the audit period. > > Sincerely yours, > > Ben > > -Original Message- > From: dev-security-policy > On Behalf Of Ryan Sleevi via dev-security-policy > Sent: Monday, August 13, 2018 3:57 PM > To: Wayne Thayer > Cc: mozilla-dev-security-policy < > mozilla-dev-security-pol...@lists.mozilla.org> > Subject: Re: Misissuance and BR Audit Statements > > Wayne, > > Thanks for raising this. I definitely find it surprising to see nothing > noted on Comodo's report, as you call out. > > As another datapoint, consider this recent audit that is reported to be > from DigiCert, by way of Amazon Trust Services' providing the audits for > their externally operated sub-CAs in [A]. The scope of the WebTrust BR > audit report in [B] contains in its scope "DigiCert ECC Extended Validation > Server CA" of hash > FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, > which [C]. During that time, this CA issued a cert [D] as part of their > improperly configured Onion issuance in [E], which was remediated in early > March, within the audit period for [B]. I couldn't find it listed in the > report. > > Looking over that period, there were two other (resolved) DigiCert issues, > [F] and [G], which affect the CAs listed in scope of [B]. > > I was a bit surprised by this, as like you, I would have expected these to > be called out by both Management's Assertion and the auditor. > http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf > provides some of the illustrative reports, but it appears to only provide > templates for management on the result of obtaining a qualified report. > > [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 > [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 > [C] https://crt.sh/?id=23432431 > [D] https://crt.sh/?id=351449246 > [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 > [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 > [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 > > On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < > dev-security-policy@lists.mozilla.org> wrote: > > > Given the number of incidents documented over the past year [1][2] for > > misissuance and other nonconformities, I would expect many of the 2018 > > period-of-time WebTrust audit statements being submitted by CAs to > > include qualifications describing these matters. In some cases, that > > is exactly what we’re seeing. One of many positive examples is > > Deloitte’s report on Entrust [3] that includes 2 of the 3 issues > documented in Bugzilla. > > > > Unfortunately, we are also beginning to see some reports that don’t > > meet my expectations. I was surprised by GlobalSign’s clean reports > > [4] from Ernst & Young, but after examining their incident bugs, it > > appears that the only documented misissuance that occurred during > > their audit period was placing metadata in Subject fields. I can > > understand how this could be regarded as a minor nonconformity rather > > than a qualification, but I would have liked to at least see the issue > noted in the reports. > > > > Ernst & Young’s clean reports on Comodo CA [5] is the example that > > prompted this message. We have documented the following issues that > > occurred during Comodo’s last audit period: > > * Misissuance using "CNAME CSR Hash 2" method of domain control > > validation (bug 1461391) > > * Assorted misissuances and failure to respond to an incident report > > within > > 24 hours (bug 1390981) > > * CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 ) > > > > I would like to know if Comodo reported these issues to EY. I asked > > Comodo this question four weeks ago [6] but have not received a response. > > > > I will acknowledge that ETSI audits are an even bigger problem > > (Actalis and SwissSign a
RE: Misissuance and BR Audit Statements
Re-sending -Original Message- From: Ben Wilson Sent: Wednesday, August 15, 2018 8:34 AM To: 'r...@sleevi.com' ; Wayne Thayer Cc: mozilla-dev-security-policy Subject: RE: Misissuance and BR Audit Statements Thanks, Ryan and Wayne, Going forward we'll work to improve our management letter disclosures to include reported mis-issuances during the audit period. Sincerely yours, Ben -Original Message- From: dev-security-policy On Behalf Of Ryan Sleevi via dev-security-policy Sent: Monday, August 13, 2018 3:57 PM To: Wayne Thayer Cc: mozilla-dev-security-policy Subject: Re: Misissuance and BR Audit Statements Wayne, Thanks for raising this. I definitely find it surprising to see nothing noted on Comodo's report, as you call out. As another datapoint, consider this recent audit that is reported to be from DigiCert, by way of Amazon Trust Services' providing the audits for their externally operated sub-CAs in [A]. The scope of the WebTrust BR audit report in [B] contains in its scope "DigiCert ECC Extended Validation Server CA" of hash FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, which [C]. During that time, this CA issued a cert [D] as part of their improperly configured Onion issuance in [E], which was remediated in early March, within the audit period for [B]. I couldn't find it listed in the report. Looking over that period, there were two other (resolved) DigiCert issues, [F] and [G], which affect the CAs listed in scope of [B]. I was a bit surprised by this, as like you, I would have expected these to be called out by both Management's Assertion and the auditor. http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf provides some of the illustrative reports, but it appears to only provide templates for management on the result of obtaining a qualified report. [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 [C] https://crt.sh/?id=23432431 [D] https://crt.sh/?id=351449246 [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Given the number of incidents documented over the past year [1][2] for > misissuance and other nonconformities, I would expect many of the 2018 > period-of-time WebTrust audit statements being submitted by CAs to > include qualifications describing these matters. In some cases, that > is exactly what we’re seeing. One of many positive examples is > Deloitte’s report on Entrust [3] that includes 2 of the 3 issues documented > in Bugzilla. > > Unfortunately, we are also beginning to see some reports that don’t > meet my expectations. I was surprised by GlobalSign’s clean reports > [4] from Ernst & Young, but after examining their incident bugs, it > appears that the only documented misissuance that occurred during > their audit period was placing metadata in Subject fields. I can > understand how this could be regarded as a minor nonconformity rather > than a qualification, but I would have liked to at least see the issue noted > in the reports. > > Ernst & Young’s clean reports on Comodo CA [5] is the example that > prompted this message. We have documented the following issues that > occurred during Comodo’s last audit period: > * Misissuance using "CNAME CSR Hash 2" method of domain control > validation (bug 1461391) > * Assorted misissuances and failure to respond to an incident report > within > 24 hours (bug 1390981) > * CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 ) > > I would like to know if Comodo reported these issues to EY. I asked > Comodo this question four weeks ago [6] but have not received a response. > > I will acknowledge that ETSI audits are an even bigger problem > (Actalis and SwissSign are recent examples [7][8][9]). Due to the > structure of those audits, there is no provision for issuing a > qualified report. WebTrust audits are theoretically much better in > this regard, but only if auditors actually find and report on issues! > I don’t think it is productive to expect auditors to search Bugzilla > for a list of issues to copy into their reports, but I do think it is > reasonable to question the competence and trustworthiness of the > auditor when so many known issues are absent from their report. > > In this particular example, unless additional facts are presented, I > plan to notate the auditor’s record in CCADB with this issue. We have > documented a number of other issues with Ernst & Young - including the > disqualification of their Hong Kong b
RE: Misissuance and BR Audit Statements
Thanks, Ryan and Wayne, Going forward we'll work to improve our management letter disclosures to include reported mis-issuances during the audit period. Sincerely yours, Ben -Original Message- From: dev-security-policy On Behalf Of Ryan Sleevi via dev-security-policy Sent: Monday, August 13, 2018 3:57 PM To: Wayne Thayer Cc: mozilla-dev-security-policy Subject: Re: Misissuance and BR Audit Statements Wayne, Thanks for raising this. I definitely find it surprising to see nothing noted on Comodo's report, as you call out. As another datapoint, consider this recent audit that is reported to be from DigiCert, by way of Amazon Trust Services' providing the audits for their externally operated sub-CAs in [A]. The scope of the WebTrust BR audit report in [B] contains in its scope "DigiCert ECC Extended Validation Server CA" of hash FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, which [C]. During that time, this CA issued a cert [D] as part of their improperly configured Onion issuance in [E], which was remediated in early March, within the audit period for [B]. I couldn't find it listed in the report. Looking over that period, there were two other (resolved) DigiCert issues, [F] and [G], which affect the CAs listed in scope of [B]. I was a bit surprised by this, as like you, I would have expected these to be called out by both Management's Assertion and the auditor. http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf provides some of the illustrative reports, but it appears to only provide templates for management on the result of obtaining a qualified report. [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 [C] https://crt.sh/?id=23432431 [D] https://crt.sh/?id=351449246 [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Given the number of incidents documented over the past year [1][2] for > misissuance and other nonconformities, I would expect many of the 2018 > period-of-time WebTrust audit statements being submitted by CAs to > include qualifications describing these matters. In some cases, that > is exactly what we’re seeing. One of many positive examples is > Deloitte’s report on Entrust [3] that includes 2 of the 3 issues documented > in Bugzilla. > > Unfortunately, we are also beginning to see some reports that don’t > meet my expectations. I was surprised by GlobalSign’s clean reports > [4] from Ernst & Young, but after examining their incident bugs, it > appears that the only documented misissuance that occurred during > their audit period was placing metadata in Subject fields. I can > understand how this could be regarded as a minor nonconformity rather > than a qualification, but I would have liked to at least see the issue noted > in the reports. > > Ernst & Young’s clean reports on Comodo CA [5] is the example that > prompted this message. We have documented the following issues that > occurred during Comodo’s last audit period: > * Misissuance using "CNAME CSR Hash 2" method of domain control > validation (bug 1461391) > * Assorted misissuances and failure to respond to an incident report > within > 24 hours (bug 1390981) > * CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 ) > > I would like to know if Comodo reported these issues to EY. I asked > Comodo this question four weeks ago [6] but have not received a response. > > I will acknowledge that ETSI audits are an even bigger problem > (Actalis and SwissSign are recent examples [7][8][9]). Due to the > structure of those audits, there is no provision for issuing a > qualified report. WebTrust audits are theoretically much better in > this regard, but only if auditors actually find and report on issues! > I don’t think it is productive to expect auditors to search Bugzilla > for a list of issues to copy into their reports, but I do think it is > reasonable to question the competence and trustworthiness of the > auditor when so many known issues are absent from their report. > > In this particular example, unless additional facts are presented, I > plan to notate the auditor’s record in CCADB with this issue. We have > documented a number of other issues with Ernst & Young - including the > disqualification of their Hong Kong branch - but this is the first > issue I’m aware of from their New York office. We also recently > received a very “good” qualified audit report from EY’s Denmark office on > Telia [10]. > > - Wayne > >
Re: Misissuance and BR Audit Statements
Wayne, Thanks for raising this. I definitely find it surprising to see nothing noted on Comodo's report, as you call out. As another datapoint, consider this recent audit that is reported to be from DigiCert, by way of Amazon Trust Services' providing the audits for their externally operated sub-CAs in [A]. The scope of the WebTrust BR audit report in [B] contains in its scope "DigiCert ECC Extended Validation Server CA" of hash FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862, which [C]. During that time, this CA issued a cert [D] as part of their improperly configured Onion issuance in [E], which was remediated in early March, within the audit period for [B]. I couldn't find it listed in the report. Looking over that period, there were two other (resolved) DigiCert issues, [F] and [G], which affect the CAs listed in scope of [B]. I was a bit surprised by this, as like you, I would have expected these to be called out by both Management's Assertion and the auditor. http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf provides some of the illustrative reports, but it appears to only provide templates for management on the result of obtaining a qualified report. [A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930 [B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669 [C] https://crt.sh/?id=23432431 [D] https://crt.sh/?id=351449246 [E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192 [F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600 [G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29 On Tue, Aug 7, 2018 at 1:32 PM, Wayne Thayer via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > Given the number of incidents documented over the past year [1][2] for > misissuance and other nonconformities, I would expect many of the 2018 > period-of-time WebTrust audit statements being submitted by CAs to include > qualifications describing these matters. In some cases, that is exactly > what we’re seeing. One of many positive examples is Deloitte’s report on > Entrust [3] that includes 2 of the 3 issues documented in Bugzilla. > > Unfortunately, we are also beginning to see some reports that don’t meet my > expectations. I was surprised by GlobalSign’s clean reports [4] from Ernst > & Young, but after examining their incident bugs, it appears that the only > documented misissuance that occurred during their audit period was placing > metadata in Subject fields. I can understand how this could be regarded as > a minor nonconformity rather than a qualification, but I would have liked > to at least see the issue noted in the reports. > > Ernst & Young’s clean reports on Comodo CA [5] is the example that prompted > this message. We have documented the following issues that occurred during > Comodo’s last audit period: > * Misissuance using "CNAME CSR Hash 2" method of domain control validation > (bug 1461391) > * Assorted misissuances and failure to respond to an incident report within > 24 hours (bug 1390981) > * CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 ) > > I would like to know if Comodo reported these issues to EY. I asked Comodo > this question four weeks ago [6] but have not received a response. > > I will acknowledge that ETSI audits are an even bigger problem (Actalis and > SwissSign are recent examples [7][8][9]). Due to the structure of those > audits, there is no provision for issuing a qualified report. WebTrust > audits are theoretically much better in this regard, but only if auditors > actually find and report on issues! I don’t think it is productive to > expect auditors to search Bugzilla for a list of issues to copy into their > reports, but I do think it is reasonable to question the competence and > trustworthiness of the auditor when so many known issues are absent from > their report. > > In this particular example, unless additional facts are presented, I plan > to notate the auditor’s record in CCADB with this issue. We have documented > a number of other issues with Ernst & Young - including the > disqualification of their Hong Kong branch - but this is the first issue > I’m aware of from their New York office. We also recently received a very > “good” qualified audit report from EY’s Denmark office on Telia [10]. > > - Wayne > > [1] https://wiki.mozilla.org/CA/Incident_Dashboard > [2] https://wiki.mozilla.org/CA/Closed_Incidents > [3] > https://www.entrustdatacard.com/-/media/documentation/ > licensingandagreements/entrust_baselinerequirements_2018.pdf?la=en= > BC08BAF5AE81B2EE66A2146EE7710FB2F4F33BA6 > [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1388488 > [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993 > [6] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993#c5 > [7] https://www.actalis.it/documenti-en/actalisca_audit_ > statement_2018.aspx > [8] > https://it-tuv.com/wp-content/uploads/2018/07/AA2018070301_ > Audit_Attestation_TA_CERT__SwissSign_Platinum_G2_signed.pdf
