Re: Symantec Issues List
On Mon, Apr 3, 2017 at 12:46 PM, Jakob Bohm via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > > How about this simple explanation (purely a guess, not at all checked): > I think we should focus on objective facts and statements. While there are a number of possible ways to interpret things, both positively and negatively, the fact that such multiple interpretations exist highlight a problem that needs public clarification and resolution - both on behalf of Symantec and their auditors. ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On 01/04/2017 16:08, Ryan Sleevi wrote: (Wearing a personal hat) This timeline hopefully highlights a particular serious issue: If NTT Docomo is operated as part of Symantec's operations, then there are several ways to interpret Symantec's audit statements: 1) KPMG failed to include NTT Docomo as part of the 5 externally operated sub-CAs noted, and instead treated it as part of Symantec's audit. If this is true, then there is an as-yet-unidentified intermediate certificate issued as part of the GeoRoot program 2) KPMG was treating NTT Docomo as part of the 5 externally operated sub-CAs noted. If this is correct, then it is in one of three sets a) The 3/5 sub-CAs for which KPMG identified as having audit reports b) The 1/5 sub-CAs for which KPMG identified as having a deficient audit report (not appropriate to the scheme) c) The 1/5 sub-CAs for which KPMG identified Symantec as having later received an audit report for. How about this simple explanation (purely a guess, not at all checked): KPMG was somehow sloppy in their wording (but not the auditing), listing NTT Docomo as external with audit report in one part of their audit work, yet that "NTT Docomo" audit report is actually either the very same "Symantec" audit report, or one of the other "Symantec" audit reports for a relevant period. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On 03/04/17 02:37, Peter Bowen wrote: > I believe Issue L is incorrectly dated. Thank you for this; I have updated Issue L to hopefully be more accurate, but I intend to keep it as a separate issue. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On 01/04/17 05:57, Peter Bowen wrote: > The GeoRoot program was very similar to that offered by many CAs a few > years ago. CyberTrust (then Verizon, now DigiCert) has the OmniRoot > program, Entrust has a root signing program[1], and GlobalSign Trusted > Root[2] are just a few examples. While this is true, it's not just about the existence of the legacy program with problems, but about how the situation is handled. Verizon were not able to bring their program into BR compliance; DigiCert bought it and worked closely with Mozilla to generate some breathing space for them to clean the system up. They posted public plans, kept us informed of the issues found and their plans for remediation, and completed the work pretty much on time. The Web PKI is a better place for it. This does not seem to be the approach taken by Symantec, if we accept Ryan's account of events. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On 01/04/17 00:38, Ryan Sleevi wrote: > On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham via dev-security-policy < > Thanks for organizing this information, as much of it was related to and > relevant to Google's recent announcement. I want to take this opportunity > to share additional details regarding the interactions for UniCredit, which > I believe may be useful and relevant both for understanding that issue and > the GeoTrust audits. Thank you for this. I will attempt to integrate it into the document and I'm sure we will hear comments from Symantec on it. Gerv ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Sun, Apr 2, 2017 at 9:36 PM, Ryan Sleevi wrote: > > On Sun, Apr 2, 2017 at 11:14 PM Peter Bowen via dev-security-policy > wrote: >> >> On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via >> dev-security-policy wrote: >> > As we continue to consider how best to react to the most recent incident >> > involving Symantec, and given that there is a question of whether it is >> > part of a pattern of behaviour, it seemed best to produce an issues list >> > as we did with WoSign. This means Symantec has proper opportunity to >> > respond to issues raised and those responses can be documented in one >> > place and the clearest overayll picture can be seen by the community. >> > >> > So I have prepared: >> > https://wiki.mozilla.org/CA:Symantec_Issues >> > >> > I will now be dropping Symantec an email asking them to begin the >> > process of providing whatever comment, factual correction or input they >> > feel appropriate. >> > >> > If anyone in this group feels they have an issue which it is appropriate >> > to add to the list, please send me email with the details. >> >> Gerv, >> >> I'm afraid that Issue V: RA Program Audit Issues (2013 or earlier - >> January 2017) has confused RAs with subordinate CAs. >> >> According to >> https://bug1334377.bmoattachments.org/attachment.cgi?id=8843448, >> Symantec has indicated that they have (had) four unconstrained third >> party RAs: CrossCert, Certisign, Certisur, and Certsuperior. These >> appear to fall into what the BRs call "Delegated Third Parties". No >> audit report seems to mention any issue with these RAs. >> >> Separately Symantec owned CAs have issued CA-certificates to several >> CAs that are not operated by Symantec. These appear to include at >> least Apple, Google, the US Government, Aetna, and Unicredit. The >> audit reports linked from Issue V appear to have qualifications >> regarding these CA-certificates. >> >> There are notable differences between third party owned CAs and third >> party operated RAs and the difference should be clearly noted. >> >> Thanks, >> Peter >> ___ >> dev-security-policy mailing list >> dev-security-policy@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-security-policy > > Both > https://www.symantec.com/content/en/us/about/media/repository/GeoTrust-WTBR-2015.pdf > (Finding number 3) and > https://www.symantec.com/content/en/us/about/media/repository/symantec-webtrust-audit-report.pdf > (Finding number 1) call out Delegated Third Parties as lacking audits. This > is called out separately from the matters related to sub-CAs, as > "Furthermore". > > Given that at least some of the sub-CAs possessed and provided audits to > Symantec, it does not seem to support your summary, but perhaps your point > was misunderstood? I think there are two parts: 1) There should be two different issues in the issues list -- one for management of Subordinate CAs and one for management of unconstrained RAs (i.e. Delegated Third Parties) 2) It is not clear that the audit reports for the GeoTrust brand roots are calling out RAs as qualifications. My read is that they were considering the subordinate CAs as DTPs, not the RAs. However I can see the other interpretation as well. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Sun, Apr 2, 2017 at 11:14 PM Peter Bowen via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via > dev-security-policy wrote: > > As we continue to consider how best to react to the most recent incident > > involving Symantec, and given that there is a question of whether it is > > part of a pattern of behaviour, it seemed best to produce an issues list > > as we did with WoSign. This means Symantec has proper opportunity to > > respond to issues raised and those responses can be documented in one > > place and the clearest overayll picture can be seen by the community. > > > > So I have prepared: > > https://wiki.mozilla.org/CA:Symantec_Issues > > > > I will now be dropping Symantec an email asking them to begin the > > process of providing whatever comment, factual correction or input they > > feel appropriate. > > > > If anyone in this group feels they have an issue which it is appropriate > > to add to the list, please send me email with the details. > > Gerv, > > I'm afraid that Issue V: RA Program Audit Issues (2013 or earlier - > January 2017) has confused RAs with subordinate CAs. > > According to > https://bug1334377.bmoattachments.org/attachment.cgi?id=8843448, > Symantec has indicated that they have (had) four unconstrained third > party RAs: CrossCert, Certisign, Certisur, and Certsuperior. These > appear to fall into what the BRs call "Delegated Third Parties". No > audit report seems to mention any issue with these RAs. > > Separately Symantec owned CAs have issued CA-certificates to several > CAs that are not operated by Symantec. These appear to include at > least Apple, Google, the US Government, Aetna, and Unicredit. The > audit reports linked from Issue V appear to have qualifications > regarding these CA-certificates. > > There are notable differences between third party owned CAs and third > party operated RAs and the difference should be clearly noted. > > Thanks, > Peter > ___ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > Both https://www.symantec.com/content/en/us/about/media/repository/GeoTrust-WTBR-2015.pdf (Finding number 3) and https://www.symantec.com/content/en/us/about/media/repository/symantec-webtrust-audit-report.pdf (Finding number 1) call out Delegated Third Parties as lacking audits. This is called out separately from the matters related to sub-CAs, as "Furthermore". Given that at least some of the sub-CAs possessed and provided audits to Symantec, it does not seem to support your summary, but perhaps your point was misunderstood? ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via dev-security-policy wrote: > As we continue to consider how best to react to the most recent incident > involving Symantec, and given that there is a question of whether it is > part of a pattern of behaviour, it seemed best to produce an issues list > as we did with WoSign. This means Symantec has proper opportunity to > respond to issues raised and those responses can be documented in one > place and the clearest overayll picture can be seen by the community. > > So I have prepared: > https://wiki.mozilla.org/CA:Symantec_Issues > > I will now be dropping Symantec an email asking them to begin the > process of providing whatever comment, factual correction or input they > feel appropriate. > > If anyone in this group feels they have an issue which it is appropriate > to add to the list, please send me email with the details. Gerv, I'm afraid that Issue V: RA Program Audit Issues (2013 or earlier - January 2017) has confused RAs with subordinate CAs. According to https://bug1334377.bmoattachments.org/attachment.cgi?id=8843448, Symantec has indicated that they have (had) four unconstrained third party RAs: CrossCert, Certisign, Certisur, and Certsuperior. These appear to fall into what the BRs call "Delegated Third Parties". No audit report seems to mention any issue with these RAs. Separately Symantec owned CAs have issued CA-certificates to several CAs that are not operated by Symantec. These appear to include at least Apple, Google, the US Government, Aetna, and Unicredit. The audit reports linked from Issue V appear to have qualifications regarding these CA-certificates. There are notable differences between third party owned CAs and third party operated RAs and the difference should be clearly noted. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Fri, Mar 31, 2017 at 11:39 AM, Gervase Markham via dev-security-policy wrote: > As we continue to consider how best to react to the most recent incident > involving Symantec, and given that there is a question of whether it is > part of a pattern of behaviour, it seemed best to produce an issues list > as we did with WoSign. This means Symantec has proper opportunity to > respond to issues raised and those responses can be documented in one > place and the clearest overayll picture can be seen by the community. > > So I have prepared: > https://wiki.mozilla.org/CA:Symantec_Issues > > I will now be dropping Symantec an email asking them to begin the > process of providing whatever comment, factual correction or input they > feel appropriate. > > If anyone in this group feels they have an issue which it is appropriate > to add to the list, please send me email with the details. Gerv, I believe Issue L is incorrectly dated. As can be seen on crt.sh, there are two CAs operated by the US federal Government which have been repeatedly issued certificates by various CAs trusted by Mozilla: https://crt.sh/?caid=1324 (Federal Bridge CA) https://crt.sh/?caid=1410 (Federal Bridge CA 2013) These two CAs have cross-certified each other and have been issued several certificates by VeriSign/Symantec and Digital Signature Trust/IdenTrust. The earliest date for VeriSign is 2011-02-03 and the earliest date for DST is 2011-01-14. I also think that Issue L should probably be combined with the GeoRoot items. Functionally they are the same issue: management and oversight of external subordinate CAs. Thanks, Peter ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Sat, Apr 1, 2017 at 12:57 AM, Peter Bowen wrote: > (Wearing my personal hat) > > Ryan, > > I haven't reviewed the audit reports myself, but I'll assume all you > wrote is true. However, I think it is important to consider it in the > appropriate context. > The GeoRoot program was very similar to that offered by many CAs a few > years ago. CyberTrust (then Verizon, now DigiCert) has the OmniRoot > program, Entrust has a root signing program[1], and GlobalSign Trusted > Root[2] are just a few examples. > > In almost every case the transition to requiring complete unqualified > audits of the subordinates by a licensed practitioner was a rocky one. > See DigiCert's thread > (https://groups.google.com/d/msg/mozilla.dev.security. > policy/tHUcqnWPt3o/U2U__7-UBQAJ) > about the OmniRoot program or look at the audits available for some of > the Entrust subordinates. > > I'm not suggesting that the GeoRoot subordinate issues should not be > considered, but it seems the GeoRoot program was not notably > exceptional a few years ago. > (Wearing a personal hat) Peter, There are a few issues to unpack from your reply. I think we're in agreement that GeoRoot was by no means unique as an offering. I think, when considering severity, it's important to instead focus on what the CAs obligations were, what they were aware of, and what they did in response. Further, in considering the broader scope of attempted remediation, it's important to consider what risks were or are present as a result of this, because it significantly impacts the ability to trust the existing set of issued certificates. On 2014-05-13, Mozilla requested all participating CAs disclose their externally operated subordinate CAs. [1] On 2014-06-03, Symantec reported it disclosed its sub-CAs in [2] On 2015-04-06, Kathleen pointed out Symantec's disclosure was incomplete, in [3] and [4] On 2016-03-29, Symantec informed Google that there were 5 participants in their GeoRoot program - Aetna, Google, Unicredit, Apple, NTT Docomo (DKHS). On 2016-05-11 (or later), Symantec received Aetna's audit. On 2016-05-13, Symantec's most recent audit for the Geotrust roots was made available [5], which states there were 5 external partner subordinate CAs. The timing of Aetna's letter suggests that this may be the audit that "Symantec subsequently received an audit report for the other" - but that cannot be confirmed without further detail from KPMG and Symantec. On 2016-06-28, Symantec informs Google that NTT Docomo is part of Symantec's audit, not separately audited. This timeline hopefully highlights a particular serious issue: If NTT Docomo is operated as part of Symantec's operations, then there are several ways to interpret Symantec's audit statements: 1) KPMG failed to include NTT Docomo as part of the 5 externally operated sub-CAs noted, and instead treated it as part of Symantec's audit. If this is true, then there is an as-yet-unidentified intermediate certificate issued as part of the GeoRoot program 2) KPMG was treating NTT Docomo as part of the 5 externally operated sub-CAs noted. If this is correct, then it is in one of three sets a) The 3/5 sub-CAs for which KPMG identified as having audit reports b) The 1/5 sub-CAs for which KPMG identified as having a deficient audit report (not appropriate to the scheme) c) The 1/5 sub-CAs for which KPMG identified Symantec as having later received an audit report for. If 2 is correct, then it's unclear of which set Aetna belongs to - that is, if NTT Docomo is 2a, then Aetna is either 2b/2c, and it suggests that KPMG may have been incomplete in its examination of the 2a set. If NTT Docomo is 2b, then Aetna is either 2a/2c, but calls into question Symantec's operations if they were themselves operating this root, as it was not part of the scope of the audit. If NTT Docomo is 2c, then Aetna is either 2a/2b, both of which would call into question KPMG. Any of these possibilities is quite troubling, but nowhere near as troubling as the possibility of 1, which would imply an undisclosed sub-CA. Based on the information provided by Unicredit, Unicredit would appear to be 2b, because it was not performed by a licensed WebTrust practitioner to the appropriate standards. Based on the information provided, Aetna would seem to be 2c, but that would require confirmation from Symantec or KPMG. This means that NTT Docomo is either 2a or 2c - either of which should be concerning. Independent of any questions regarding how other CAs (such as the critically mismanaged Omniroot program) responded to disclosure, the questions about the scope of "which sub-CAs were examined by KPMG" is very much relevant to the discussion at hand, and gets to the heart of whether or not there can be sufficient confidence to trust the existing set of certificates. This also sets aside the question about whether or not Symantec can/should be trusted going forward. It also highlights the limits of relying on a report such as [5], which may no
Re: Symantec Issues List
On Fri, Mar 31, 2017 at 4:38 PM, Ryan Sleevi via dev-security-policy wrote: > On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham wrote: > >> As we continue to consider how best to react to the most recent incident >> involving Symantec, and given that there is a question of whether it is >> part of a pattern of behaviour, it seemed best to produce an issues list >> as we did with WoSign. This means Symantec has proper opportunity to >> respond to issues raised and those responses can be documented in one >> place and the clearest overayll picture can be seen by the community. > > (Wearing a Google hat) (Wearing my normal personal non-work hat) > In March of last year, Symantec provided us a list of five sub-CAs which > they termed GeoRoots: Apple, Google, Unicredit, Aetna, NTT Docomo - and > requested they be excluded from this requirement. We asked Symantec to > provide current audit statements for each of these CAs. > > Symantec indicated that the audit information for these sub-CAs would be > added to the CCADB. This was on 3/29. > > We then followed-up with Symantec, again, because as of 6/28, there were > several outstanding issues with Symantec's disclosures: > > - Apple IST CA 3 was not covered by the general set of Apple audits > - No audit information for Aetna was provided, and its CPS was dated in 2011 > - No audit information for Unicredit was provided > - NTT Docomo (DKHS and DKHS CA2) were disclosed as being part of Symantec's > audit > > Upon follow-up, Symantec provided Aetna's WebTrust for BRs audit. On it, > there were 15 qualifications, some of which would have spanned the totality > of operation. > > Regarding Unicredit, Google requested that Symantec place us in direct > contact with Unicredit. We had several calls with Unicredit's management > team regarding the issues, attempting to find a path to see if they would > be able to complete a Baseline Requirements audit. > > I want to share these details so that a fuller picture of the GeoRoot > issues can be noted. Particularly concerning is the seriousness of the > Aetna issues and the failure to remedy them, and the failure to identify > the NTT Docomo (DKHS) roots as part of Symantec's infrastructure. (some portions of the quoted text omitted) Ryan, I haven't reviewed the audit reports myself, but I'll assume all you wrote is true. However, I think it is important to consider it in the appropriate context. The GeoRoot program was very similar to that offered by many CAs a few years ago. CyberTrust (then Verizon, now DigiCert) has the OmniRoot program, Entrust has a root signing program[1], and GlobalSign Trusted Root[2] are just a few examples. In almost every case the transition to requiring complete unqualified audits of the subordinates by a licensed practitioner was a rocky one. See DigiCert's thread (https://groups.google.com/d/msg/mozilla.dev.security.policy/tHUcqnWPt3o/U2U__7-UBQAJ) about the OmniRoot program or look at the audits available for some of the Entrust subordinates. I'm not suggesting that the GeoRoot subordinate issues should not be considered, but it seems the GeoRoot program was not notably exceptional a few years ago. Thanks, Peter [1] https://web-beta.archive.org/web/20140818191044/http://www.entrust.net/about/third-party-sub-ca.htm [2] https://www.globalsign.com/en/certificate-authority-root-signing/ and https://web-beta.archive.org/web/20101008151742/http://globalsign.com/certificate-authority-root-signing/ ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: Symantec Issues List
On Fri, Mar 31, 2017 at 2:39 PM, Gervase Markham via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > As we continue to consider how best to react to the most recent incident > involving Symantec, and given that there is a question of whether it is > part of a pattern of behaviour, it seemed best to produce an issues list > as we did with WoSign. This means Symantec has proper opportunity to > respond to issues raised and those responses can be documented in one > place and the clearest overayll picture can be seen by the community. > > So I have prepared: > https://wiki.mozilla.org/CA:Symantec_Issues > > I will now be dropping Symantec an email asking them to begin the > process of providing whatever comment, factual correction or input they > feel appropriate. > > If anyone in this group feels they have an issue which it is appropriate > to add to the list, please send me email with the details. > (Wearing a Google hat) Gerv, Thanks for organizing this information, as much of it was related to and relevant to Google's recent announcement. I want to take this opportunity to share additional details regarding the interactions for UniCredit, which I believe may be useful and relevant both for understanding that issue and the GeoTrust audits. As the Chrome team announced at https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html , we made steps to require that all Symantec-issued certificates be disclosed via Certificate Transparency. In March of last year, Symantec provided us a list of five sub-CAs which they termed GeoRoots: Apple, Google, Unicredit, Aetna, NTT Docomo - and requested they be excluded from this requirement. We asked Symantec to provide current audit statements for each of these CAs. Symantec indicated that the audit information for these sub-CAs would be added to the CCADB. This was on 3/29. We then followed-up with Symantec, again, because as of 6/28, there were several outstanding issues with Symantec's disclosures: - Apple IST CA 3 was not covered by the general set of Apple audits - No audit information for Aetna was provided, and its CPS was dated in 2011 - No audit information for Unicredit was provided - NTT Docomo (DKHS and DKHS CA2) were disclosed as being part of Symantec's audit Upon follow-up, Symantec provided Aetna's WebTrust for BRs audit. On it, there were 15 qualifications, some of which would have spanned the totality of operation. If Symantec is not willing or able to provide this audit, we would be happy to, in the interest of public transparency. This audit was dated May 11, 2016, but covered the period January 1, 2015 through December 31, 2015. I highlight this, because this means it was provided 132 days after the close of the period, or 42 days after provided for by the Baseline Requirements. This audit was performed by Symantec's auditors, KPMG, and thus demonstrates the pattern of delayed audits that KPMG has shown a tendency for, and extends beyond just Symantec. I want to highlight some of the qualifications on this audit: "It was noted that: physical access to the cage housing the CA system was not logged; event logs for the CA prior to July 2015 were not available; and logs are not reviewed periodically." "It was noted that no training programs were in place for Trusted Role personnel." "It was noted that a security risk assessment of the Aetna CA operations was not performed during the examination period." "It was noted that a security risk assessment of the Aetna CA operations was not performed during the examination period." "It was noted that penetration testing was not performed on the PKI environment during the examination period." Symantec's proposed remediation for this was to allow their contract to expire, and proposed revoking this certificate in October. Note that this was still nearly 3 months later. Regarding NTT Docomo, Symantec repeatedly asserted they controlled issuance for these CAs. However, I highlight this, because the Geotrust audits note 5 sub-CAs, so the fifth sub-CA, if not NTT Docomo, remains unknown and unidentified by Symantec during the same time as their audit. If NTT Docomo was issued and managed by Symantec, then their auditor (KPMG) did not examine this. Regarding Apple's IST CA 3 (and subsequently issued by Symantec, IST CA 8 - G1), Apple requested and Google accepted a discussion with their CA operations team. After discussions with the team, we received suitable assurances from Apple that these two CAs were operated in accordance with their CP/CPS and would be part of the next audit. As has been discussed in other threads, there is a natural period of time where the scope of the audit does not cover any newly issued intermediates, but we ensured that these were listed within Apple's next CP/CPS, and thus scoped for the next audit. Because of this reason alone, we excluded these sub-CAs from our CT requirement. Regarding Unicredit, Google requested that