First, let me apologize for the delay in my response, I have had a draft of this letter in my inbox for a while and have just been unable to get back to it and finish it due to scheduling conflicts. I promise to address all other questions in a more prompt manner.
> pzb: Mozilla recognizes 2.23.140.1.1 as being a valid OID for EVcertificates for all EV-enabled roots > (https://bugzilla.mozilla.org/show_bug.cgi?id=1243923). > 1) Do you consider it mis-issuance for Google to issue a certificate containing the 2.23.140.1.1 OID? > Policy Suggestion A) When transferring a root that is EV enabled, it should be clearly stated whether the > recipient of the root is also receiving the EV policy OID(s). rmh: Yes. We believe that until we have: - The associated policies, procedures, and other associated work completed, - Have successfully completed an EV audit, - And have been approved by one or more of the various root programs as an EV issuer. That it would be an example of miss-issuance for us to issue such a certificate. > pzb: Second, according to the GTS CPS v1.3, "Between 11 August 2016 and 8 December 2016, Google Inc. operated these Roots according to Google Inc.’s Certification Practice Statement." The basic WebTrust for CA and WebTrust BR audit reports for the period ending September 30, 2016 explicitly state they are for "subordinate CA under external Root CA" and do not list the roots in the GTS CPS at all. > > rmh: I believe this will be answered by my responses to your third and fourth observations. > It was not. rmh: I just attached two opinion letters from our auditors, I had previously provided these to the root programs directly but it took some time to get permission to release them publicly. One letter is covering the key generation ceremony of the new roots, and another covering the transfer of the keys to our facilities. In this second report you will find the following statement: ``` In our opinion, as of November 17, 2016, Google Trust Services LLC Management’s Assertion, as referred to above, is fairly stated, in all material respects, based on Certification Practices Statement Management Criterion 2.2, Asset Classification and Management Criterion 3.2, and Key Storage, Backup and Recovery Criterion 4.2 of the WebTrust Principles and Criteria for Certification Authorities v2.0. ``` Based on our conversations with the various root program operator's prior to our acquisition it has been our plan and understanding, that we can utilize these opinion letters to augment the webtrust audit with the material details, relating to these activities. It is our hope that this also addresses you specific concern here. > 2) Will Google be publishing an audit report for a period starting 11 > August 2016 that covers the transferred GS roots? If so, can you > estimate the end of period date? rmh: It is our belief, based on our conversations with the various root store operators, as well as our own auditors that the transfer itself is covered by the opinion letters. With that said our audit period is October 1st to the end of September. The associated report will be released between October and November, depending on our auditors schedules. > pzb: I think that this is the key issue. In my reading, "root > certificates" are not members of the program. Rather organizations > (legal entities) are members and each member has some number of root > certificates. > Google was not a member of the program and had not applied to be a > member of the program at the time they received the roots already in > the program. This seems problematic to me. > Policy Suggestion B) Require that any organization wishing to become a > member of the program submit a bug with links to content demonstrating > compliance with the Mozilla policy. Require that this be public prior > to taking control of any root in the program. > Policy Suggestion C) Recognize that root transfers are distinct from > the acquisition of a program member. Acquisition of a program member > (meaning purchase of the company) is a distinctly different activity > from moving only a private key, as the prior business controls no > longer apply in the latter case. We discussed the topic of disclosure with the root program administrators prior to our acquisition. Our goal was to tell the community as soon as possible, the complexity of this transaction made it hard to get a hard date for that announcement. Based on our conversations with root program administrators we were told the policy did not require disclosure to be public which left the timing of that notification up to us. As for the recommendation to clarify the policy in this area, I think it would be valuable to do that. Both of your recommendations seem reasonable, my concern with B) is how to do so in a way that does not make it impossible or even more complicated to successfully negotiate such a deal. As long as the disclosure was limited to intent to become a member then I think that goal would be achieved. > pzb: I don't see how having a subordinate issuing CA leads to the > conclusion that they have "facilities and procedures appropriate for > offline key management". Running an online and offline CA are two > rather different things and usually have different controls. rmh: We understand this concern. We believe that not all operators of a subordinate CA have the policies and procedures to effectively manage a root ca and its keys. However, in our case, for various reasons, Google did have the associated policies and procedures. Prior to the acquisition we discussed this situation with the browser root programs, and mutually came to the conclusion that relying on the opinion letters from the auditors to demonstrate these would be sufficient. > Policy Suggestion D) Moving from being a RA to a CA or moving from > being a single tier/online (i.e. Subordinate-only) CA to being a > multi-tier/root CA requires a PITRA > We have provided these letters directly to the root programs and have recently secured permission from the auditors to release them publicly (I will add them to the bug). > > For those not familiar with the process, If Google had never been a WebPKI CA, this situation would have been addressed with a pre-issuance audit and a subsequent full audit 90 days later. > > Since Google is a long-time WebTrust audited CA and our audits and acquisition were going to happen approximately at the same time, this would have provided no new evidence to the root programs or the community. >pzb: I disagree. There is no evidence of operational controls over > Subordinate CA Certificate Life Cycle Management. This is highlighted > by the fact that the neither the management assertion nor the opinion > list this topic at all. First, for others, Google was, and is, operating as a subordinate CA with its own facilities, policies, and procedures and not an RA. As part of this, Google has maintained operational policies, procedures for all the common processes, including but not limited to generation and CA keys, managing offline keys, generating and publishing CRLs, OCSP responses, and incident response. In the opinion letter you will find the following statement: ``` In our opinion, as of November 17, 2016, Google Trust Services LLC Management’s Assertion, as referred to above, is fairly stated, in all material respects, based on Certification Practices Statement Management Criterion 2.2, Asset Classification and Management Criterion 3.2, and Key Storage, Backup and Recovery Criterion 4.2 of the WebTrust Principles and Criteria for Certification Authorities v2.0. ``` It was both our hope and understanding that this is sufficient to concern here. As you have noted to me personally in the past you found it odd that Google generated a new GIAG2 regularly, this is a function of the way Symantec has structured our agreement with them. This required, as you might imagine, Google to have developed and maintained the frameworks to support this activity. If there are issues here with the scope of the opinion letter as worded, I would be happy to work with the auditors to clarify it. > pzb: One more item: The GTS CPS says that "Between 11 August 2016 and 8 > December 2016, Google Inc. operated these Roots according to Google > Inc.’s Certification Practice Statement." The latest Google Inc CPS I > could find is the one dated July 2016 ( http://static.googleusercontent.com/media/pki.google.com/en/GIAG2-CPS-1.4.pdf ). We determine we were not explicit enough in the original publication of the GTS CPS, for this reason we updated it ( http://static.googleusercontent.com/media/pki.goog/en//GTS-CPS-1.5.pdf) and it now states: “Prior to 11 August 2016, the Roots R2, R4, GTS Root R1, GTS Root R2, GTS Root R3 and GTS Root R4 were operated by GMO GlobalSign, Inc. according to GMO GlobalSign, Inc.’s Certificate Policy and Certification Practice Statement. Between 11 August 2016 and 8 December 2016, Google Inc. operated these Roots according to Google Inc.’s Certification Practice Statement. As of 9 December 2016, Google Trust Services LLC operates these Roots under Google Trust Services LLC’s Certificate Policy and Certification Practice Statement.” It is our hope this makes it clearer. > pzb The Google CPS says "The Google Internet Authority may issue > Subscriber Certificates only to the following organizations: Google > and Google Affiliates." > 4) Do you believe that this restriction is appropriate for roots in > the Mozilla program? This statement reflects the contractual limitation Symantec has on Google relative to GIAG2. To your question of applicability, I believe it is appropriate in this case. Google and Google Affiliates operate some of the most popular and frequented sites on the web, as part of that Google often hosts customer applications and content. As I understand it, the goal of the Mozilla root program is to enable sites just like these to offer their services over SSL. Enabling Google to do so for its own properties and its customers seems well within the intent of the program. > pzb: The Google CPS says it only covers Google Internet Authority G2. > 5) Is there a version of the CPS that covers the GS roots? After a review of the GTS CPS, it became clear we were not sufficiently clear when the transition from the GIAG2 CPS to the GTS CPS happened, as per above we have since made a text clarification we hope addresses this question: “Prior to 11 August 2016, the Roots R2, R4, GTS Root R1, GTS Root R2, GTS Root R3 and GTS Root R4 were operated by GMO GlobalSign, Inc. according to GMO GlobalSign, Inc.’s Certificate Policy and Certification Practice Statement. Between 11 August 2016 and 8 December 2016, Google Inc. operated these Roots according to Google Inc.’s Certification Practice Statement. As of 9 December 2016, Google Trust Services LLC operates these Roots under Google Trust Services LLC’s Certificate Policy and Certification Practice Statement.” Ryan Hurst Product Manager On Thu, Feb 9, 2017 at 2:55 PM, Peter Bowen <pzbo...@gmail.com> wrote: > Ryan, > > Thank you for the quick reply. My comments and questions are inline. > > On Thu, Feb 9, 2017 at 11:55 AM, Ryan Hurst via dev-security-policy > <dev-security-policy@lists.mozilla.org> wrote: > > Peter, > > > > Thank you very much for your, as always, thorough review. > > > > Let me start by saying I agree there is an opportunity for improving the > policies around how key transfers such your recent transfer and Google's > are handled. > > > > It is my hope we can, through our respective recent experiences > performing such transfers, help Mozilla revise their policy to provide > better guidance for such cases in the future. > > Where I see opportunities below, I'm marking them with "Policy Suggestion". > > > As for your specific questions, my responses follow: > > > > pzb: First, according to the GTS website, there is no audit using the > WebTrust Principles and Criteria for Certification Authorities – Extended > Validation SSL. However the two roots in the Mozilla CA program currently > are EV enabled and at least one subordinate CA under them is issuing EV > certificates. > > > > rmh: Prior to our final stage of the acquisition we contacted both > Mozilla and Microsoft about this particular situation. > > > > At this time, we do not have any interest in the issuance of EV SSL > certificates, however GlobalSign does. Based on our conversations with > representatives from both organizations we were told that since: > > - The EV OID associated with this permission is associated with > GlobalSign and not Google and, > > - GlobalSign is active member in good standing with the respective root > programs and, > > - Google will not be issuing EV SSL certificates, > > - Google will operate these roots under their own CP/CPS’s and > associated OIDs, > > - Google issuing a certificate with the GlobalSign OIDs would qualify as > miss-issuance. > > Mozilla recognizes 2.23.140.1.1 as being a valid OID for EV > certificates for all EV-enabled roots > (https://bugzilla.mozilla.org/show_bug.cgi?id=1243923). > > 1) Do you consider it mis-issuance for Google to issue a certificate > containing the 2.23.140.1.1 OID? > > Policy Suggestion A) When transferring a root that is EV enabled, it > should be clearly stated whether the recipient of the root is also > receiving the EV policy OID(s). > > > That it would be acceptable for us not to undergo a EV SSL audit, and > that GlobalSign could keep the EV right for the associated subordinate CA > for the remaining validity period to facilitate the transition (assuming > continued compliance). > > > > As a former manager of a root program, this seems an appropriate > position to take. And as one who has been involved in several such root > transfers I think differences in intended use are common enough that they > should be explicitly handled by policy. > > > > pzb: Second, according to the GTS CPS v1.3, "Between 11 August 2016 and > 8 December 2016, Google Inc. operated these Roots according to Google > Inc.’s Certification Practice Statement." The basic WebTrust for CA and > WebTrust BR audit reports for the period ending September 30, 2016 > explicitly state they are for "subordinate CA under external Root CA" and > do not list the roots in the GTS CPS at all. > > > > rmh: I believe this will be answered by my responses to your third and > fourth observations. > > It was not. > > 2) Will Google be publishing an audit report for a period starting 11 > August 2016 that covers the transferred GS roots? If so, can you > estimate the end of period date? > > > pzb: Third, the Google CPS says Google took control of these roots on > August 11, 2016. The Mozilla CA policy explicitly says that a bug report > must be filed to request to be included in the Mozilla CA program. It was > not until December 22, 2016 that Google requested inclusion as a CA in > Mozilla's CA program (https://bugzilla.mozilla.org/show_bug.cgi?id=1325532). > This does not appear to align with Mozilla requirements for public > disclosure. > > > > rmh: As has been mentioned, timing for a transaction like this is very > complicated. The process of identifying candidates that could meet our > needs took many months with several false starts with different > organizations. That said, prior to beginning this process we proactively > reached out to both Microsoft and Mozilla root programs to let them know we > were beginning the process. Once it looked like we would be able to come to > an agreement with GlobalSign we again reached out and notified both > programs of our intent to secure these specific keys. Then once the > transaction was signed we again notified the root programs that the deal > was done. > > > > As you know the process to ensure a secure, audited and well structured > key migration is also non-trivial. Once this migration was performed we > again notified both root programs. > > > > Our intention was to notify all parties, including the public, shortly > after the transfer but it took some time for our auditors, for reasons > unrelated to our audit, to produce the associated audit letters. > > > > Once we received said letters, we then filed the bugs. > > > > This is, although not our ideal timeline, and based on our > understanding, in compliance with the Mozilla root program in that since > these roots were already members they did not require us to publicly > disclose the above negotiation, contracting, planning, migration and other > intermediate steps. > > I think that this is the key issue. In my reading, "root > certificates" are not members of the program. Rather organizations > (legal entities) are members and each member has some number of root > certificates. > > Google was not a member of the program and had not applied to be a > member of the program at the time they received the roots already in > the program. This seems problematic to me. > > Policy Suggestion B) Require that any organization wishing to become a > member of the program submit a bug with links to content demonstrating > compliance with the Mozilla policy. Require that this be public prior > to taking control of any root in the program. > > Policy Suggestion C) Recognize that root transfers are distinct from > the acquisition of a program member. Acquisition of a program member > (meaning purchase of the company) is a distinctly different activity > from moving only a private key, as the prior business controls no > longer apply in the latter case. > > > pzb: Fourth, the audit reports linked in the bug explicitly set the > scope of "subordinate CA operated under external Root CA" and do not > include any indication of controls around the issuance of subordinate CA > certificates. These audit reports do not have an appropriate scope for a > root CA. > > > > rmh: Yes, we were also concerned about this topic, especially with the > recent scope issues with audits. As such, we discussed this with both our > auditors, and the the root programs prior to acquisition of the key > material. > > > > When looking at this issue it is important to keep in mind Google has > operated a WebTrust audited subordinate CA under Symantec for quite a long > time. As part of this they have maintained audited facilities, and > procedures appropriate for offline key management, CRL/OCSP generation, and > other related activities. Based on this, and the timing of both our audit, > and key transfer all parties concluded it would be sufficient to have the > auditors provide an opinion letter about the transfer of the keys and have > those keys covered by the subsequent annual audit. > > I don't see how having a subordinate issuing CA leads to the > conclusion that they have "facilities and procedures appropriate for > offline key management". Running an online and offline CA are two > rather different things and usually have different controls. > > Policy Suggestion D) Moving from being a RA to a CA or moving from > being a single tier/online (i.e. Subordinate-only) CA to being a > multi-tier/root CA requires a PITRA > > > We have provided these letters directly to the root programs and have > recently secured permission from the auditors to release them publicly (I > will add them to the bug). > > > > For those not familiar with the process, If Google had never been a > WebPKI CA, this situation would have been addressed with a pre-issuance > audit and a subsequent full audit 90 days later. > > > > Since Google is a long-time WebTrust audited CA and our audits and > acquisition were going to happen approximately at the same time, this would > have provided no new evidence to the root programs or the community. > > I disagree. There is no evidence of operational controls over > Subordinate CA Certificate Life Cycle Management. This is highlighted > by the fact that the neither the management assertion nor the opinion > list this topic at all. > > 3) Does Google have audited controls around offline operations and > Subordinate CA certificate issuance? > > One more item: The GTS CPS says that "Between 11 August 2016 and 8 > December 2016, Google Inc. operated these Roots according to Google > Inc.’s Certification Practice Statement." The latest Google Inc CPS I > could find is the one dated July 2016 > (http://static.googleusercontent.com/media/pki.google.com/en/GIAG2-CPS-1. > 4.pdf). > > The Google CPS says "The Google Internet Authority may issue > Subscriber Certificates only to the following organizations: Google > and Google Affiliates." > > 4) Do you believe that this restriction is appropriate for roots in > the Mozilla program? > > The Google CPS says it only covers Google Internet Authority G2. > > 5) Is there a version of the CPS that covers the GS roots? > > > The purpose of the audit is to provide assurances to the root program > and community that best practices are being followed and the relying > parties best interests are being met. In this case following the procedure > defined for an new CA would not have aided that goal. > > > > As an example, consider the case of a WebPKI trusted root with one key > already trusted, they can generate a new key and include it in its next > audit without the need to do the pre-issuance audit. > > I agree creation of a new CA under the existing audited controls does > not require a pre-issuance audit. However that is not applicable to > this case. > > > I think this is an appropriate position to take and an opportunity to > clarify the Mozilla root program to better inform similar cases in the > future. > > Thanks, > Peter > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy