On Tue, Oct 24, 2017 at 12:28 PM, Daymion Reynolds via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote:
> Godaddy LLC first became aware of possible ROCA vulnerability exposure on > Monday October 16th 2017 at 9:30am. The following are the steps we took for > detection, revocation, and the permanent fix of certificate provisioning: > • Monday October 16th 2017 AZ, first became aware of the ROCA > vulnerability. We downloaded and modified the open source detection tool > to audit 100% of the non-revoked and non-expired certs we had issued. > • Early am Wednesday October 18th AZ we had our complete list of 7 > certs with the ROCA defect. We verified the results and proceeded to start > the revocation process. While cert revocation was in progress we started > researching the long-term detection and prevention of the weak CSR > vulnerability. > • Early am Wednesday October 18th Rob Stradling released a list of > certs with the vulnerability. 2/7 we revoked were on the list. > https://misissued.com/batch/28/ > • Thursday October 19th by 2:02am AZ, we completed the 7 cert > revocations. Revocations included customer outreach to advise the customer > of the vulnerability. > • Thursday October 19th AZ, two CSRs were submitted for commonNames “ > scada2.emsglobal.net” & “scada.emsglobal.net” and were issued. Each > request had used the vulnerable keys for CSR generation. We revoked the > certs again on Thursday October 19th AZ. During this period, we reached out > to the customer to educate them regarding the vulnerability and informing > them they needed to generate a new keypair from an unimpacted device. > Customer was unreachable. Friday October 20thAZ, another cert was issued > for commonName “scada.emsglobal.net” using a CSR generated with a weak > key. We then took measures to prevent future certs from being issued to the > same common name and revoked the cert on October 20th 2017 AZ. > commonName crt.sh-link > scada.emsglobal.net https://crt.sh/?id=3084867 > > scada.emsglobal.net https://crt.sh/?id=238721704 > > scada.emsglobal.net https://crt.sh/?id=238721807 > > scada2.emsglobal.net https://crt.sh/?id=238720969 > > scada2.emsglobal.net https://crt.sh/?id=238721559 > > • Saturday October 21st 2017 AZ & Sunday October 22nd 2017 AZ, we > scanned our cert store and identified 0 vulnerable certs. > • Monday October 23, 2017 AZ, we have deployed a permanent fix to > prevent future CSRs generated using weak keys from being submitted. Post > scanning of the environment concluded 0 certificates at risk. > > Below is a complete list of certs under GoDaddy management impacted by > this vulnerability. > > Alias crt.sh-link > alarms.realtimeautomation.net https://crt.sh/?id=33966207 > > scada.emsglobal.net https://crt.sh/?id=3084867 > https://crt.sh/?id=238721704 > https://crt.sh/?id=238721807 > > www.essicorp-scada.com https://crt.sh/?id=238720405 > > marlboro.bonavistaenergy.com https://crt.sh/?id=238720743 > > scada2.emsglobal.net https://crt.sh/?id=238720969 > https://crt.sh/?id=238721559 > > www.jointboardclearscada.com https://crt.sh/?id=238721242 > > *.forgenergy.com https://crt.sh/?id=238721435 > > Daymion, Thanks for providing this detailed report. I want to especially thank you for providing an actual timeline - so many CAs have unfortunately misunderstood what a timeline means, or how to effectively communicate it. Your timeline provides a useful description of pre-existing state, when the issue or incident was introduced, when it was detected, what steps were taken initially, when the issue was resolved, and what steps will be taken in the future. In looking at what the expectations of CAs are, and how well GoDaddy upheld them, the specific view is that: - With the disclosure of the ROCA vulnerability, private keys subject to it are noted to have suffered a Key Compromise event (BRs 1.5.1, Section 1.6.1, "Key Compromise") - CAs are required to revoke a certificate within 24 hours if "The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate has suffered a Key compromise" (BRs 1.5.1, 4.9.1.1, "Reasons for Revoking a Subscriber Certificate", Item 3) - CAs are required to reject CSRs if the private key does not meet the requirements set forth in Sections 6.1.5, 6.1.6, or if it has a known-weak Private Key (BRs 1.5.1, Section 6.1.1.3, "Subscriber Key Pair Generation") Looking at the timing, it looks like: - ~36 hours to detecting certificates - ~60 hours to revoke - ~60 hours to set up initial CSR rejection - ~1 week to setup full scanning/rejection That said, the level of detail provided - and the many challenges a number of folks encountered with the initial ROCA code (including seemingly some obfuscation by the authors) - arguably establish that this is a reasonable and understandable timeline. This further is in line with some of the discussions proceeding in the CA/Browser Forum with respect to revocation timelines and transparency. While the BRs presently provide only 24 hours for CAs to fully implement revocation (and 0 hours to implement CSR rejection), truly exceptional cases such as this are arguably justifiable. I think that, with respect to CA incident reporting, the Mozilla Module Peers should consider this closed. I defer to Gerv and Kathleen whether or not the deviations in timeline warrant filing a bug to ensure tracking, but I do not think this report should be seen as looking unfavorably on GoDaddy. In particular, it's worth noting that a number of CAs - Symantec/GeoTrust, GlobalSign, and Symantec/VeriSign still have unrevoked certificates out there despite awareness of this issue. _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
