On Tue, Jan 10, 2017 at 7:02 PM, Wayne Thayer <wtha...@godaddy.com> wrote:
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our 
> domain validation processing system. The bug that caused the issue was fixed 
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to 
> determine the scope of the problem, and identified 8850 certificates that 
> were issued without proper domain validation as a result of the bug. The 
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and 
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to 
> n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to 
> holiday vacations and the fact that the issue was not reported properly per 
> our CPS, we did not become aware of the issue until one of the employees 
> opened the email on Friday Jan 6th and promptly alerted management. The issue 
> was originally reported to Microsoft by one of their own customers and was 
> described as only affecting certificate requests when the DNS A record of the 
> domain was set to An investigation was initiated immediately and 
> within a few hours we determined that the problem was broader in scope. The 
> root cause of the problem was fixed via a code change at approximately 10 PM 
> MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on 
> July 29th, 2016 as part of a routine code change intended to improve our 
> certificate issuance process. The bug is related to our use of practical 
> demonstration of control to validate authority to receive a certificate for a 
> given fully-qualified domain name. In the problematic case, we provide a 
> random code to a customer and ask them to place it in a specific location on 
> their website. Our system automatically checks for the presence of that code 
> via an HTTP and/or HTTPS request to the website. If the code is found, the 
> domain control check is completed successfully.  Prior to the bug, the 
> library used to query the website and check for the code was configured to 
> return a failure if the HTTP status code was not 200 (success). A 
> configuration change to the library caused it to return results even when the 
> HTTP status code was not 200. Since many web servers are configured to 
> include the URL of the r
>  uest in the body of a 404 (not found) response, and the URL also contained 
> the random code, any web server configured this way caused domain control 
> verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to procure 
> a certificate for a domain that was not authorized. The customer who 
> discovered the bug revoked the certificate they obtained, and subsequent 
> certificates issued as the result of requests used for testing by Microsoft 
> and GoDaddy have been revoked. Further, any certificate requests made for 
> domains we flag as high-risk were also subjected to manual review (rather 
> than being issued purely based on an invalid domain authorization).
> We have re-verified domain control on every certificate issued using this 
> method of validation in the period from when the bug was introduced until it 
> was fixed. A list of 8850 potentially unverified certificates (representing 
> less than 2% of the total issued during the period) was compiled at 10 PM PST 
> on Monday Jan 9th. As mentioned above, potentially impacted certificates will 
> be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. 
> Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to 
> prevent the re-issuance of certificates using cached and potentially 
> unverified domain validation information. However, prior to identifying and 
> shutting down this path, an additional 101 certificates were reissued using 
> such cached and potentially unverified domain validation information, 
> resulting in an overall total of 8951 certificates that were issued without 
> proper domain validation as a result of the bug.
> Next Steps:
> While we are confident that we have completely resolved the problem, we are 
> watching our system closely to ensure that no more certificates are issued 
> without proper domain validation, and we will take immediate action and 
> report any further issues if found. A full post-mortem review of this 
> incident will occur and steps will be taken to prevent a recurrence, 
> including the addition of automated tests designed to detect this type of 
> scenario. If more information about the cause or impact of this incident 
> becomes available, we will publish updates to this report.
> Wayne Thayer
> GoDaddy


Thanks for sharing these details.

What's unclear is what steps GoDaddy has taken to remedy this.

For example:
1) Disabling domain control demonstrations through the use of a file on a server
2) Switching to /.well-known/pkivalidation
3) Ensuring that the random value is not part of the HTTP[S] request


Could you speak further to how GoDaddy has resolved this problem? My
hope is that it doesn't involve "Only look for 200 responses" =)
dev-security-policy mailing list

Reply via email to