On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our 
domain validation processing system. The bug that caused the issue was fixed 
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to 
determine the scope of the problem, and identified 8850 certificates that were 
issued without proper domain validation as a result of the bug. The impacted 
certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and will also 
be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to<> and two GoDaddy employees. Due to 
holiday vacations and the fact that the issue was not reported properly per our 
CPS, we did not become aware of the issue until one of the employees opened the 
email on Friday Jan 6th and promptly alerted management. The issue was 
originally reported to Microsoft by one of their own customers and was 
described as only affecting certificate requests when the DNS A record of the 
domain was set to An investigation was initiated immediately and 
within a few hours we determined that the problem was broader in scope. The 
root cause of the problem was fixed via a code change at approximately 10 PM 
MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on 
July 29th, 2016 as part of a routine code change intended to improve our 
certificate issuance process. The bug is related to our use of practical 
demonstration of control to validate authority to receive a certificate for a 
given fully-qualified domain name. In the problematic case, we provide a random 
code to a customer and ask them to place it in a specific location on their 
website. Our system automatically checks for the presence of that code via an 
HTTP and/or HTTPS request to the website. If the code is found, the domain 
control check is completed successfully.  Prior to the bug, the library used to 
query the website and check for the code was configured to return a failure if 
the HTTP status code was not 200 (success). A configuration change to the 
library caused it to return results even when the HTTP status code was not 200. 
Since many web servers are configured to include the URL of the req
 uest in the body of a 404 (not found) response, and the URL also contained the 
random code, any web server configured this way caused domain control 
verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to procure a 
certificate for a domain that was not authorized. The customer who discovered 
the bug revoked the certificate they obtained, and subsequent certificates 
issued as the result of requests used for testing by Microsoft and GoDaddy have 
been revoked. Further, any certificate requests made for domains we flag as 
high-risk were also subjected to manual review (rather than being issued purely 
based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this 
method of validation in the period from when the bug was introduced until it 
was fixed. A list of 8850 potentially unverified certificates (representing 
less than 2% of the total issued during the period) was compiled at 10 PM PST 
on Monday Jan 9th. As mentioned above, potentially impacted certificates will 
be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. 
Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to 
prevent the re-issuance of certificates using cached and potentially unverified 
domain validation information. However, prior to identifying and shutting down 
this path, an additional 101 certificates were reissued using such cached and 
potentially unverified domain validation information, resulting in an overall 
total of 8951 certificates that were issued without proper domain validation as 
a result of the bug.
Next Steps:
While we are confident that we have completely resolved the problem, we are 
watching our system closely to ensure that no more certificates are issued 
without proper domain validation, and we will take immediate action and report 
any further issues if found. A full post-mortem review of this incident will 
occur and steps will be taken to prevent a recurrence, including the addition 
of automated tests designed to detect this type of scenario. If more 
information about the cause or impact of this incident becomes available, we 
will publish updates to this report.
Wayne Thayer
dev-security-policy mailing list

Reply via email to