On Tue, Jan 10, 2017 at 7:02 PM, Wayne Thayer <wtha...@godaddy.com> wrote: > Summary: > On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our > domain validation processing system. The bug that caused the issue was fixed > late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to > determine the scope of the problem, and identified 8850 certificates that > were issued without proper domain validation as a result of the bug. The > impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and > will also be logged to the Google Pilot CT log. > Detailed Description: > On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to > n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to > holiday vacations and the fact that the issue was not reported properly per > our CPS, we did not become aware of the issue until one of the employees > opened the email on Friday Jan 6th and promptly alerted management. The issue > was originally reported to Microsoft by one of their own customers and was > described as only affecting certificate requests when the DNS A record of the > domain was set to 127.0.0.1. An investigation was initiated immediately and > within a few hours we determined that the problem was broader in scope. The > root cause of the problem was fixed via a code change at approximately 10 PM > MST on Friday, Jan 6th. > On Saturday, January 7th, we determined that the bug was first introduced on > July 29th, 2016 as part of a routine code change intended to improve our > certificate issuance process. The bug is related to our use of practical > demonstration of control to validate authority to receive a certificate for a > given fully-qualified domain name. In the problematic case, we provide a > random code to a customer and ask them to place it in a specific location on > their website. Our system automatically checks for the presence of that code > via an HTTP and/or HTTPS request to the website. If the code is found, the > domain control check is completed successfully. Prior to the bug, the > library used to query the website and check for the code was configured to > return a failure if the HTTP status code was not 200 (success). A > configuration change to the library caused it to return results even when the > HTTP status code was not 200. Since many web servers are configured to > include the URL of the r eq > uest in the body of a 404 (not found) response, and the URL also contained > the random code, any web server configured this way caused domain control > verification to complete successfully. > We are currently unaware of any malicious exploitation of this bug to procure > a certificate for a domain that was not authorized. The customer who > discovered the bug revoked the certificate they obtained, and subsequent > certificates issued as the result of requests used for testing by Microsoft > and GoDaddy have been revoked. Further, any certificate requests made for > domains we flag as high-risk were also subjected to manual review (rather > than being issued purely based on an invalid domain authorization). > We have re-verified domain control on every certificate issued using this > method of validation in the period from when the bug was introduced until it > was fixed. A list of 8850 potentially unverified certificates (representing > less than 2% of the total issued during the period) was compiled at 10 PM PST > on Monday Jan 9th. As mentioned above, potentially impacted certificates will > be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. > Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to > prevent the re-issuance of certificates using cached and potentially > unverified domain validation information. However, prior to identifying and > shutting down this path, an additional 101 certificates were reissued using > such cached and potentially unverified domain validation information, > resulting in an overall total of 8951 certificates that were issued without > proper domain validation as a result of the bug. > Next Steps: > While we are confident that we have completely resolved the problem, we are > watching our system closely to ensure that no more certificates are issued > without proper domain validation, and we will take immediate action and > report any further issues if found. A full post-mortem review of this > incident will occur and steps will be taken to prevent a recurrence, > including the addition of automated tests designed to detect this type of > scenario. If more information about the cause or impact of this incident > becomes available, we will publish updates to this report. > Wayne Thayer > GoDaddy
Wayne, Thanks for sharing these details. What's unclear is what steps GoDaddy has taken to remedy this. For example: 1) Disabling domain control demonstrations through the use of a file on a server 2) Switching to /.well-known/pkivalidation 3) Ensuring that the random value is not part of the HTTP[S] request etc Could you speak further to how GoDaddy has resolved this problem? My hope is that it doesn't involve "Only look for 200 responses" =) _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
