Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
domain validation processing system. The bug that caused the issue was fixed
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
determine the scope of the problem, and identified 8850 certificates that were
issued without proper domain validation as a result of the bug. The impacted
certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and will also
be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to
holiday vacations and the fact that the issue was not reported properly per our
CPS, we did not become aware of the issue until one of the employees opened the
email on Friday Jan 6th and promptly alerted management. The issue was
originally reported to Microsoft by one of their own customers and was
described as only affecting certificate requests when the DNS A record of the
domain was set to 127.0.0.1. An investigation was initiated immediately and
within a few hours we determined that the problem was broader in scope. The
root cause of the problem was fixed via a code change at approximately 10 PM
MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on
July 29th, 2016 as part of a routine code change intended to improve our
certificate issuance process. The bug is related to our use of practical
demonstration of control to validate authority to receive a certificate for a
given fully-qualified domain name. In the problematic case, we provide a random
code to a customer and ask them to place it in a specific location on their
website. Our system automatically checks for the presence of that code via an
HTTP and/or HTTPS request to the website. If the code is found, the domain
control check is completed successfully. Prior to the bug, the library used to
query the website and check for the code was configured to return a failure if
the HTTP status code was not 200 (success). A configuration change to the
library caused it to return results even when the HTTP status code was not 200.
Since many web servers are configured to include the URL of the req
uest in the body of a 404 (not found) response, and the URL also contained the
random code, any web server configured this way caused domain control
verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to procure a
certificate for a domain that was not authorized. The customer who discovered
the bug revoked the certificate they obtained, and subsequent certificates
issued as the result of requests used for testing by Microsoft and GoDaddy have
been revoked. Further, any certificate requests made for domains we flag as
high-risk were also subjected to manual review (rather than being issued purely
based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this
method of validation in the period from when the bug was introduced until it
was fixed. A list of 8850 potentially unverified certificates (representing
less than 2% of the total issued during the period) was compiled at 10 PM PST
on Monday Jan 9th. As mentioned above, potentially impacted certificates will
be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log.
Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to
prevent the re-issuance of certificates using cached and potentially unverified
domain validation information. However, prior to identifying and shutting down
this path, an additional 101 certificates were reissued using such cached and
potentially unverified domain validation information, resulting in an overall
total of 8951 certificates that were issued without proper domain validation as
a result of the bug.
Next Steps:
While we are confident that we have completely resolved the problem, we are
watching our system closely to ensure that no more certificates are issued
without proper domain validation, and we will take immediate action and report
any further issues if found. A full post-mortem review of this incident will
occur and steps will be taken to prevent a recurrence, including the addition
of automated tests designed to detect this type of scenario. If more
information about the cause or impact of this incident becomes available, we
will publish updates to this report.
Wayne Thayer
GoDaddy
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
