Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

On Tue, Jan 10, 2017 at 7:02 PM, Wayne Thayer  wrote:
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our 
> domain validation processing system. The bug that caused the issue was fixed 
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to 
> determine the scope of the problem, and identified 8850 certificates that 
> were issued without proper domain validation as a result of the bug. The 
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and 
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to 
> n...@godaddy.com and two GoDaddy employees. Due to 
> holiday vacations and the fact that the issue was not reported properly per 
> our CPS, we did not become aware of the issue until one of the employees 
> opened the email on Friday Jan 6th and promptly alerted management. The issue 
> was originally reported to Microsoft by one of their own customers and was 
> described as only affecting certificate requests when the DNS A record of the 
> domain was set to 127.0.0.1. An investigation was initiated immediately and 
> within a few hours we determined that the problem was broader in scope. The 
> root cause of the problem was fixed via a code change at approximately 10 PM 
> MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on 
> July 29th, 2016 as part of a routine code change intended to improve our 
> certificate issuance process. The bug is related to our use of practical 
> demonstration of control to validate authority to receive a certificate for a 
> given fully-qualified domain name. In the problematic case, we provide a 
> random code to a customer and ask them to place it in a specific location on 
> their website. Our system automatically checks for the presence of that code 
> via an HTTP and/or HTTPS request to the website. If the code is found, the 
> domain control check is completed successfully.  Prior to the bug, the 
> library used to query the website and check for the code was configured to 
> return a failure if the HTTP status code was not 200 (success). A 
> configuration change to the library caused it to return results even when the 
> HTTP status code was not 200. Since many web servers are configured to 
> include the URL of the r
 eq
>  uest in the body of a 404 (not found) response, and the URL also contained 
> the random code, any web server configured this way caused domain control 
> verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to procure 
> a certificate for a domain that was not authorized. The customer who 
> discovered the bug revoked the certificate they obtained, and subsequent 
> certificates issued as the result of requests used for testing by Microsoft 
> and GoDaddy have been revoked. Further, any certificate requests made for 
> domains we flag as high-risk were also subjected to manual review (rather 
> than being issued purely based on an invalid domain authorization).
> We have re-verified domain control on every certificate issued using this 
> method of validation in the period from when the bug was introduced until it 
> was fixed. A list of 8850 potentially unverified certificates (representing 
> less than 2% of the total issued during the period) was compiled at 10 PM PST 
> on Monday Jan 9th. As mentioned above, potentially impacted certificates will 
> be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. 
> Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to 
> prevent the re-issuance of certificates using cached and potentially 
> unverified domain validation information. However, prior to identifying and 
> shutting down this path, an additional 101 certificates were reissued using 
> such cached and potentially unverified domain validation information, 
> resulting in an overall total of 8951 certificates that were issued without 
> proper domain validation as a result of the bug.
> Next Steps:
> While we are confident that we have completely resolved the problem, we are 
> watching our system closely to ensure that no more certificates are issued 
> without proper domain validation, and we will take immediate action and 
> report any further issues if found. A full post-mortem review of this 
> incident will occur and steps will be taken to prevent a recurrence, 
> including the addition of automated tests designed to detect this type of 
> scenario. If more information about the cause or impact of this incident 
> becomes available, we will publish updates to this report.
> Wayne Thayer
> GoDaddy

Wayne,

Thanks for sharing these details.

What's unclear is what steps GoDaddy has taken to remedy this.

For example:
1) Disabling domain control demonstrations through 

Incident Report – Certificates issued without proper domain validation

[Wayne Thayer]

Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our 
domain validation processing system. The bug that caused the issue was fixed 
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to 
determine the scope of the problem, and identified 8850 certificates that were 
issued without proper domain validation as a result of the bug. The impacted 
certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and will also 
be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to 
n...@godaddy.com and two GoDaddy employees. Due to 
holiday vacations and the fact that the issue was not reported properly per our 
CPS, we did not become aware of the issue until one of the employees opened the 
email on Friday Jan 6th and promptly alerted management. The issue was 
originally reported to Microsoft by one of their own customers and was 
described as only affecting certificate requests when the DNS A record of the 
domain was set to 127.0.0.1. An investigation was initiated immediately and 
within a few hours we determined that the problem was broader in scope. The 
root cause of the problem was fixed via a code change at approximately 10 PM 
MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on 
July 29th, 2016 as part of a routine code change intended to improve our 
certificate issuance process. The bug is related to our use of practical 
demonstration of control to validate authority to receive a certificate for a 
given fully-qualified domain name. In the problematic case, we provide a random 
code to a customer and ask them to place it in a specific location on their 
website. Our system automatically checks for the presence of that code via an 
HTTP and/or HTTPS request to the website. If the code is found, the domain 
control check is completed successfully.  Prior to the bug, the library used to 
query the website and check for the code was configured to return a failure if 
the HTTP status code was not 200 (success). A configuration change to the 
library caused it to return results even when the HTTP status code was not 200. 
Since many web servers are configured to include the URL of the req
 uest in the body of a 404 (not found) response, and the URL also contained the 
random code, any web server configured this way caused domain control 
verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to procure a 
certificate for a domain that was not authorized. The customer who discovered 
the bug revoked the certificate they obtained, and subsequent certificates 
issued as the result of requests used for testing by Microsoft and GoDaddy have 
been revoked. Further, any certificate requests made for domains we flag as 
high-risk were also subjected to manual review (rather than being issued purely 
based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this 
method of validation in the period from when the bug was introduced until it 
was fixed. A list of 8850 potentially unverified certificates (representing 
less than 2% of the total issued during the period) was compiled at 10 PM PST 
on Monday Jan 9th. As mentioned above, potentially impacted certificates will 
be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. 
Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to 
prevent the re-issuance of certificates using cached and potentially unverified 
domain validation information. However, prior to identifying and shutting down 
this path, an additional 101 certificates were reissued using such cached and 
potentially unverified domain validation information, resulting in an overall 
total of 8951 certificates that were issued without proper domain validation as 
a result of the bug.
Next Steps:
While we are confident that we have completely resolved the problem, we are 
watching our system closely to ensure that no more certificates are issued 
without proper domain validation, and we will take immediate action and report 
any further issues if found. A full post-mortem review of this incident will 
occur and steps will be taken to prevent a recurrence, including the addition 
of automated tests designed to detect this type of scenario. If more 
information about the cause or impact of this incident becomes available, we 
will publish updates to this report.
Wayne Thayer
GoDaddy
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Update on transition of the Verizon roots and issuance of SHA1 certificates

[Erwann Abalea]

Bonjour,

Le lundi 9 janvier 2017 18:02:57 UTC+1, Jeremy Rowley a écrit :
> Not many websites, but all of the Belgium ID cards would end up being
> revoked. 

Not exactly. The "Belgium Root CAx" CA certificates issued by Cybertrust would 
be revoked, but since these CAs also have self-signed certificates, Belgium ID 
cards will still have a valid chain up to these self-signed "Belgium Root CAx" 
certificates.

> Although Belgium is only issuing client certs, the issuing CA is not
> technically constrained, meaning a BR, Network security, and standard
> WebTrust audit is required. We are currently waiting for the results of the
> audit report.

And maybe the opinion report from a Qualified Auditor regarding the private key 
generation of these subordinate CAs.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy