Re: Incident Report – Certificates issued without proper domain validation

[kanepyork]

On Thursday, January 19, 2017 at 6:05:22 PM UTC-8, Jakob Bohm wrote:
> On 20/01/2017 00:35, Nick Lamb wrote:
> > On Thursday, 19 January 2017 20:20:24 UTC, Jakob Bohm  wrote:
> >> Google's CT initiative in its current form has serious privacy problems
> >> for genuine certificate holders.  I applaud any well-run CA that stands
> >> up to this attack on the Internet at large.
> >
> > I notice that you have not specifically identified which Certificate 
> > Authorities you believe are "well-run", perhaps your argument would have 
> > more force if you could name some market leaders in that category.
> >
> 
> Presumably most that haven't been distrusted by Mozilla or otherwise
> publicly shamed for massive misissuance.
You presume a lot, I fear. "No disasters yet" is not proof of disaster 
prevention.

Apologies for the late reply and this diversion from the thread, but I felt I 
should respond to some of this.

> > As a Relying Party for the Web PKI I think Google's initiative makes a 
> > sensible trade off, you can't have privacy while also delivering oversight. 
> > The public CAs are clearly in need of oversight. This did not happen in a 
> > vacuum but as a consequence of trusted Certificate Authorities exhibiting 
> > incompetence and greed over many years.
> >
> 
> As both a relying party and a certificate holder, I see no reason for
> public listing of most of the details in the CT logs, and I do take
> specific measures to not get public certificates for a number of things
> (such as my e-mail addresses) that I don't want listed in Google
> searches or attacked by spammers.
> 
> So far, I have not seen any good uses for CT logging stuff such as:
> 
> - (list cut)

I don't really see the point of putting a Citizen ID number in a website 
certificate, either. If you don't put it in the certificate, it doesn't need to 
be logged.

Remember that CT as specified is only for WebPKI website validation 
certificates, not S/MIME certificates (though I suspect you could submit them).

> What is really needed for most non-malicious CT uses are the relevant
> 2nd/3rd level domain (1 level below public suffix), country, state and
> organization names (or CN if not an internet name and no O name part),
> slow one way hash of full name entries (e.g.
> SHA-512**65536("someu...@gmail.com"),
> full issuer details, cryptographic algorithm and strength, plus serial
> number and technical details such as EKUs and other non-special cased
> items.

This has enormous complexity requirements above "log the full certificate". 
History shows us this would result in even less adoption than we have right now.

> For example, to check if someone issued a fraudulent certificate for
> any domain or address under google.com, Google Inc could check the list
> of redacted CT entries for *@*.google.com, then compare it against an
> in-house non-public database of such certificates authorized by their
> internal management procedures.
>
> To check for certificates issued to non-existent / suspect domains such
> as example.com and/or test[1-9].com (recent Symantec related post in
> this group), this would still be fully visible too.  So would SHA-1
> certificates issued in 2016, duplicate serial numbers, etc.  Someone
> getting a misissued wildcard cert for a semi-public suffix such as
> "sf.net" or "blogblog.com" would also show up.

Disregarding the issue of S/MIME certificates, what do you propose for Honest 
Achmed's Car Dealership, whom do not need nor have rigorous management 
procedures for SSL certificates? (Replying to the first 3 items in your list 
here.)

Hypothetical: A monitoring service says "We saw a certificate logged for 
.honestachmedcars.com. It came from [CA name], SHA256, serial number 
xx."

  Achmed asks around the office and nobody's quite sure why this certificate 
was created. It would help a lot if the monitor delivered the full list of 
subject names for the certificate, because if it did they would've seen it was 
issued for mail.honestachmedcars.com, which would point them to questioning 
their managed e-mail provider (who just did a scheduled renewal of all customer 
certificates).

> 
> For a service such as gmail.com, the information suggested above would
> allow someone knowing a specific e-mail address such as
> "someu...@gmail.com" to check if a certificate was misissued for that
> address, but would not provide an easy way for a third party (such as a
> spammer) to extract a list of all @gmail.com user names that happen to
> have S/MIME certificates (Of cause Google has the original list of
> their users already, but no one else should).
Again, disregarding this because CT was never meant for e-mail certificates.

~Kane York
Speaking as an individual.

> 
> Enjoy
> 
> Jakob
> -- 
> Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
> Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> 

Re: Incident Report – Certificates issued without proper domain validation

[Jakob Bohm]

On 20/01/2017 00:35, Nick Lamb wrote:

On Thursday, 19 January 2017 20:20:24 UTC, Jakob Bohm  wrote:

Google's CT initiative in its current form has serious privacy problems
for genuine certificate holders.  I applaud any well-run CA that stands
up to this attack on the Internet at large.


I notice that you have not specifically identified which Certificate Authorities you 
believe are "well-run", perhaps your argument would have more force if you 
could name some market leaders in that category.



Presumably most that haven't been distrusted by Mozilla or otherwise
publicly shamed for massive misissuance.


As a Relying Party for the Web PKI I think Google's initiative makes a sensible 
trade off, you can't have privacy while also delivering oversight. The public 
CAs are clearly in need of oversight. This did not happen in a vacuum but as a 
consequence of trusted Certificate Authorities exhibiting incompetence and 
greed over many years.



As both a relying party and a certificate holder, I see no reason for
public listing of most of the details in the CT logs, and I do take
specific measures to not get public certificates for a number of things
(such as my e-mail addresses) that I don't want listed in Google
searches or attacked by spammers.

So far, I have not seen any good uses for CT logging stuff such as:

- Full domain name (below public suffix + 1) for things like employee
 /contractor portals.
- Full domain name (below public suffix + 1) for alpha / beta tests,
 staging servers etc.
- Full domain name (below public suffic + 1) for CDN / cluster names,
 such as r2---sn-op5oxu-j2il.googlevideo.com
- E-mail addresses other than the RFC2142 special ones.
- City and street address.
- Telephone number.
- Citizens ID/Social security number/Company registration ID.

What is really needed for most non-malicious CT uses are the relevant
2nd/3rd level domain (1 level below public suffix), country, state and
organization names (or CN if not an internet name and no O name part),
slow one way hash of full name entries (e.g.
SHA-512**65536("someu...@gmail.com"),
full issuer details, cryptographic algorithm and strength, plus serial
number and technical details such as EKUs and other non-special cased
items.

For example, to check if someone issued a fraudulent certificate for
any domain or address under google.com, Google Inc could check the list
of redacted CT entries for *@*.google.com, then compare it against an
in-house non-public database of such certificates authorized by their
internal management procedures.

To check for certificates issued to non-existent / suspect domains such
as example.com and/or test[1-9].com (recent Symantec related post in
this group), this would still be fully visible too.  So would SHA-1
certificates issued in 2016, duplicate serial numbers, etc.  Someone
getting a misissued wildcard cert for a semi-public suffix such as
"sf.net" or "blogblog.com" would also show up.

For a service such as gmail.com, the information suggested above would
allow someone knowing a specific e-mail address such as
"someu...@gmail.com" to check if a certificate was misissued for that
address, but would not provide an easy way for a third party (such as a
spammer) to extract a list of all @gmail.com user names that happen to
have S/MIME certificates (Of cause Google has the original list of
their users already, but no one else should).


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Nick Lamb]

On Thursday, 19 January 2017 20:20:24 UTC, Jakob Bohm  wrote:
> Google's CT initiative in its current form has serious privacy problems
> for genuine certificate holders.  I applaud any well-run CA that stands
> up to this attack on the Internet at large.

I notice that you have not specifically identified which Certificate 
Authorities you believe are "well-run", perhaps your argument would have more 
force if you could name some market leaders in that category.

As a Relying Party for the Web PKI I think Google's initiative makes a sensible 
trade off, you can't have privacy while also delivering oversight. The public 
CAs are clearly in need of oversight. This did not happen in a vacuum but as a 
consequence of trusted Certificate Authorities exhibiting incompetence and 
greed over many years.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Jakob Bohm]

On 19/01/2017 01:33, montel.bahn...@gmail.com wrote:

On Thursday, January 12, 2017 at 7:38:47 PM UTC-5, Itzhak Daniel wrote:

Why not posting _ALL_ certificates issues via that method to CT log?


We had to nag and whine for a year to get IXSystems and FreeNAS folks to 
finally, begrudgingly use TLS (for Download of ISOs and SHA256 no less!). The 
'Volunteers' and staff deleted my posts, accused me of trolling and stated that 
the CAs' system was something like bunk or a laughing stock. Though not a 
commiter or security guru, I submit that:

If a CA refuses to take advantage of Google's Certificate Transparency 
Project or otherwise public log per RFC 6962, then Mozilla MUST shun them!



Google's CT initiative in its current form has serious privacy problems
for genuine certificate holders.  I applaud any well-run CA that stands
up to this attack on the Internet at large.


I mean who dares disagree? Surely this is a non-partisan issue with Mozilla 
Devs AND majority of Firefox Users? Let's keep on topic of GoDaddy's second 
insufficiency, though it's not alone on the consensus naughty-list. I assume 
some relevant browser Devs were shown proof of what happened in detail? Can 
they complain their spaghetti code is that proprietary, really. It surely is 
not valuable now as a work product. Just sign NDAs if they won't the bother. 
The 'lapses' WILL keep getting more convoluted and ridiculous if Mozilla, 
Google et al. don't finally draw the line.



I have no reason to believe Mozilla employees have any relevant GoDaddy
information not posted right here on this newsgroup and the associated
public web pages, bug trackers etc.

This newsgroup is *the* place where Mozilla finds out these things.
you and I are essentially standing inside the room where all this is
happening, seeing and hearing almost everything that goes on, and even
getting to contribute our opinions.



PS: FreeNAS is still using GoDadddy, even though they have other valid 
certificates per:
https://www.google.com/transparencyreport/https/ct/


Not at all relevant to this newsgroup.


...somebody has to lead by example and soon!



Hopefully not you.


Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[montel . bahniii]

On Thursday, January 12, 2017 at 7:38:47 PM UTC-5, Itzhak Daniel wrote:
> Why not posting _ALL_ certificates issues via that method to CT log?

We had to nag and whine for a year to get IXSystems and FreeNAS folks to 
finally, begrudgingly use TLS (for Download of ISOs and SHA256 no less!). The 
'Volunteers' and staff deleted my posts, accused me of trolling and stated that 
the CAs' system was something like bunk or a laughing stock. Though not a 
commiter or security guru, I submit that:

If a CA refuses to take advantage of Google's Certificate Transparency 
Project or otherwise public log per RFC 6962, then Mozilla MUST shun them!

I mean who dares disagree? Surely this is a non-partisan issue with Mozilla 
Devs AND majority of Firefox Users? Let's keep on topic of GoDaddy's second 
insufficiency, though it's not alone on the consensus naughty-list. I assume 
some relevant browser Devs were shown proof of what happened in detail? Can 
they complain their spaghetti code is that proprietary, really. It surely is 
not valuable now as a work product. Just sign NDAs if they won't the bother. 
The 'lapses' WILL keep getting more convoluted and ridiculous if Mozilla, 
Google et al. don't finally draw the line.

PS: FreeNAS is still using GoDadddy, even though they have other valid 
certificates per:
https://www.google.com/transparencyreport/https/ct/
...somebody has to lead by example and soon!
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Itzhak Daniel]

On Wednesday, January 11, 2017 at 5:03:08 AM UTC+2, Wayne Thayer wrote:
> ... and will also be logged to the Google Pilot CT log.

Why not posting _ALL_ certificates issues via that method to CT log?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

RE: Incident Report – Certificates issued without proper domain validation

[Wayne Thayer]

> From: Gervase Markham [mailto:g...@mozilla.org]
> Sent: Thursday, January 12, 2017 3:07 AM
> To: Wayne Thayer <wtha...@godaddy.com>; mozilla-dev-security-
> pol...@lists.mozilla.org
> Subject: Re: Incident Report – Certificates issued without proper domain
> validation
> 
> Hi Wayne,
> 
> Thanks for these prompt and detailed responses.
> 
> On 12/01/17 00:27, Wayne Thayer wrote:
> > Our initial response as reported yesterday was to fix the bug
> > introduced in July. Based on internal discussions and comments here,
> > as of 12 midnight PST last night (1/11) we stopped using this method
> > of file based domain control validation.
> 
> That seems like an excellent idea, at least until you can alter the system to
> make it so that the random value is not part of the URL requested.
> 
> > As soon as we learned of this issue, we went through every certificate
> > that was validated with the HTML method utilized during this period
> > and attempted to verify. If it could not be immediately verified, we
> > revoked the certificate.
> 
> That seems like an excellent process.
> 
> > When we learned of this issue, we re-validated every affected
> > certificate. If we were unable to properly validate, we revoked the
> > certificate. That is how we got the total of 8,951 revoked
> > certificates.
> 
> Are you able to say how many certificates were successfully revalidated?
> 
Approximately 7500. In addition, as of earlier today, new certificates that 
cover over 50% of the CNs in the set of revoked certs have successfully gone 
through our domain validation process.
>
> > As soon as we discovered the bug, we ran a report to identify every
> > certificate that didn’t fail the domain validation check during the
> > period the bug was active. We then started scanning websites to see
> > which ones were able to re-pass the proper validation check. If they
> > passed, we removed the certificate from the list. If we were unable to
> > revalidate the certificate, we revoked it. If there was any question
> > if the certificate was properly verified, we revoked it.
> 
> So you re-validated pretty much everything? Wow. That must be a lot of
> sites.
> 
> Not a requirement or a command, but it may be wise to improve your
> logging, because if you had stored the website's response and status code
> verbatim, you would not have needed to revalidate as many certificates
> (because you could have skipped those that responded "200"
> first time), and may have been able to revoke far fewer.
> 
Clearly this is good advice, thank you.
>
> Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Gervase Markham]

Hi Wayne,

Thanks for these prompt and detailed responses.

On 12/01/17 00:27, Wayne Thayer wrote:
> Our initial response as reported yesterday was to fix the bug
> introduced in July. Based on internal discussions and comments here,
> as of 12 midnight PST last night (1/11) we stopped using this method
> of file based domain control validation.

That seems like an excellent idea, at least until you can alter the
system to make it so that the random value is not part of the URL requested.

> As soon as we learned of this issue, we went through every
> certificate that was validated with the HTML method utilized during
> this period and attempted to verify. If it could not be immediately
> verified, we revoked the certificate.

That seems like an excellent process.

> When we learned of this issue, we re-validated every affected
> certificate. If we were unable to properly validate, we revoked the
> certificate. That is how we got the total of 8,951 revoked
> certificates.

Are you able to say how many certificates were successfully revalidated?

> As soon as we discovered the bug, we ran a report to identify every
> certificate that didn’t fail the domain validation check during the
> period the bug was active. We then started scanning websites to see
> which ones were able to re-pass the proper validation check. If they
> passed, we removed the certificate from the list. If we were unable
> to revalidate the certificate, we revoked it. If there was any
> question if the certificate was properly verified, we revoked it.

So you re-validated pretty much everything? Wow. That must be a lot of
sites.

Not a requirement or a command, but it may be wise to improve your
logging, because if you had stored the website's response and status
code verbatim, you would not have needed to revalidate as many
certificates (because you could have skipped those that responded "200"
first time), and may have been able to revoke far fewer.

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Yuhong Bao]

In this case, Nest's 404 page happens not to include the original URL in the 
HTML so they are not affected, but you see what I mean now.

From: Ryan Sleevi <r...@sleevi.com>
Sent: Wednesday, January 11, 2017 6:41:46 PM
To: Yuhong Bao
Cc: Richard Wang; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
Sleevi
Subject: Re: Incident Report – Certificates issued without proper domain 
validation

Hi Yuhong,

Perhaps it be best if you create a separate thread for your question -
it's not really clear at all how it relates to the topic at hand.

On Wed, Jan 11, 2017 at 6:08 PM, Yuhong Bao <yuhongbao_...@hotmail.com> wrote:
> That is what the current certificate by Google Internet Authority says.
> What I am referring to is that before Google bought Nest they used GoDaddy as 
> the CA.
> 
> From: Richard Wang <rich...@wosign.com>
> Sent: Wednesday, January 11, 2017 5:01:08 PM
> To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
> Sleevi
> Subject: RE: Incident Report – Certificates issued without proper domain 
> validation
>
> The nest.com certificate subject is:
> CN = www.nest.com
> O = Google Inc
> L = Mountain View
> S = California
> C = US
>
> This means this website owned by Google Inc. Right?
>
>
> Best Regards,
>
> Richard
>
> -Original Message-
> From: dev-security-policy
> [mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
> Behalf Of Yuhong Bao
> Sent: Thursday, January 12, 2017 6:12 AM
> To: Wayne Thayer <wtha...@godaddy.com>;
> dev-security-policy@lists.mozilla.org; Ryan Sleevi <r...@sleevi.com>
> Subject: Re: Incident Report - Certificates issued without proper domain
> validation
>
> I wonder if nest.com is now considered high-risk now. They recently switched
> from GoDaddy to Google Internet Authority.
> 
> From: dev-security-policy
> <dev-security-policy-bounces+yuhongbao_386=hotmail@lists.mozilla.org> on
> behalf of Wayne Thayer <wtha...@godaddy.com>
> Sent: Tuesday, January 10, 2017 7:02:28 PM
> To: dev-security-policy@lists.mozilla.org
> Subject: Incident Report - Certificates issued without proper domain
> validation
>
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
> domain validation processing system. The bug that caused the issue was fixed
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
> determine the scope of the problem, and identified 8850 certificates that
> were issued without proper domain validation as a result of the bug. The
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
> n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to
> holiday vacations and the fact that the issue was not reported properly per
> our CPS, we did not become aware of the issue until one of the employees
> opened the email on Friday Jan 6th and promptly alerted management. The
> issue was originally reported to Microsoft by one of their own customers and
> was described as only affecting certificate requests when the DNS A record
> of the domain was set to 127.0.0.1. An investigation was initiated
> immediately and within a few hours we determined that the problem was
> broader in scope. The root cause of the problem was fixed via a code change
> at approximately 10 PM MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on
> July 29th, 2016 as part of a routine code change intended to improve our
> certificate issuance process. The bug is related to our use of practical
> demonstration of control to validate authority to receive a certificate for
> a given fully-qualified domain name. In the problematic case, we provide a
> random code to a customer and ask them to place it in a specific location on
> their website. Our system automatically checks for the presence of that code
> via an HTTP and/or HTTPS request to the website. If the code is found, the
> domain control check is completed successfully.  Prior to the bug, the
> library used to query the website and check for the code was configured to
> return a failure if the HTTP status code was not 200 (success). A
> configuration change to the library caused it to return results even when
> the HTTP status code was not 200. Since many web servers are configured to
> include the URL of the req  uest in the body of a 404 (not found) response,
> and the URL also conta

Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

Hi Yuhong,

Perhaps it be best if you create a separate thread for your question -
it's not really clear at all how it relates to the topic at hand.

On Wed, Jan 11, 2017 at 6:08 PM, Yuhong Bao <yuhongbao_...@hotmail.com> wrote:
> That is what the current certificate by Google Internet Authority says.
> What I am referring to is that before Google bought Nest they used GoDaddy as 
> the CA.
> 
> From: Richard Wang <rich...@wosign.com>
> Sent: Wednesday, January 11, 2017 5:01:08 PM
> To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
> Sleevi
> Subject: RE: Incident Report – Certificates issued without proper domain 
> validation
>
> The nest.com certificate subject is:
> CN = www.nest.com
> O = Google Inc
> L = Mountain View
> S = California
> C = US
>
> This means this website owned by Google Inc. Right?
>
>
> Best Regards,
>
> Richard
>
> -Original Message-
> From: dev-security-policy
> [mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
> Behalf Of Yuhong Bao
> Sent: Thursday, January 12, 2017 6:12 AM
> To: Wayne Thayer <wtha...@godaddy.com>;
> dev-security-policy@lists.mozilla.org; Ryan Sleevi <r...@sleevi.com>
> Subject: Re: Incident Report - Certificates issued without proper domain
> validation
>
> I wonder if nest.com is now considered high-risk now. They recently switched
> from GoDaddy to Google Internet Authority.
> 
> From: dev-security-policy
> <dev-security-policy-bounces+yuhongbao_386=hotmail@lists.mozilla.org> on
> behalf of Wayne Thayer <wtha...@godaddy.com>
> Sent: Tuesday, January 10, 2017 7:02:28 PM
> To: dev-security-policy@lists.mozilla.org
> Subject: Incident Report - Certificates issued without proper domain
> validation
>
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
> domain validation processing system. The bug that caused the issue was fixed
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
> determine the scope of the problem, and identified 8850 certificates that
> were issued without proper domain validation as a result of the bug. The
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
> n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to
> holiday vacations and the fact that the issue was not reported properly per
> our CPS, we did not become aware of the issue until one of the employees
> opened the email on Friday Jan 6th and promptly alerted management. The
> issue was originally reported to Microsoft by one of their own customers and
> was described as only affecting certificate requests when the DNS A record
> of the domain was set to 127.0.0.1. An investigation was initiated
> immediately and within a few hours we determined that the problem was
> broader in scope. The root cause of the problem was fixed via a code change
> at approximately 10 PM MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on
> July 29th, 2016 as part of a routine code change intended to improve our
> certificate issuance process. The bug is related to our use of practical
> demonstration of control to validate authority to receive a certificate for
> a given fully-qualified domain name. In the problematic case, we provide a
> random code to a customer and ask them to place it in a specific location on
> their website. Our system automatically checks for the presence of that code
> via an HTTP and/or HTTPS request to the website. If the code is found, the
> domain control check is completed successfully.  Prior to the bug, the
> library used to query the website and check for the code was configured to
> return a failure if the HTTP status code was not 200 (success). A
> configuration change to the library caused it to return results even when
> the HTTP status code was not 200. Since many web servers are configured to
> include the URL of the req  uest in the body of a 404 (not found) response,
> and the URL also contained the random code, any web server configured this
> way caused domain control verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to
> procure a certificate for a domain that was not authorized. The customer who
> discovered the bug revoked the certificate they obtained, and subsequent
> certificates issued as the result of requests used for testing by Microsoft
> a

Re: Incident Report – Certificates issued without proper domain validation

[Yuhong Bao]

That is what the current certificate by Google Internet Authority says.
What I am referring to is that before Google bought Nest they used GoDaddy as 
the CA.

From: Richard Wang <rich...@wosign.com>
Sent: Wednesday, January 11, 2017 5:01:08 PM
To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan Sleevi
Subject: RE: Incident Report – Certificates issued without proper domain 
validation

The nest.com certificate subject is:
CN = www.nest.com
O = Google Inc
L = Mountain View
S = California
C = US

This means this website owned by Google Inc. Right?


Best Regards,

Richard

-Original Message-
From: dev-security-policy
[mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
Behalf Of Yuhong Bao
Sent: Thursday, January 12, 2017 6:12 AM
To: Wayne Thayer <wtha...@godaddy.com>;
dev-security-policy@lists.mozilla.org; Ryan Sleevi <r...@sleevi.com>
Subject: Re: Incident Report - Certificates issued without proper domain
validation

I wonder if nest.com is now considered high-risk now. They recently switched
from GoDaddy to Google Internet Authority.

From: dev-security-policy
<dev-security-policy-bounces+yuhongbao_386=hotmail@lists.mozilla.org> on
behalf of Wayne Thayer <wtha...@godaddy.com>
Sent: Tuesday, January 10, 2017 7:02:28 PM
To: dev-security-policy@lists.mozilla.org
Subject: Incident Report - Certificates issued without proper domain
validation

Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
domain validation processing system. The bug that caused the issue was fixed
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
determine the scope of the problem, and identified 8850 certificates that
were issued without proper domain validation as a result of the bug. The
impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
will also be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to
holiday vacations and the fact that the issue was not reported properly per
our CPS, we did not become aware of the issue until one of the employees
opened the email on Friday Jan 6th and promptly alerted management. The
issue was originally reported to Microsoft by one of their own customers and
was described as only affecting certificate requests when the DNS A record
of the domain was set to 127.0.0.1. An investigation was initiated
immediately and within a few hours we determined that the problem was
broader in scope. The root cause of the problem was fixed via a code change
at approximately 10 PM MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on
July 29th, 2016 as part of a routine code change intended to improve our
certificate issuance process. The bug is related to our use of practical
demonstration of control to validate authority to receive a certificate for
a given fully-qualified domain name. In the problematic case, we provide a
random code to a customer and ask them to place it in a specific location on
their website. Our system automatically checks for the presence of that code
via an HTTP and/or HTTPS request to the website. If the code is found, the
domain control check is completed successfully.  Prior to the bug, the
library used to query the website and check for the code was configured to
return a failure if the HTTP status code was not 200 (success). A
configuration change to the library caused it to return results even when
the HTTP status code was not 200. Since many web servers are configured to
include the URL of the req  uest in the body of a 404 (not found) response,
and the URL also contained the random code, any web server configured this
way caused domain control verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to
procure a certificate for a domain that was not authorized. The customer who
discovered the bug revoked the certificate they obtained, and subsequent
certificates issued as the result of requests used for testing by Microsoft
and GoDaddy have been revoked. Further, any certificate requests made for
domains we flag as high-risk were also subjected to manual review (rather
than being issued purely based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this
method of validation in the period from when the bug was introduced until it
was fixed. A list of 8850 potentially unverified certificates (representing
less than 2% of the total issued during the period) was compiled at 10 PM
PST on Monday Jan 9th. As mentioned above, potentially impacted certificates
will be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google 

RE: Incident Report – Certificates issued without proper domain validation

[Richard Wang]

The nest.com certificate subject is:
CN = www.nest.com
O = Google Inc
L = Mountain View
S = California
C = US

This means this website owned by Google Inc. Right?


Best Regards,

Richard

-Original Message-
From: dev-security-policy
[mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
Behalf Of Yuhong Bao
Sent: Thursday, January 12, 2017 6:12 AM
To: Wayne Thayer <wtha...@godaddy.com>;
dev-security-policy@lists.mozilla.org; Ryan Sleevi <r...@sleevi.com>
Subject: Re: Incident Report - Certificates issued without proper domain
validation

I wonder if nest.com is now considered high-risk now. They recently switched
from GoDaddy to Google Internet Authority.

From: dev-security-policy
<dev-security-policy-bounces+yuhongbao_386=hotmail@lists.mozilla.org> on
behalf of Wayne Thayer <wtha...@godaddy.com>
Sent: Tuesday, January 10, 2017 7:02:28 PM
To: dev-security-policy@lists.mozilla.org
Subject: Incident Report - Certificates issued without proper domain
validation

Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
domain validation processing system. The bug that caused the issue was fixed
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
determine the scope of the problem, and identified 8850 certificates that
were issued without proper domain validation as a result of the bug. The
impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
will also be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
n...@godaddy.com<mailto:n...@godaddy.com> and two GoDaddy employees. Due to
holiday vacations and the fact that the issue was not reported properly per
our CPS, we did not become aware of the issue until one of the employees
opened the email on Friday Jan 6th and promptly alerted management. The
issue was originally reported to Microsoft by one of their own customers and
was described as only affecting certificate requests when the DNS A record
of the domain was set to 127.0.0.1. An investigation was initiated
immediately and within a few hours we determined that the problem was
broader in scope. The root cause of the problem was fixed via a code change
at approximately 10 PM MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on
July 29th, 2016 as part of a routine code change intended to improve our
certificate issuance process. The bug is related to our use of practical
demonstration of control to validate authority to receive a certificate for
a given fully-qualified domain name. In the problematic case, we provide a
random code to a customer and ask them to place it in a specific location on
their website. Our system automatically checks for the presence of that code
via an HTTP and/or HTTPS request to the website. If the code is found, the
domain control check is completed successfully.  Prior to the bug, the
library used to query the website and check for the code was configured to
return a failure if the HTTP status code was not 200 (success). A
configuration change to the library caused it to return results even when
the HTTP status code was not 200. Since many web servers are configured to
include the URL of the req  uest in the body of a 404 (not found) response,
and the URL also contained the random code, any web server configured this
way caused domain control verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to
procure a certificate for a domain that was not authorized. The customer who
discovered the bug revoked the certificate they obtained, and subsequent
certificates issued as the result of requests used for testing by Microsoft
and GoDaddy have been revoked. Further, any certificate requests made for
domains we flag as high-risk were also subjected to manual review (rather
than being issued purely based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this
method of validation in the period from when the bug was introduced until it
was fixed. A list of 8850 potentially unverified certificates (representing
less than 2% of the total issued during the period) was compiled at 10 PM
PST on Monday Jan 9th. As mentioned above, potentially impacted certificates
will be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT
log. Additional code changes were deployed on Monday Jan 9th and Tuesday
10th to prevent the re-issuance of certificates using cached and potentially
unverified domain validation information. However, prior to identifying and
shutting down this path, an additional 101 certificates were reissued using
such cached and potentially unverified domain validation information,
resulting in an overall total of 8951 certificates that were issued without
proper

RE: Incident Report – Certificates issued without proper domain validation

[Wayne Thayer]

Responding to Ryan, Nick and Gerv:

> What's unclear is what steps GoDaddy has taken to remedy this.
> 
> For example:
> 1) Disabling domain control demonstrations through the use of a file on a
> server
> 2) Switching to /.well-known/pkivalidation
> 3) Ensuring that the random value is not part of the HTTP[S] request
> 
> etc
> 
> Could you speak further to how GoDaddy has resolved this problem? My
> hope is that it doesn't involve "Only look for 200 responses" =)

Our process for verifying domain control via a change to the website worked by 
generating a random alphanumeric code. The code was then placed in a file in 
the root directory of the website. The structure of the filename is 
.html. Our system queries for that URL and looks for the code in the 
response.

Our initial response as reported yesterday was to fix the bug introduced in 
July. Based on internal discussions and comments here, as of 12 midnight PST 
last night (1/11) we stopped using this method of file based domain control 
validation.

> Has GoDaddy been following ACME
> https://datatracker.ietf.org/wg/acme/charter/ development, either with a
> view to eventually implementing ACME, or just to learn the same lessons
> about automating domain validation ?

We are aware of the work being done on ACME. We’ll continue to watch its 
development and evolution. Our current focus is on implementing the new domain 
validation processes published by the CAB Forum.

> Perhaps the most surprising thing the ACME WG discovered was that due to
> a common misconfiguration customers sharing a bulk host can often answer
> HTTPS requests for other people's sites that haven't for whatever reason
> enabled SSL yet. GoDaddy's validation method as described would be
> vulnerable to this problem. Can you say what, if anything, GoDaddy does to
> avoid being tricked into issuing a certificate on this basis ?

Here is what we were doing to prevent this: When performing the domain control 
check over HTTPS, we validate the certificate – including verifying that the 
domain name of the site matches one in the cert and that the cert chains to a 
root in the Java root store – and the check fails if the certificate does not 
validate.

> As you will know, the method being used by GoDaddy here corresponds
> broadly to method 3.2.2.4.6 from ballot 169 - "Agreed-Upon Change to
> Website". (Although this method is not currently in the Baseline
> Requirements due to it being part of ballot 182 and having a related IPR
> disclosure, at least one root store operator has suggested they are going to
> require strict adherence to the methods listed in that ballot by 1st March.)
> https://cabforum.org/2016/08/05/ballot-169-revised-validation-
> requirements/
> 
> One of the sentences in 3.2.2.4.6 is the following:
> 
> "The entire Required Website Content MUST NOT appear in the request
> used to retrieve the file or web page"
> 
> This sentence is there precisely because the problem which hit GoDaddy was
> anticipated when the Validation WG was discussing the possible problems
> with this validation method.
> 
> Has GoDaddy already, or is GoDaddy planning to, update its implementation
> to conform to that requirement?

GoDaddy was on track to implement the new requirements prior to March 1, but we 
hope to be able to implement them even sooner than that date.

> > We are currently unaware of
> > any malicious exploitation of this bug to procure a certificate for a
> > domain that was not authorized.
> 
> Does that mean "we have revalidated all the domains", or does it mean "no-
> one has actively reported to us that someone else is using a certificate for a
> domain name the reporter owns"?

As soon as we learned of this issue, we went through every certificate that was 
validated with the HTML method utilized during this period and attempted to 
verify. If it could not be immediately verified, we revoked the certificate.

> > The customer who discovered the bug
> > revoked the certificate they obtained, and subsequent certificates
> > issued as the result of requests used for testing by Microsoft and
> > GoDaddy have been revoked.
> 
> I would hope and assume that such testing was done using domains owned
> by Microsoft and/or GoDaddy, or someone else whose permission you had
> gained?

Yes, testing was done using domains registered by Microsoft and GoDaddy 
employees.

> > authorization). We have re-verified domain control on every
> > certificate issued using this method of validation in the period from
> > when the bug was introduced until it was fixed.
> 
> How was that possible for all domains, as surely some domain owners will
> have taken the necessary file down?

When we learned of this issue, we re-validated every affected certificate. If 
we were unable to properly validate, we revoked the certificate. That is how we 
got the total of 8,951 revoked certificates.

> > A list of 8850
> > potentially unverified certificates (representing less than 2% of the
> > total 

Re: Incident Report – Certificates issued without proper domain validation

[Nick Lamb]

On Wednesday, 11 January 2017 18:31:32 UTC, Ryan Sleevi  wrote:

> Because you're not required to setup the webserver for
> smtp.example.com.

Ah yes, silly me

> I'm not saying they'd be right in arguing so

I suppose that from a husbanding of resources point of view it makes sense to 
wait and see what happens to the remnants of ballot 169 before Mozilla jumps up 
and down about this.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Jakob Bohm]

On 11/01/2017 19:17, Patrick Figel wrote:

On 11/01/2017 19:06, Nick Lamb wrote:

For those who haven't (unlike Patrick) sat down and read the ACME
specification, ACME http-01 won't get tripped here because the
checked content of the URL is very much not the random string (it's a
JWS signature over a data structure containing that random string,
thereby proving it was made by whoever the ACME server is talking
to). But yes, doing something that _looks_ superficially like the
ACME style of validation without such subtlety will trip you up.


Thanks, that's a better way of phrasing what I was trying to say. ;-)


In this very particular case, where the affected validation was
specific to web servers, it seems extremely likely that almost all of
the legitimate certificates (which may be, and we hope is, all of
them) were subsequently put into use on a web server.

Why go to the bother of setting up a web server on say,
smtp.example.com, only to get yourself a certificate, and then turn
off the web server and use the certificate for SMTP? It's not
impossible, but it would be very much the exception.


I don't know this specifically for GoDaddy, but many commercial CAs I've
dealt with in the past typically only validate the "registered domain"
portion (e.g. example.com) of the FQDN and then give you certificates
for any subdomain (e.g. smtp.example.com) under that domain. I think the
approach used by Let's Encrypt, where each FQDN has to be validated
individually, is not all that common otherwise.





For your information, here are the ones I have encountered:

AlphaSSL (Globalsign sub) and one other (a Symantec sub, as far as I
recall) verify control of the corresponding hostmaster e-mail address
for the second level domain (hostmas...@example.com).

Google (for non-certificate purposes) used to verify a url that just
had to return a string that said "google-site-verification: URL" where
URL was the file name part of the Url, this may or may not have been
foolable.  This was done per exact domain.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

On Wed, Jan 11, 2017 at 10:06 AM, Nick Lamb  wrote:
> Why go to the bother of setting up a web server on say, smtp.example.com, 
> only to get yourself a certificate, and then turn off the web server and use 
> the certificate for SMTP? It's not impossible, but it would be very much the 
> exception.

Because you're not required to setup the webserver for
smtp.example.com. It's sufficient to setup the webserver for
example.com to authorize the name, by creatively interpreting the
Method 7 (prior to Ballot 169) and applying the logic from Method 4 to
suggest it's OK to prune the domain (despite Method 6 not allowing
this).

I'm not saying they'd be right in arguing so, but they wouldn't be the
only CA who applied such an interpretation.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Paul Wouters]

On Wed, 11 Jan 2017, Patrick Figel wrote:


On 11/01/2017 04:08, Ryan Sleevi wrote:

Could you speak further to how GoDaddy has resolved this problem? My
hope is that it doesn't involve "Only look for 200 responses" =)


In case anyone is wondering why this is problematic, during the Ballot
169 review process, Peter Bowen ran a check against the top 10,000 Alexa
domains and noted that more than 400 sites returned a HTTP 200 response
for a request to
http://www.$DOMAIN/.well-known/pki-validation/4c079484040e32529577b6a5aade31c5af6fe0c7
[1]. A number of those included the URL in the response body, which
would presumably be good enough for GoDaddy's domain validation process
if they indeed only check for a HTTP 200 response.

[1]: https://cabforum.org/pipermail/public/2016-April/007506.html


Are you saying that for an unknown amount of time (years?) someone could
have faked the domain validation check, and once it was publicly pointed
out so everyone could do this, it took one registrar 10 months to fix,
during which 8800 domains could have been falsely obtained and been used
in targetted attacks? Have other registrars made any statement on
whether they were or were not vulnerable to this attack?

Is there a way to find out if this has actually happened for any domain?
I would expect this would show up as "validated" certificates that were
logged in CT but that were never deployed on the real public TLS servers.
Is anyone monitoring that? I assume that for the "big players" who do
self-monitoring, were not affected? *crosses fingers*

Paul
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Patrick Figel]

On 11/01/2017 04:08, Ryan Sleevi wrote:
> Could you speak further to how GoDaddy has resolved this problem? My
> hope is that it doesn't involve "Only look for 200 responses" =)

In case anyone is wondering why this is problematic, during the Ballot
169 review process, Peter Bowen ran a check against the top 10,000 Alexa
domains and noted that more than 400 sites returned a HTTP 200 response
for a request to
http://www.$DOMAIN/.well-known/pki-validation/4c079484040e32529577b6a5aade31c5af6fe0c7
[1]. A number of those included the URL in the response body, which
would presumably be good enough for GoDaddy's domain validation process
if they indeed only check for a HTTP 200 response.

[1]: https://cabforum.org/pipermail/public/2016-April/007506.html
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Gervase Markham]

Hi Wayne,

As others have said, thanks for bringing this to our attention.

On 11/01/17 03:02, Wayne Thayer wrote:
> results even when the HTTP status code was not 200. Since many web
> servers are configured to include the URL of the request in the body
> of a 404 (not found) response, and the URL also contained the random
> code, 

As you will know, the method being used by GoDaddy here corresponds
broadly to method 3.2.2.4.6 from ballot 169 - "Agreed-Upon Change to
Website". (Although this method is not currently in the Baseline
Requirements due to it being part of ballot 182 and having a related IPR
disclosure, at least one root store operator has suggested they are
going to require strict adherence to the methods listed in that ballot
by 1st March.)
https://cabforum.org/2016/08/05/ballot-169-revised-validation-requirements/

One of the sentences in 3.2.2.4.6 is the following:

"The entire Required Website Content MUST NOT appear in the request used
to retrieve the file or web page"

This sentence is there precisely because the problem which hit GoDaddy
was anticipated when the Validation WG was discussing the possible
problems with this validation method.

Has GoDaddy already, or is GoDaddy planning to, update its
implementation to conform to that requirement?

> We are currently unaware of
> any malicious exploitation of this bug to procure a certificate for a
> domain that was not authorized. 

Does that mean "we have revalidated all the domains", or does it mean
"no-one has actively reported to us that someone else is using a
certificate for a domain name the reporter owns"?

> The customer who discovered the bug
> revoked the certificate they obtained, and subsequent certificates
> issued as the result of requests used for testing by Microsoft and
> GoDaddy have been revoked.

I would hope and assume that such testing was done using domains owned
by Microsoft and/or GoDaddy, or someone else whose permission you had
gained?

> authorization). We have re-verified domain control on every
> certificate issued using this method of validation in the period from
> when the bug was introduced until it was fixed.

How was that possible for all domains, as surely some domain owners will
have taken the necessary file down?

> A list of 8850
> potentially unverified certificates (representing less than 2% of the
> total issued during the period) was compiled at 10 PM PST on Monday
> Jan 9th. 

How were you able to create that list? Do you store the HTTP status code
and content returned from the website, and just searched for non-200
codes? Or some other way?

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Nick Lamb]

As Ryan said, thanks for informing m.d.s.policy about this issue. I am 
interested in the same general area as Ryan but I will ask my question 
separately, feel free to answer them together.

Has GoDaddy been following ACME https://datatracker.ietf.org/wg/acme/charter/ 
development, either with a view to eventually implementing ACME, or just to 
learn the same lessons about automating domain validation ?

Perhaps the most surprising thing the ACME WG discovered was that due to a 
common misconfiguration customers sharing a bulk host can often answer HTTPS 
requests for other people's sites that haven't for whatever reason enabled SSL 
yet. GoDaddy's validation method as described would be vulnerable to this 
problem. Can you say what, if anything, GoDaddy does to avoid being tricked 
into issuing a certificate on this basis ?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

On Tue, Jan 10, 2017 at 7:02 PM, Wayne Thayer  wrote:
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our 
> domain validation processing system. The bug that caused the issue was fixed 
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to 
> determine the scope of the problem, and identified 8850 certificates that 
> were issued without proper domain validation as a result of the bug. The 
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and 
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to 
> n...@godaddy.com and two GoDaddy employees. Due to 
> holiday vacations and the fact that the issue was not reported properly per 
> our CPS, we did not become aware of the issue until one of the employees 
> opened the email on Friday Jan 6th and promptly alerted management. The issue 
> was originally reported to Microsoft by one of their own customers and was 
> described as only affecting certificate requests when the DNS A record of the 
> domain was set to 127.0.0.1. An investigation was initiated immediately and 
> within a few hours we determined that the problem was broader in scope. The 
> root cause of the problem was fixed via a code change at approximately 10 PM 
> MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on 
> July 29th, 2016 as part of a routine code change intended to improve our 
> certificate issuance process. The bug is related to our use of practical 
> demonstration of control to validate authority to receive a certificate for a 
> given fully-qualified domain name. In the problematic case, we provide a 
> random code to a customer and ask them to place it in a specific location on 
> their website. Our system automatically checks for the presence of that code 
> via an HTTP and/or HTTPS request to the website. If the code is found, the 
> domain control check is completed successfully.  Prior to the bug, the 
> library used to query the website and check for the code was configured to 
> return a failure if the HTTP status code was not 200 (success). A 
> configuration change to the library caused it to return results even when the 
> HTTP status code was not 200. Since many web servers are configured to 
> include the URL of the r
 eq
>  uest in the body of a 404 (not found) response, and the URL also contained 
> the random code, any web server configured this way caused domain control 
> verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to procure 
> a certificate for a domain that was not authorized. The customer who 
> discovered the bug revoked the certificate they obtained, and subsequent 
> certificates issued as the result of requests used for testing by Microsoft 
> and GoDaddy have been revoked. Further, any certificate requests made for 
> domains we flag as high-risk were also subjected to manual review (rather 
> than being issued purely based on an invalid domain authorization).
> We have re-verified domain control on every certificate issued using this 
> method of validation in the period from when the bug was introduced until it 
> was fixed. A list of 8850 potentially unverified certificates (representing 
> less than 2% of the total issued during the period) was compiled at 10 PM PST 
> on Monday Jan 9th. As mentioned above, potentially impacted certificates will 
> be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT log. 
> Additional code changes were deployed on Monday Jan 9th and Tuesday 10th to 
> prevent the re-issuance of certificates using cached and potentially 
> unverified domain validation information. However, prior to identifying and 
> shutting down this path, an additional 101 certificates were reissued using 
> such cached and potentially unverified domain validation information, 
> resulting in an overall total of 8951 certificates that were issued without 
> proper domain validation as a result of the bug.
> Next Steps:
> While we are confident that we have completely resolved the problem, we are 
> watching our system closely to ensure that no more certificates are issued 
> without proper domain validation, and we will take immediate action and 
> report any further issues if found. A full post-mortem review of this 
> incident will occur and steps will be taken to prevent a recurrence, 
> including the addition of automated tests designed to detect this type of 
> scenario. If more information about the cause or impact of this incident 
> becomes available, we will publish updates to this report.
> Wayne Thayer
> GoDaddy

Wayne,

Thanks for sharing these details.

What's unclear is what steps GoDaddy has taken to remedy this.

For example:
1) Disabling domain control demonstrations through