Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[Ryan Sleevi via dev-security-policy]

On Fri, Jan 12, 2018 at 5:46 AM Jakob Bohm via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> On 11/01/2018 05:38, Ryan Sleevi wrote:
> > On Thu, Jan 11, 2018 at 2:46 AM Jakob Bohm via dev-security-policy <
> > dev-security-policy@lists.mozilla.org> wrote:
> >
> >> On 11/01/2018 01:08, Ryan Sleevi wrote:
> >>> On Wed, Jan 10, 2018 at 6:35 PM, Jakob Bohm via dev-security-policy <
> >>> dev-security-policy@lists.mozilla.org> wrote:
> 
>  Agree.
> 
>  Hence my suggestion that TLS-SNI-0next use a name under the customer's
>  domain (such as the name used for DNS-01), not a name under .invalid.
> >>>
> >>>
> >>> I thought it had been pointed out, but that doesn't actually address
> the
> >>> vulnerability. It presumes a hierarchal nature of certificate/account
> >>> management that simply does not align with how the real world and how
> >> real
> >>> providers work.
> >>>
> >>
> >> There are TWO related vulnerabilities at work here:
> >>
> >> 1. A lot of hosting providers allow users to provision certificates for
> >> whatever.acme.invalid on SNI capable hosts, even though those users
> >> are not owners of whatever domain was issued a challenge for the
> >> number in the "whatever" part.  Other than adding code to
> specifically
> >> block "acme.invalid" to every software stack/configuration used by
> >> hosting providers, this is almost unsolvable at the host provider
> end,
> >> thus it may be easier to change the TLS-SNI-xx method in some way.
> >>
> >> 2. A much smaller group of hosting providers allow users to set up
> >> hosting for subdomains of domains already provisioned for other
> users
> >> (e.g. user B setting up hosting for whatever.acme.example.com when
> >> user A is already using the host for example.com).  This case is
> not
> >> solved by changing the SNI challenge to be a subdomain of the domain
> >> being validated.  But since this is a smaller population of hosting
> >> providers, getting them to at least enforce that the parent domain
> >> user needs to authorize which other users can host a subdomain with
> >> them is much more tractable, especially as it has obvious direct
> >> security benefits outside the ACME context.
> >
> >
> > This is categorically false. It is, itself, more complex, more error
> prone,
> > and more complexity (for example, due to the nature of authorization
> > domains) and that, at the end of the day, fails to achieve its goals.
> >
>
> I explicitly stated why fixing #2 would be simpler than fixing #1, you
> are making no factual argument.


It would not be, because you do not understand the constraints of the
servers that exist that are affected or the issue, it would seem.

It would appear you are guessing as to how they work, but they do not work
that way, thus this proposal would not in any way be better.


>
> Which specific "nature of authorization domains" are you referring to?


Read the Baseline Requirements for Authorization Domain Name and how label
pruning work. Your design assumes a single account is associates with a
registerable domain and all its subdomains, but if you read about who the
issue affects, you would see that it precisely affects those who do not
have such a notion of authorized domains to accounts.

I was referring to strict subdomains.  So User B would have to give
> permission to User C (e.g. in a control panel or via an admin procedure,
> that's up to the host).


Correct. And these are not the folks affected and thus not why this is
worth discussing further.


>
> User B would also have to give permission (in the same way) for user A
> (or vice versa if user A requested before user B).  Again the details
> can be left to the discretion of the host, as long as user C would need
> permission from user B, issue 2 goes away as far as a hypothetical
> improved TLS-SNI-0next using a subdomain of the requested domain is
> concerned.


If and only if Authorization is done by accounts, which it is not, which
means this solves nothing.

> Finally, the assumption there will be fewer of X so it’s easier to fix is,
> > also, counterintuitively false - the fewer there are and the more baroque
> > and complex the solution is, the harder it is to make any assumption
> about
> > adoption uptake.
> >
>
> I am trying to make solutions simpler and less baroque than what Let's
> encrypt is apparently (according to 3rd party posts here) proposing
> namely to introduce a special rule that hosting providers must follow to
> protect their users against the newly discovered vulnerability).


“According to third party posts” - they’ve already made first party
updates. I fail to see what you hope to attain from that comment, but it is
also clear that you don’t understand the constraints - at best, you are
guessing, but speaking as if it’s correct (rather than soliciting feedback
or displaying uncertainty) and ignoring criticism, which is completely

Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[Jakob Bohm via dev-security-policy]

On 11/01/2018 05:38, Ryan Sleevi wrote:

On Thu, Jan 11, 2018 at 2:46 AM Jakob Bohm via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:


On 11/01/2018 01:08, Ryan Sleevi wrote:

On Wed, Jan 10, 2018 at 6:35 PM, Jakob Bohm via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:


Agree.

Hence my suggestion that TLS-SNI-0next use a name under the customer's
domain (such as the name used for DNS-01), not a name under .invalid.



I thought it had been pointed out, but that doesn't actually address the
vulnerability. It presumes a hierarchal nature of certificate/account
management that simply does not align with how the real world and how

real

providers work.



There are TWO related vulnerabilities at work here:

1. A lot of hosting providers allow users to provision certificates for
whatever.acme.invalid on SNI capable hosts, even though those users
are not owners of whatever domain was issued a challenge for the
number in the "whatever" part.  Other than adding code to specifically
block "acme.invalid" to every software stack/configuration used by
hosting providers, this is almost unsolvable at the host provider end,
thus it may be easier to change the TLS-SNI-xx method in some way.

2. A much smaller group of hosting providers allow users to set up
hosting for subdomains of domains already provisioned for other users
(e.g. user B setting up hosting for whatever.acme.example.com when
user A is already using the host for example.com).  This case is not
solved by changing the SNI challenge to be a subdomain of the domain
being validated.  But since this is a smaller population of hosting
providers, getting them to at least enforce that the parent domain
user needs to authorize which other users can host a subdomain with
them is much more tractable, especially as it has obvious direct
security benefits outside the ACME context.



This is categorically false. It is, itself, more complex, more error prone,
and more complexity (for example, due to the nature of authorization
domains) and that, at the end of the day, fails to achieve its goals.



I explicitly stated why fixing #2 would be simpler than fixing #1, you
are making no factual argument.

Which specific "nature of authorization domains" are you referring to?


The simplest way I can try to get you to think about it is to consider a
cert for foo.bar.example.com being requested by Iser C, and preexisting
domains of www.example.com (User A) and example.com (Iser B). Think about
how that would be “checked” - or even simply who the authorizors should be.



I was referring to strict subdomains.  So User B would have to give
permission to User C (e.g. in a control panel or via an admin procedure,
that's up to the host).

User B would also have to give permission (in the same way) for user A
(or vice versa if user A requested before user B).  Again the details
can be left to the discretion of the host, as long as user C would need
permission from user B, issue 2 goes away as far as a hypothetical
improved TLS-SNI-0next using a subdomain of the requested domain is
concerned.


I assure you, it both fails to address the problem (of limiting risk) and
increases the complexity. Put simply, it doesn’t work - so there is no
value in doubling down trying to make it work, especially given that it
also fails to provide a solution for the overall population (like
blacklisting does).


Again you make no factual argument.



Finally, the assumption there will be fewer of X so it’s easier to fix is,
also, counterintuitively false - the fewer there are and the more baroque
and complex the solution is, the harder it is to make any assumption about
adoption uptake.



I am trying to make solutions simpler and less baroque than what Let's
encrypt is apparently (according to 3rd party posts here) proposing
namely to introduce a special rule that hosting providers must follow to
protect their users against the newly discovered vulnerability).

I am also trying to avoid a solution likely to suffer from a "long tail"
problem of hosts that won't implement needed fixes anytime soon.


(Hosting providers who allow uploading certificates for the specific

DNS/SNI names of other users are a security problem in itself, as it
could allow e.g. uploading an untrusted exact domain cert to disrupt
another user's site having only a wildcard certificate).



Not really. You say this but that is the reality today and can and is
mitigated.



How is it mitigated today in a way that would not stop users from
uploading a cert for somenumber._acme.www.example.com on the same host
where some other user is hosting www.example.com?


On the other hand, such providers will often (included or at extra fee)

allow provisioning arbitrary subdomains that are then typically added to
the HTTP(S) vhost configuration and the hosted DNS configuration, which
is good enough for TLS-SNI-modified-to-use-subdomain 

Re: DYMO Root CA installed by Label Printing Software

[Nicholas Humfrey via dev-security-policy]

Thank you very much to everyone who replied to my original post. I think 
the fact that so many people are making the same mistakes indicates that 
the correct solutions are not obvious to many developers.



I have added a "How could this be done better?" section to my README:
https://github.com/njh/dymo-root-ca-security-risk/blob/master/README.md

Please let me know if I have misunderstood any of it.


DYMO have replied to my enquiry saying:
"After investigating the problem that you described, I escalated the 
issue to the developers."


So fingers crossed they are taking this seriously.


nick.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[Wayne Thayer via dev-security-policy]

On Thu, Jan 11, 2018 at 3:28 PM, josh--- via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

>
> https://community.letsencrypt.org/t/2018-01-11-update-regard
> ing-acme-tls-sni-and-shared-hosting-infrastructure/50188
>
> Speaking for myself, this is an excellent game plan that prioritizes the
protection of Mozilla users and the Web PKI in general.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[josh--- via dev-security-policy]

On Thursday, January 11, 2018 at 3:36:50 PM UTC-6, Ryan Sleevi wrote:
> On Wed, Jan 10, 2018 at 4:33 AM, josh--- via dev-security-policy <
> dev-security-policy@lists.mozilla.org> wrote:
> 
> > At approximately 5 p.m. Pacific time on January 9, 2018, we received a
> > report from Frans Rosén of Detectify outlining a method of exploiting some
> > shared hosting infrastructures to obtain certificates for domains he did
> > not control, by making use of the ACME TLS-SNI-01 challenge type. We
> > quickly confirmed the issue and mitigated it by entirely disabling
> > TLS-SNI-01 validation in Let’s Encrypt. We’re grateful to Frans for finding
> > this issue and reporting it to us.
> >
> > We’d like to describe the issue and our plans for possibly re-enabling
> > TLS-SNI-01 support.
> >
> > Problem Summary
> >
> > In the ACME protocol’s TLS-SNI-01 challenge, the ACME server (the CA)
> > validates a domain name by generating a random token and communicating it
> > to the ACME client. The ACME client uses that token to create a self-signed
> > certificate with a specific, invalid hostname (for example,
> > 773c7d.13445a.acme.invalid), and configures the web server on the domain
> > name being validated to serve that certificate. The ACME server then looks
> > up the domain name’s IP address, initiates a TLS connection, and sends the
> > specific .acme.invalid hostname in the SNI extension. If the response is a
> > self-signed certificate containing that hostname, the ACME client is
> > considered to be in control of the domain name, and will be allowed to
> > issue certificates for it.
> >
> > However, Frans noticed that at least two large hosting providers combine
> > two properties that together violate the assumptions behind TLS-SNI:
> >
> > * Many users are hosted on the same IP address, and
> > * Users have the ability to upload certificates for arbitrary names
> > without proving domain control.
> >
> > When both are true of a hosting provider, an attack is possible. Suppose
> > example.com’s DNS is pointed at the same shared hosting IP address as a
> > site controlled by the attacker. The attacker can run an ACME client to get
> > a TLS-SNI-01 challenge, then install their .acme.invalid certificate on the
> > hosting provider. When the ACME server looks up example.com, it will
> > connect to the hosting provider’s IP address and use SNI to request the
> > .acme.invalid hostname. The hosting provider will serve the certificate
> > uploaded by the attacker. The ACME server will then consider the attacker’s
> > ACME client authorized to issue certificates for example.com, and be
> > willing to issue a certificate for example.com even though the attacker
> > doesn’t actually control it.
> >
> > This issue only affects domain names that use hosting providers with the
> > above combination of properties. It is independent of whether the hosting
> > provider itself acts as an ACME client.
> >
> > Our Plans
> >
> > Shortly after the issue was reported, we disabled TLS-SNI-01 in Let’s
> > Encrypt. However, a large number of people and organizations use the
> > TLS-SNI-01 challenge type to get certificates. It’s important that we
> > restore service if possible, though we will only do so if we’re confident
> > that the TLS-SNI-01 challenge type is sufficiently secure.
> >
> > At this time, we believe that the issue can be addressed by having certain
> > services providers implement stronger controls for domains hosted on their
> > infrastructure. We have been in touch with the providers we know to be
> > affected, and mitigations will start being deployed for their systems
> > shortly.
> >
> > Over the next 48 hours we will be building a list of vulnerable providers
> > and their associated IP addresses. Our tentative plan, once the list is
> > completed, is to re-enable the TLS-SNI-01 challenge type with vulnerable
> > providers blocked from using it.
> >
> > We’re also going to be soliciting feedback on our plans from our
> > community, partners and other PKI stakeholders prior to re-enabling the
> > TLS-SNI-01 challenge. There is a lot to consider here and we’re looking
> > forward to feedback.
> >
> > We will post more information and details as our plans progress.
> >
> 
> (Wearing a Google Chrome hat on behalf of our root store policy)
> 
> Josh,
> 
> Thanks for bringing this rapidly to the attention of the broader community
> and proactively reaching out to root programs.
> 
> As framing to the discussion, we still believe TLS-SNI is fully permitted
> by the Baseline Requirements, which, while not ideal, still permits
> issuance using this method. As such, the 'root' cause is that the Baseline
> Requirements permit methods that are less secure than desired, and the
> discussion that follows is now around what steps to take - as CAs, as Root
> Programs, for site operators, and for the CA/Browser Forum.
> 
> When faced with a vulnerable validation method that is permitted, it's
> always a challenge to balance 

Re: Possible Issue with Domain Validation Method 9 in a shared hosting environment

[Ryan Sleevi via dev-security-policy]

On Thu, Jan 11, 2018 at 4:50 PM, Doug Beattie via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

>
> Based on reported issues with TLS-SNI-01, we started investigation of our
> systems late yesterday regarding the use of "Test Certificate" validation,
> BR section  3.2.2.4.9.
>
> We found that this method may be vulnerable to the some of the same
> underlying issue as the ACME TLS-SNI-01 so we disabled it at 10:51 AM today
> EST, January 11th.
>
> While TLS-SNI-01 uses a host name like 773c7d.13445a.acme.invalid,
> GlobalSign uses the actual host name, www.example.com example.com> which limits abuse, but we believe that the process might be
> vulnerable in some cases.
>
> We're continuing to research this and will let you know what we find.
>
> Doug
>

(Wearing a Chrome Hat, again)

Doug,

Thanks for the update. That seems consistent with Chrome's evaluation, as
shared at
https://groups.google.com/d/msg/mozilla.dev.security.policy/RHsIInIjJA0/HACyY9tMAAAJ


We'd like to ask that you work with the community and browsers on this
prior to re-enabling this validation method, for the reasons outlined in
that thread.

In particular, unlike the ACME methods that are specified, it's unclear the
validation process used by GlobalSign in applying 3.2.2.4.9. This makes it
particularly more challenging to evaluate the potential risks and/or
mitigations that may exist, and so sharing full and complete details about
the method you use publicly can assist browsers and the broader community
to evaluate and assess, much the same as TLS-SNI for ACME.

Further, as called out in that other thread, there's a risk calculus based
on the lifetime of the certificates issued that directly plays into whether
or not the method can be re-enabled and for how long. We'd love to work
with you to better understand these tradeoffs, as applied to .9, so that we
can make informed decisions about the risk to sites and users.

Thanks for your report, for disabling the validation, and for continued
collaboration to best protect users.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Possible Issue with Domain Validation Method 9 in a shared hosting environment

[Doug Beattie via dev-security-policy]

Based on reported issues with TLS-SNI-01, we started investigation of our 
systems late yesterday regarding the use of "Test Certificate" validation, BR 
section  3.2.2.4.9.

We found that this method may be vulnerable to the some of the same underlying 
issue as the ACME TLS-SNI-01 so we disabled it at 10:51 AM today EST, January 
11th.

While TLS-SNI-01 uses a host name like 773c7d.13445a.acme.invalid, GlobalSign 
uses the actual host name, www.example.com which limits 
abuse, but we believe that the process might be vulnerable in some cases.

We're continuing to research this and will let you know what we find.

Doug


Doug Beattie
Vice President of Product Management
GlobalSign
Two International Drive | Suite 150 | Portsmouth, NH 03801
Email: doug.beat...@globalsign.com
www.globalsign.com

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[Ryan Sleevi via dev-security-policy]

On Wed, Jan 10, 2018 at 4:33 AM, josh--- via dev-security-policy <
dev-security-policy@lists.mozilla.org> wrote:

> At approximately 5 p.m. Pacific time on January 9, 2018, we received a
> report from Frans Rosén of Detectify outlining a method of exploiting some
> shared hosting infrastructures to obtain certificates for domains he did
> not control, by making use of the ACME TLS-SNI-01 challenge type. We
> quickly confirmed the issue and mitigated it by entirely disabling
> TLS-SNI-01 validation in Let’s Encrypt. We’re grateful to Frans for finding
> this issue and reporting it to us.
>
> We’d like to describe the issue and our plans for possibly re-enabling
> TLS-SNI-01 support.
>
> Problem Summary
>
> In the ACME protocol’s TLS-SNI-01 challenge, the ACME server (the CA)
> validates a domain name by generating a random token and communicating it
> to the ACME client. The ACME client uses that token to create a self-signed
> certificate with a specific, invalid hostname (for example,
> 773c7d.13445a.acme.invalid), and configures the web server on the domain
> name being validated to serve that certificate. The ACME server then looks
> up the domain name’s IP address, initiates a TLS connection, and sends the
> specific .acme.invalid hostname in the SNI extension. If the response is a
> self-signed certificate containing that hostname, the ACME client is
> considered to be in control of the domain name, and will be allowed to
> issue certificates for it.
>
> However, Frans noticed that at least two large hosting providers combine
> two properties that together violate the assumptions behind TLS-SNI:
>
> * Many users are hosted on the same IP address, and
> * Users have the ability to upload certificates for arbitrary names
> without proving domain control.
>
> When both are true of a hosting provider, an attack is possible. Suppose
> example.com’s DNS is pointed at the same shared hosting IP address as a
> site controlled by the attacker. The attacker can run an ACME client to get
> a TLS-SNI-01 challenge, then install their .acme.invalid certificate on the
> hosting provider. When the ACME server looks up example.com, it will
> connect to the hosting provider’s IP address and use SNI to request the
> .acme.invalid hostname. The hosting provider will serve the certificate
> uploaded by the attacker. The ACME server will then consider the attacker’s
> ACME client authorized to issue certificates for example.com, and be
> willing to issue a certificate for example.com even though the attacker
> doesn’t actually control it.
>
> This issue only affects domain names that use hosting providers with the
> above combination of properties. It is independent of whether the hosting
> provider itself acts as an ACME client.
>
> Our Plans
>
> Shortly after the issue was reported, we disabled TLS-SNI-01 in Let’s
> Encrypt. However, a large number of people and organizations use the
> TLS-SNI-01 challenge type to get certificates. It’s important that we
> restore service if possible, though we will only do so if we’re confident
> that the TLS-SNI-01 challenge type is sufficiently secure.
>
> At this time, we believe that the issue can be addressed by having certain
> services providers implement stronger controls for domains hosted on their
> infrastructure. We have been in touch with the providers we know to be
> affected, and mitigations will start being deployed for their systems
> shortly.
>
> Over the next 48 hours we will be building a list of vulnerable providers
> and their associated IP addresses. Our tentative plan, once the list is
> completed, is to re-enable the TLS-SNI-01 challenge type with vulnerable
> providers blocked from using it.
>
> We’re also going to be soliciting feedback on our plans from our
> community, partners and other PKI stakeholders prior to re-enabling the
> TLS-SNI-01 challenge. There is a lot to consider here and we’re looking
> forward to feedback.
>
> We will post more information and details as our plans progress.
>

(Wearing a Google Chrome hat on behalf of our root store policy)

Josh,

Thanks for bringing this rapidly to the attention of the broader community
and proactively reaching out to root programs.

As framing to the discussion, we still believe TLS-SNI is fully permitted
by the Baseline Requirements, which, while not ideal, still permits
issuance using this method. As such, the 'root' cause is that the Baseline
Requirements permit methods that are less secure than desired, and the
discussion that follows is now around what steps to take - as CAs, as Root
Programs, for site operators, and for the CA/Browser Forum.

When faced with a vulnerable validation method that is permitted, it's
always a challenge to balance the need for security - for sites and users -
with the risk of compatibility and breakage from the removal of such a
method. Fundamentally, the issues you raise call into question the level of
assurance of 3.2.2.4.9 and 3.2.2.4.10 in the Baseline Requirements, 

Re: 2018.01.09 Issue with TLS-SNI-01 and Shared Hosting Infrastructure

[Gervase Markham via dev-security-policy]

On 10/01/18 17:39, Matthew Hardeman wrote:
> Here again, I think we have a problem.  It's regarded as normal and
> acceptable at many web host infrastructures to pre-stage sites for
> domain-labels not yet in use to allow for development and test deployment.

I agree that "no unknown domain names" is hard to implement "No domain
names owned by another customer" is easier and doesn't cause the
problems you raise. "No acme.invalid" is even easier.

> In the course of adopting the 10 blessed methods, did any of the methods
> move forward with the expectation that active effort on the part of non-CA
> participants versus the status quo would be required in order to ensure the
> continuing reliability of the method?

"Active effort vs. the status quo" is a skewed framing because security
always requires active effort in the face of change, new entrants etc. A
new entrant in the email market has to make an active effort to make
sure that the special addresses are not claimable, even though that
issue has been known for years.

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy