Re: [Acme] Stale DNS and high-risk issuance

[Ilari Liusvaara]

On Wed, Jan 03, 2018 at 12:48:45PM +, Tobias Fiebig wrote:
> Hello Ilari,
> > [1] Just as example, the following kinds of stuff are (legimately) seen 
> > fairly
> > often or more:
> > 
> > 
> > - Rolling over the account key for each renewal (this is usually caused
> >   by containers forgetting things).
> > - Rolling over the TLS key for each renewal (some actually recommend
> >   this, I do not). Sometimes with very frequent renewals.
> > - Getting multiple certificates with different keys for multiple
> >   servers with the same name.
> > - Handling validation on separate system and pushing the certificates
> >   (and likely the TLS keys too, probably not in safe manner!) to server
> >   systems. These systems might utilize HTTP redirects (which HTTP-01
> >   does follow) or DNS CNAMEs (which DNS-01 does follow).
> > - Users wiping out the TLS keys and account keys (without backup!) to
> >   "reset" something (admin mistake).
> > - Transferring site between servers and losing the keys (account and
> >   TLS) in the process.
> > - Users using all sorts of whacky ACME clients that just do not
> >   implement anything more than bare minimum for the common case.
> > - Users using HTTPS on servers they don't have proper control of
> >   DNS for (can't edit records, or can only use very few record
> >   types, at worst only A/, or worse yet, A only).
> > 
> > Thinking about recovery is rather important. One of the major reasons HPKP
> > is so hated is lack of pretty much any kind of recovery.
>
> I am seriously interested in surveying this kind of behavior, not only
> to quantify it, but also to understand MOs during systems' operation.
> While some of these cases seem relatively easy to measure (yet sometimes
> not to distinguish, i.e., cases 1 and 2), others seem to be harder
> harder to measure, e.g., the 4th case. The main problem here is that
> this would certainly require cooperation from LE. What are your thoughts
> on this?

I am not sure one could get much useful information about frequency
of many of these even with full validation and issuance logs from CA
(which one probably can not get).

Well, the point was: Keys get rotated or lost. One needs recovery
method. In worst case, DNS can not be used or recovery needs to be
automatic. Also, nasty hacks like how TLS-SNI is done with most
webservers are really to be avoided.


One obvious way that would not need to deal with recovery would be
validating all global IPs seen, or random subset if there are too
many. However, even this might not be problem free (because IPv6).


> > [2] This was actually one of the major reasons why the PoP challenge
> > was removed.
> Well, technically it should be sufficient to do the channel over HTTPs
> to reach the same result in comparison to actually signing the challenge.

Intestingly, the test certificate method in BRs requires keys to
match. The random number in certificate method was split from that,
and it does not have key-matching requirement.

(ACME as protocol was not designed to be able to really deal with key-
matching requirements).



-Ilari

___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme

Re: [Acme] Stale DNS and high-risk issuance

[Tobias Fiebig]

Hello Ilari,
> [1] Just as example, the following kinds of stuff are (legimately) seen fairly
> often or more:
> 
> 
> - Rolling over the account key for each renewal (this is usually caused
>   by containers forgetting things).
> - Rolling over the TLS key for each renewal (some actually recommend
>   this, I do not). Sometimes with very frequent renewals.
> - Getting multiple certificates with different keys for multiple
>   servers with the same name.
> - Handling validation on separate system and pushing the certificates
>   (and likely the TLS keys too, probably not in safe manner!) to server
>   systems. These systems might utilize HTTP redirects (which HTTP-01
>   does follow) or DNS CNAMEs (which DNS-01 does follow).
> - Users wiping out the TLS keys and account keys (without backup!) to
>   "reset" something (admin mistake).
> - Transferring site between servers and losing the keys (account and
>   TLS) in the process.
> - Users using all sorts of whacky ACME clients that just do not
>   implement anything more than bare minimum for the common case.
> - Users using HTTPS on servers they don't have proper control of
>   DNS for (can't edit records, or can only use very few record
>   types, at worst only A/, or worse yet, A only).
> 
> Thinking about recovery is rather important. One of the major reasons HPKP
> is so hated is lack of pretty much any kind of recovery.
I am seriously interested in surveying this kind of behavior, not only to 
quantify it, but also to understand MOs during systems' operation. While some 
of these cases seem relatively easy to measure (yet sometimes not to 
distinguish, i.e., cases 1 and 2), others seem to be harder harder to measure, 
e.g., the 4th case. The main problem here is that this would certainly require 
cooperation from LE. What are your thoughts on this?

> [2] This was actually one of the major reasons why the PoP challenge was
> removed.
Well, technically it should be sufficient to do the channel over HTTPs to reach 
the same result in comparison to actually signing the challenge.


Met vriendelijke groet,
 
Tobias Fiebig,
Department Engineering Systems and Services

Informatie- en Communicatie Technologie (ICT)
 
TU Delft / Dept. ESS
Faculty of Technology, Policy and Management (TBM)
Building 31
Jaffalaan 5 - room B3.170
2628 BX  Delft
P.O.Box 5015
2600 GA Delft, The Netherlands
T +31 (0)15 27 85700
E  t.fie...@tudelft.nl

Present: Monday t/m Friday
___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme

Re: [Acme] Stale DNS and high-risk issuance

[Ilari Liusvaara]

On Fri, Dec 29, 2017 at 10:43:43PM +0100, Kevin Borgolte wrote:
> Hi all,
> 
> we have reached out a while ago about IP address staleness and domain
> takeover attacks, and a way to prevent certificate issuance in these
> cases (July 2017).
> 
> Since then, we have conducted some measurements on the practicality of
> the attacks. Furthermore, we wrote down our defense in a structured
> manner. The results of this will appear as a paper at the Network and
> Distributed Systems Symposium (NDSS) in February 2018.
> 
> As you suggested when we approached the list earlier, we would love to
> contribute a RFC on it. However, based on that initial feedback, we
> were wondering how to move forward? Would you prefer an additional RFC
> for the proposed ACME challenge, or would you want to incorporate it
> into an existing RFC?

I think it is bit too late for the existing RFC. So it would have to
be new RFC. Which means you would need to post draft for it.

I think there would be much to discuss about ecosystem impact on the
proposed countermeasures[1], and also cryptographic cleaniness (e.g., it
is not cryptographically kosher to use TLS private key for JWS[2]).


[1] Just as example, the following kinds of stuff are (legimately) seen
fairly often or more:


- Rolling over the account key for each renewal (this is usually caused
  by containers forgetting things).
- Rolling over the TLS key for each renewal (some actually recommend
  this, I do not). Sometimes with very frequent renewals.
- Getting multiple certificates with different keys for multiple
  servers with the same name.
- Handling validation on separate system and pushing the certificates
  (and likely the TLS keys too, probably not in safe manner!) to server
  systems. These systems might utilize HTTP redirects (which HTTP-01
  does follow) or DNS CNAMEs (which DNS-01 does follow).
- Users wiping out the TLS keys and account keys (without backup!) to
  "reset" something (admin mistake).
- Transferring site between servers and losing the keys (account and
  TLS) in the process.
- Users using all sorts of whacky ACME clients that just do not
  implement anything more than bare minimum for the common case.
- Users using HTTPS on servers they don't have proper control of
  DNS for (can't edit records, or can only use very few record
  types, at worst only A/, or worse yet, A only).

Thinking about recovery is rather important. One of the major reasons
HPKP is so hated is lack of pretty much any kind of recovery.


[2] This was actually one of the major reasons why the PoP challenge
was removed.


-Ilari

___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme

[Acme] Stale DNS and high-risk issuance

[Kevin Borgolte]

Hi all,

we have reached out a while ago about IP address staleness and domain
takeover attacks, and a way to prevent certificate issuance in these
cases (July 2017).

Since then, we have conducted some measurements on the practicality of
the attacks. Furthermore, we wrote down our defense in a structured
manner. The results of this will appear as a paper at the Network and
Distributed Systems Symposium (NDSS) in February 2018.

As you suggested when we approached the list earlier, we would love to
contribute a RFC on it. However, based on that initial feedback, we
were wondering how to move forward? Would you prefer an additional RFC
for the proposed ACME challenge, or would you want to incorporate it
into an existing RFC?

If you are interested in reading the paper, please reach out. While we
have not made the paper public yet, we are happy to share it privately
here.

Best,
Kevin

P.S. We plan to release the paper publicly early January.

___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme

Re: [Acme] Stale DNS and high-risk issuance

[Ilari Liusvaara]

On Thu, Jul 20, 2017 at 11:20:58PM -0700, Kevin Borgolte wrote:
> Hi Ilari,
> 
> 
> > The methods in base spec just fundamentially rely on security of DNS.
> > The only way to remove dependence on routing would be:
> >
> > 1) Deploy DNSSEC and
> > 2) Add acme-CAA record restricting validation to DNS.
> 
> This would indeed be an alternative solution to our proposal, as it
> would similarly solve the problem of DNS-related issues. However, a
> DNS-based solution would not be as practical for most ACME users (yet)
> because they might not have full control over their nameservers (the
> web interface of their registrar that they might have to use might not
> even support CAA), or it might require manual work to renew the
> certificate. Further considering regular key renewals, a DNS-based
> solution appears to be less practical and could hamper adoption.

Yes, requiring this would have very large impact, so it would be
nonstarter.

There are substantial amount of DNS APIs that can be used to automate
DNS updates for DNS validation, making DNS validation totally automatic
(and most of the time DNS validation is used, it is totally automatic).

The nasty part would be requiring DNSSec, which is substantially more
difficult than automated record updates.

> > > For example: Consider that Let's Encrypt is used by a service running
> > > a.example.com, where the A record of a.example.com points to 192.0.2.1 and
> > > 198.51.100.1, with the record pointing to 198.51.100.1 being a stale DNS
> > > record, and 198.51.100.1 being an address for a VM instance in a cloud
> > > provider's pool. If an attacker now finds a way to allocate 198.51.100.1,
> > > she might also use Let's Encrypt to obtain a valid certificate. The
> > > certificate request by the attacker is indistinguishable from the outside
> > > and from interpreting the certificate transparency logs from what the
> > > legitimate operator would do to fix the service behind 198.51.100.1, which
> > > previously went stale.
> 
> > Stale IP addresses cause problems when renewing (however, that does not
> > occur that often). Also, due to recent "fixes", IPv6 stale addresses can
> > be missed (there are just so many folks with busted DNS configuraiton).
> >
> > And I think that would also cause large amounts of problems for users
> > if the stale record is IPv4 (if it is IPv6, then "happy eyeballs" hides
> > the problem).
> 
> We are not 100% sure what you mean exactly here, but for us, the
> problem with stale DNS records is not with renewing itself, but with
> an attacker exploiting the stale DNS information combined with the
> high IP address churn (which we see particularly for cloud providers).

I would think that having any stale IPv4 addresses in the RRset would
cause major issues for users accessing the site.
 
> > In the "order flow" change, did put the CSR first so one can look up
> > the requested key first. However, this would cause major problems if
> > trying to change keys (and many (I am not among them) believe that
> > rotating private keys often is important).
> 
> Would you mind elaborate where this would cause major problems exactly
> when changing keys?

The ACME server can detect if the key requested is the same as the
last time.
 
> > This would also complicate "disaster recovery" where the previous keys
> > are lost (usually due to admin mistake, and backups being AWOL like
> > usual). You would not believe how often that happens (and I hear just
> > a small subset).
> 
> Correct, disaster recovery would be made slightly more manual by
> requiring a second channel to verify the request (e.g., setting a DNS
> record in addition to HTTPS). Although this might happen quite
> frequently, we strongly believe that we should opt for secure by
> default, instead of allowing a practical attack because of disaster
> recovery eventualities. The burden of our solution is not significant
> in that case (see below) and what is to be expected for recovering
> lost credentials (e.g., your email account): a second channel.

"slightly"? You obviously have very different definition of "slightly"
thn me.

At scale like LE, this would have large impact. There would be huge
amount of users locked out. They can't (mostly skill) to validate
using DNS or any othe second factor.

If I knew I was dealing with anything like that, I would basically
take HPKP-level precauntions (which are far beyond most users). And
which will lock the key in place (well, the key would be locked
for me anyway)


Basically, I think this would cause huge amount of problems for users
without much gain, hurting HTTPS adoption greatly.




-Ilari

___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme

Re: [Acme] Stale DNS and high-risk issuance

[Kevin Borgolte]

Hi Jacob,

On Thu, Jul 20, 2017 at 11:46 AM, Jacob Hoffman-Andrews  wrote:
>
> On 07/20/2017 12:18 AM, Kevin Borgolte wrote:
> > we are currently conducting a measurement study about DNS staleness
> > issues with a focus on IP address churning.
>
> Excellent, I look forward to reading it!

Great! We will be happy to reach out to you in a separate thread if
you are interested in reading a draft version, and providing some
feedback?

> > We encountered the issue that Let's Encrypt (and ACME in general) can
> > be (ab)used to request and receive a valid certificate for a domain,
> > as long as the attacker obtains access to the IP address to which the
> > DNS record points.
>
> This is true of HTTP and TLS based validation in general. I think the
> place to address these issues is not in the ACME standard, but
> potentially as a separate IETF draft with the goal of later adoption by
> the CA/Browser Forum. For instance, if HTTP and TLS validation were
> required to contact every IP address they found in DNS, that would
> mitigate the stale DNS issue you raise. On the other hand, it would hurt
> usability and reliability. And my intuition is that truly "high risk"
> domains (those with lots of users, or important data) do not tend to
> leave stale entries in DNS, because it would harm performance. Hopefully
> your study will shed light on that assumption!

Thanks for the suggestion about the IETF draft, we will look into. I
do not think anyone of us has any prior IETF draft experience, so
hopping onto an existing one would be the better option for us :-)

>From our other response to Ilari:

Our definition of stale DNS seems be slightly different or broader
than yours: specifically, we also consider domains as stale for which
there is only A record, and not only cases in which a single IP
address out of multiple might be stale. For instance, a practice we
encountered quite often is that VM instances on a cloud provider are
freed and the associated IP addresses are released, but the respective
DNS records are not updated for some days. In turn, someone else can
claim the same IP address and then request a certificate for the
domain. In practice, some hours are often sufficient to claim a
released address to successfully launch the attack (which might be
possible with caching depending on the used nameservers) and we have
seen some cases with customer-specific sub-domains being susceptible
to the attack (which might be a worthwhile phishing target).

We definitely agree with you that requiring all IP addresses they
found in DNS to complete HTTP/TLS challenges to combat stale DNS is
not a solution, simply for performance and reliability reasons as you
mentioned.

> > We also propose, focusing on high-risk targets, a stricter issuance
> > policy:
> >
> > If a valid certificate (e.g., issued by the same operator or a set of
> > operators, checked via CT logs) exists for the requested domain, then
> > the current challenge should be signed by the key. If the last
> > certificate has expired, a grace period set by operators could apply
> > (e.g., 1 month or 3 months). If the expiration date has passed a long
> > time ago, or if no grace period is used by the operator, then a second
> > channel should be used to verify the request (e.g., DNS CRP).
>
> As Ilari mentioned, this was originally in the plan for ACME, but we
> removed it because it was not fully baked. I think it's definitely
> valuable to consider a successor as a separate document. Currently the
> idea I have running around my head is to specify an HTTPS challenge
> alongside the HTTP challenge, where the server expects to receive a
> currently valid certificate that chains either to its own roots or to
> some other roots it chooses to trust. This would be an optional extra
> level of validation that CAs could apply to domains they consider high
> risk. Upsides: Easy to implement for the both the CA and the Subscriber.
> Downsides: If all certificate keys are lost, issuance would be blocked;
> HTTPS validation still assumes that it's hard for third parties to
> upload unauthorized files to certain paths on the web server.

Possibly overly naive, but a solution here could be:

* Adapt the HTTP challenge to HTTPS, requiring a valid certificate chain.
* Upon receiving a certificate request, offer HTTPS and a second
challenge that is considered more trustworthy (DNS? out-of-band?).
* Default to attempting HTTPS, fail if the certificate chain is not
following the stricter policy.
* The remaining challenge is DNS/out-of-band.

It would introduce a stricter policy, but would not require changing
any language in the current draft, but only require adding two
sub-sections:

1. HTTPS challenge
2. How and why to use a stricter policy

Furthermore, a CA could default to HTTPS over HTTP if it has issued a
certificate to the same domain in the past.

Best,
Kevin

On Thu, Jul 20, 2017 at 11:46 AM, Jacob Hoffman-Andrews  wrote:
> On 07/20/2017 

Re: [Acme] Stale DNS and high-risk issuance

[Jacob Hoffman-Andrews]

On 07/20/2017 12:18 AM, Kevin Borgolte wrote:
> we are currently conducting a measurement study about DNS staleness
> issues with a focus on IP address churning.
Excellent, I look forward to reading it!
> We encountered the issue that Let's Encrypt (and ACME in general) can
> be (ab)used to request and receive a valid certificate for a domain,
> as long as the attacker obtains access to the IP address to which the
> DNS record points.
This is true of HTTP and TLS based validation in general. I think the
place to address these issues is not in the ACME standard, but
potentially as a separate IETF draft with the goal of later adoption by
the CA/Browser Forum. For instance, if HTTP and TLS validation were
required to contact every IP address they found in DNS, that would
mitigate the stale DNS issue you raise. On the other hand, it would hurt
usability and reliability. And my intuition is that truly "high risk"
domains (those with lots of users, or important data) do not tend to
leave stale entries in DNS, because it would harm performance. Hopefully
your study will shed light on that assumption!
> We also propose, focusing on high-risk targets, a stricter issuance
> policy:
>
> If a valid certificate (e.g., issued by the same operator or a set of
> operators, checked via CT logs) exists for the requested domain, then
> the current challenge should be signed by the key. If the last
> certificate has expired, a grace period set by operators could apply
> (e.g., 1 month or 3 months). If the expiration date has passed a long
> time ago, or if no grace period is used by the operator, then a second
> channel should be used to verify the request (e.g., DNS CRP).
As Ilari mentioned, this was originally in the plan for ACME, but we
removed it because it was not fully baked. I think it's definitely
valuable to consider a successor as a separate document. Currently the
idea I have running around my head is to specify an HTTPS challenge
alongside the HTTP challenge, where the server expects to receive a
currently valid certificate that chains either to its own roots or to
some other roots it chooses to trust. This would be an optional extra
level of validation that CAs could apply to domains they consider high
risk. Upsides: Easy to implement for the both the CA and the Subscriber.
Downsides: If all certificate keys are lost, issuance would be blocked;
HTTPS validation still assumes that it's hard for third parties to
upload unauthorized files to certain paths on the web server.

___
Acme mailing list
Acme@ietf.org
https://www.ietf.org/mailman/listinfo/acme