Re: [dns-operations] RRL specified in a stable place?

[Doug Barton]

On 02/04/2013 03:16 PM, Paul Hoffman wrote:

On Feb 4, 2013, at 2:32 PM, Edward Lewis 
wrote:


Why an IETF document?


Because that's where implementers look for such documents.

Because there are implementers who are active in the IETF who might
have valuable opinions on what the doc might say that would make it
more valuable.


I agree with these reasons. Even if it was published as something like a 
BCP, it would fall into the IETF category, IMO. I'm not tied to the IETF 
path, but I think it's perfectly reasonable for people to look to us for 
DNS expertise.


As was pointed out elsewhere in the thread, there seems to be confusion, 
even in our community, about what RRL is, and what it isn't, so 
clarifying that would be a good additional purpose for the document 
(above and beyond simply describing what/how/why).



In what way does Response Rate Limiting impact interoperability of
implementations?


It does not. So what? Did you somehow miss all the operational
documents that the IETF helps produce?


I am not sure it would be defined strictly as an interoperability issue, 
but I could see utility in a document that answers the questions that 
may arise downstream when someone sees the effects of RRL and doesn't 
understand what they are seeing.



If this is not an independent submission, how does it fit into a
working group?  The implementations are pretty much out there,
what's to work on?


There are two implementations. Are you saying there should be no
more? Or, if there are more, that the implementers should have no
clue about what earlier implementers thought about?


+1

If this document takes shape I would gladly volunteer to review it.

hth,

Doug
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Vernon Schryver]

> From: Andrew Sullivan 

> > Perhaps I misunderstand, but I think that's wrong in general and based
> > on the persistent and by now very irritating confusion between client
> > rate limiting and response rate limiting (RRL).  
>
> No, it isn't.

Yes, it is.  Please pause to understand the difference between
response rate limiting and client rate limiting or DoS defenses.

> Suppose that DNSprov provides DNS service on behalf of HiProfile.

> Now, suppose that MrISP is a very large US-based cable provider with
> something like (say) 10% of the domestic market.  All MsAttacker needs
> do to cause significant pain is to send a DoS towards DNSprov with
> source addresses of MrISP's resolver querying for HiProfile's names.
> RRL will work, of course, in the sense that it will stop spewing
> garbage.  But it will also rate limit responses to anyone apparently
> coming from that address. 

That is where Andrew Sullivan goes wrong in confounding client rate
limiting at DNSprov with response rate limiting at DNSprov or DoS
defenses at MrISP.

MrISP's resolver should not be asking about HiProfile's RRs more often
than once per TTL.  Since TTLs are always more than 1 second (to
understate the common case by a factor of 100X or 1000X), MrISP's
resolver will only barely affected by RRL provided it gets a response
now and then.  Of course, there are some caveats:

   - if MrISP's resolver is a farm hiding behind a single NAT or
  load balanced address or CIDR block and there are enough
  resolvers in the farm, "it" might ask often enough to be
  rate limited.

   - MrISP's resolver must get some answers.  With the RRL "slip"
  default of 2 and the typical 3 retries, there is an 87.5%
  probability that MrISP's resolver will get a "slipped" or
  truncated response  from DNSprov despite the flood of requests
  for the A RR for HiProfile forged from MrISP's resolver
  If a user at a browser who gets an "page not working" error
  tries a second time, the odds of MrISP's resolver working
  rise to 98.4%.  For at least minutes and usually hours afterwards,
  MrISP's resolver's cache will deliver A RRs for HiProfile and
  not notice anything amiss.

Note also that without RRL at DNSprov, MrISP's resolver will be flooded
with responses from DNSprov to the forged requests.  The DoS defenses
at MrISP are likely to "scrub" most of the responses from DNSprov
whether to real or forged requests, because the responses are a DoS
attack.  Because of that, MrISP's resolver has less chance of resolving
HiProfile without RRL at DNSprov than with RRL at DNSprov.


Vernon Schryverv...@rhyolite.com
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Hoffman]

On Feb 4, 2013, at 2:32 PM, Edward Lewis  wrote:

> Why an IETF document?  

Because that's where implementers look for such documents.

Because there are implementers who are active in the IETF who might have 
valuable opinions on what the doc might say that would make it more valuable.

> In what way does Response Rate Limiting impact interoperability of 
> implementations?

It does not. So what? Did you somehow miss all the operational documents that 
the IETF helps produce?

> If this is not an independent submission, how does it fit into a working 
> group?  The implementations are pretty much out there, what's to work on?

There are two implementations. Are you saying there should be no more? Or, if 
there are more, that the implementers should have no clue about what earlier 
implementers thought about?

> I understand that would be useful is a reference-able document describing the 
> RRL.  That is, something stable and reviewed - and that could be an RFC.   
> But an RFC does not have to come through the IETF.

Nor did anyone say it had to.

Please consider putting your straw man back on the shelf, and maybe help people 
who want to work cooperatively do so.

--Paul Hoffman
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Vixie]

may i suggest that the ratelimits mailing list is a better place for
this argument. but herewith:

Andrew Sullivan wrote:
> Suppose that DNSprov provides DNS service on behalf of HiProfile.
> Suppose that HiProfile has one of those services that is really
> susceptible to the "no response, kill the page load" problems that the
> big web presences -- Amazon, Yahoo, Twitter, &c -- keep worrying
> about. Moreover, because of the usual commercial reasons such services
> have, the TTLs on HiProfile records are short. Now, suppose that MrISP
> is a very large US-based cable provider with something like (say) 10%
> of the domestic market. All MsAttacker needs do to cause significant
> pain is to send a DoS towards DNSprov with source addresses of MrISP's
> resolver querying for HiProfile's names. RRL will work, of course, in
> the sense that it will stop spewing garbage. But it will also rate
> limit responses to anyone apparently coming from that address. This
> means that MsAttacker can cause MrISP's resolvers to be rate-limited
> as long as MsAttacker can keep up the attack for something longer than
> $TTL on some record for HiProfile.

yes.

> In other words, MsAttacker can cause a short but probably effective
> DoS against HiProfile, and with a little work can probably cause
> intermittent outages for a significant percentage of MrISP's customers
> every few minutes. 

no.

> This is perhaps a less bad attack than before, depending on DNSprov's
> provisioning; but it is contractually devastating for DNSprov, who
> promised not to drop queries on the floor for HiProfile, but who has
> dropped such queries on purpose. One can mitigate this to some extent
> with various additional epicycles to the RRL approach (and I note that
> you've done so, and congratulate you in your acumen and creativity),
> but one cannot solve the fundamental issue, which has to do with very
> high-value targets and very large communities behind certain
> high-value resolvers. It's a trade-off. RRL works well in some --
> maybe most -- cases, but it's not something one can do in others.
> That's hardly surprising, I think. 

wrong.

factually, rrl can't fix everything, but it makes no case worse than it
would otherwise be. i've heard from a lot of experts who said that rrl
creates a new DoS vector, but none of those claims has held up.

in the case you outline, there would already not be (that is, without
RRL) service for HiProfile at MrISP, but due to congestion rather than
RRL's prevention of that congestion.

if anyone wants to blame RRL for not solving all the world's problems,
my shoulders are very broad -- bring it on. but if you want to blame RRL
for creating a new problem -- tell me more.

paul

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Vixie]

Edward Lewis wrote:
> Why an /IETF/ document?  In what way does Response Rate Limiting
> impact interoperability of implementations?
>
> If this is not an independent submission, how does it fit into a
> working group?  The implementations are pretty much out there, what's
> to work on?
>
> I understand that would be useful is a reference-able document
> describing the RRL.  That is, something stable and reviewed - and that
> could be an RFC.   But an RFC does not have to come through the IETF.

gentlemen, this is gangster on gangster violence. please figure out
which i to dot and which t to cross, and let me know. i promise on my
honour to do my best to appease the internet standards gods in their
current invocation.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Vixie]

Edward Lewis wrote:
> Why an /IETF/ document?  In what way does Response Rate Limiting
> impact interoperability of implementations?
>
> If this is not an independent submission, how does it fit into a
> working group?  The implementations are pretty much out there, what's
> to work on?
>
> I understand that would be useful is a reference-able document
> describing the RRL.  That is, something stable and reviewed - and that
> could be an RFC.   But an RFC does not have to come through the IETF.

gentlemen, this is gangster on gangster violence. please figure out
which i to dot and which t to cross, and let me know. i promise on my
honour to do my best to appease the internet standards gods in their
current invocation.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Andrew Sullivan]

Hi,

On Mon, Feb 04, 2013 at 09:02:14PM +, Vernon Schryver wrote:

> >  Consider that, if you spoof $ISP's resolver addresses
> > and perform one of these attacks, then $ISP gets at least degraded
> > service during the rate limit period.
> 
> Perhaps I misunderstand, but I think that's wrong in general and based
> on the persistent and by now very irritating confusion between client
> rate limiting and response rate limiting (RRL).  

No, it isn't.

Suppose that DNSprov provides DNS service on behalf of HiProfile.
Suppose that HiProfile has one of those services that is really
susceptible to the "no response, kill the page load" problems that the
big web presences -- Amazon, Yahoo, Twitter, &c -- keep worrying
about.  Moreover, because of the usual commercial reasons such
services have, the TTLs on HiProfile records are short.

Now, suppose that MrISP is a very large US-based cable provider with
something like (say) 10% of the domestic market.  All MsAttacker needs
do to cause significant pain is to send a DoS towards DNSprov with
source addresses of MrISP's resolver querying for HiProfile's names.
RRL will work, of course, in the sense that it will stop spewing
garbage.  But it will also rate limit responses to anyone apparently
coming from that address.  This means that MsAttacker can cause
MrISP's resolvers to be rate-limited as long as MsAttacker can keep up
the attack for something longer than $TTL on some record for
HiProfile.  In other words, MsAttacker can cause a short but probably
effective DoS against HiProfile, and with a little work can probably
cause intermittent outages for a significant percentage of MrISP's
customers every few minutes.  

This is perhaps a less bad attack than before, depending on DNSprov's
provisioning; but it is contractually devastating for DNSprov, who
promised not to drop queries on the floor for HiProfile, but who has
dropped such queries on purpose.  One can mitigate this to some extent
with various additional epicycles to the RRL approach (and I note that
you've done so, and congratulate you in your acumen and creativity),
but one cannot solve the fundamental issue, which has to do with very
high-value targets and very large communities behind certain
high-value resolvers.

It's a trade-off.  RRL works well in some -- maybe most -- cases, but
it's not something one can do in others.  That's hardly surprising, I
think.

> If you think not letting the IETF "improve" and then freeze the
> specification will lead to fragmentation and disparate implementations,
> then you should oppose an RFC.

I wasn't opposing anything.  I was trying to clear the ground so that
we understood, going in, what the bureaucratic hurdles would be so
that we wouldn't have to worry about them when we settled on an
approach to pursuing an RFC.  I'm neither here to defend nor bury the
IETF.  I intended only to ask what one wanted to do there _before_ we
got the IETF process wheel griding our bones to dust.

Best,

A

-- 
Andrew Sullivan
a...@anvilwalrusden.com
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Edward Lewis]

Why an IETF document?  In what way does Response Rate Limiting impact 
interoperability of implementations?

If this is not an independent submission, how does it fit into a working group? 
 The implementations are pretty much out there, what's to work on?

I understand that would be useful is a reference-able document describing the 
RRL.  That is, something stable and reviewed - and that could be an RFC.   But 
an RFC does not have to come through the IETF.

On Feb 4, 2013, at 14:39, Andrew Sullivan wrote:
> On Mon, Feb 04, 2013 at 10:54:36AM -0800, Paul Hoffman wrote:
>> We now have two implementation of response rate limiting (RRL). In order for 
>> it to be widely adopted, an Internet-Draft followed by an RFC would be 
>> Really Helpful.
> What track do you expect this to go along?  Is this a DNSOP draft?

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis 
NeuStarYou can leave a voice message at +1-571-434-5468

There are no answers - just tradeoffs, decisions, and responses.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Vernon Schryver]

> >   ... an Internet-Draft followed by an RFC would be Really Helpful.

> What track do you expect this to go along?  Is this a DNSOP draft?

Those are best boring details except to those with non-technical
interests in the IETF.

> Because the implementations are really just a way of using existing
> parts of the specifications in creative ways. 

That covers a lot of RFCs.

>(They're also risky for
> some operators. 

That covers a lot of RFCs.

>  Consider that, if you spoof $ISP's resolver addresses
> and perform one of these attacks, then $ISP gets at least degraded
> service during the rate limit period.

Perhaps I misunderstand, but I think that's wrong in general and based
on the persistent and by now very irritating confusion between client
rate limiting and response rate limiting (RRL).  In addition, ISPs
have reported that installing and turning on RRL has restored DNS
service that had been degraded by apparent DNS refection DoS attacks.
While your DNS servers are trying to respond to Mqps of bogus requests,
they are often not only flooding the DoS victim but also not getting
out other responses.


> it's not a panacea either,

That cover a lot of RFCs.

>and certainly cannot be considered a BCP
> for all use cases.)

Ok, so don't make as a BCP.  Let it be Informational.  Or don't advance
it after publishing it as an I-D.  Keeping change control out of the
IETF will not slow the spread of the idea or harm interoperability
(which in the old days was the reason for RFCs).

If you think not letting the IETF "improve" and then freeze the
specification will lead to fragmentation and disparate implementations,
then you should oppose an RFC.  Without an RFC, there is more room for
better ideas.  Because there is no directly involved on-the-wire
protocol, there is much less need for an RFC.

Personally, I think it would be nice if it were published at least as
an I-D ensure that the idea reaches more potential implementors.  However,
it would not be a big deal if the IETF doesn't want it even as an I-D.


The one thing the IETF really should do (if it has not become
interchangable with the ISO/ITU/UN) is to add two check list items
before advancing future protocols:

   - all servers MUST deal appropriately with excessive requests such 
  as by rate limiting by client, network, request, and/or type of
  request.  This is particularly important for services that
  do not require long lived state.

   - all clients MUST rate limit their requests, both retries and de novo,
  including using at least exponential back-off.


Vernon Schryverv...@rhyolite.com
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Vixie]

Paul Hoffman wrote:
> ...
>
> I think it should be Experimental, it should discuss any differences that the 
> BIND and NSD folks have, and it should be an individual submission. 
>
> After than, people can discuss the different approaches over maybe a year, 
> and if there is kinda general agreement, it can come to DNSOP for BCP 
> consideration. If it fails there, the Experimental RFC still lives on.
>
> Old-style IETF (RFC that really requests comments), and only later settling 
> on what to tell the community as "best".

suits me.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Andrew Sullivan]

On Mon, Feb 04, 2013 at 12:31:28PM -0800, Paul Hoffman wrote:
> Old-style IETF (RFC that really requests comments), and only later settling 
> on what to tell the community as "best".
> 

This sounds like an excellent idea to me.

A

-- 
Andrew Sullivan
a...@anvilwalrusden.com
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Hoffman]

On Feb 4, 2013, at 11:39 AM, Andrew Sullivan  wrote:

> On Mon, Feb 04, 2013 at 10:54:36AM -0800, Paul Hoffman wrote:
>> We now have two implementation of response rate limiting (RRL). In order for 
>> it to be widely adopted, an Internet-Draft followed by an RFC would be 
>> Really Helpful.
>> 
> 
> What track do you expect this to go along?  Is this a DNSOP draft?
> Because the implementations are really just a way of using existing
> parts of the specifications in creative ways.  (They're also risky for
> some operators.  Consider that, if you spoof $ISP's resolver addresses
> and perform one of these attacks, then $ISP gets at least degraded
> service during the rate limit period.  For most of us, that's probably
> an acceptable trade off, but not for all operators unfortunately.  So
> it's not a panacea either, and certainly cannot be considered a BCP
> for all use cases.)

I think it should be Experimental, it should discuss any differences that the 
BIND and NSD folks have, and it should be an individual submission. 

After than, people can discuss the different approaches over maybe a year, and 
if there is kinda general agreement, it can come to DNSOP for BCP 
consideration. If it fails there, the Experimental RFC still lives on.

Old-style IETF (RFC that really requests comments), and only later settling on 
what to tell the community as "best".

--Paul Hoffman
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Andrew Sullivan]

On Mon, Feb 04, 2013 at 10:54:36AM -0800, Paul Hoffman wrote:
> We now have two implementation of response rate limiting (RRL). In order for 
> it to be widely adopted, an Internet-Draft followed by an RFC would be Really 
> Helpful.
> 

What track do you expect this to go along?  Is this a DNSOP draft?
Because the implementations are really just a way of using existing
parts of the specifications in creative ways.  (They're also risky for
some operators.  Consider that, if you spoof $ISP's resolver addresses
and perform one of these attacks, then $ISP gets at least degraded
service during the rate limit period.  For most of us, that's probably
an acceptable trade off, but not for all operators unfortunately.  So
it's not a panacea either, and certainly cannot be considered a BCP
for all use cases.)

A

-- 
Andrew Sullivan
a...@anvilwalrusden.com
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Hoffman]

On Feb 4, 2013, at 11:07 AM, Paul Vixie  wrote:

> Paul Hoffman wrote:
>> We now have two implementation of response rate limiting (RRL). In order for 
>> it to be widely adopted, an Internet-Draft followed by an RFC would be 
>> Really Helpful.
> 
> agreed.
> 
>> If this is not possible due to overcommitment on the parts of the authors, I 
>> would be willing to be primary editor of the draft. This work should begin 
>> sooner, not after another implementation has started but gone in a different 
>> direction.
> 
> i have time, and i am willing to upgrade the tech note to an internet
> draft and submit it. it's somewhat out of date but the general idea is
> present at .

Having the Internet Draft be up-to-date and reflect both implementations would 
be great.

This could easily start off as an Experimental RFC.

--Paul Hoffman
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

Re: [dns-operations] RRL specified in a stable place?

[Paul Vixie]

Paul Hoffman wrote:
> We now have two implementation of response rate limiting (RRL). In order for 
> it to be widely adopted, an Internet-Draft followed by an RFC would be Really 
> Helpful.

agreed.

> If this is not possible due to overcommitment on the parts of the authors, I 
> would be willing to be primary editor of the draft. This work should begin 
> sooner, not after another implementation has started but gone in a different 
> direction.

i have time, and i am willing to upgrade the tech note to an internet
draft and submit it. it's somewhat out of date but the general idea is
present at .

paul
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

[dns-operations] RRL specified in a stable place?

[Paul Hoffman]

We now have two implementation of response rate limiting (RRL). In order for it 
to be widely adopted, an Internet-Draft followed by an RFC would be Really 
Helpful.

If this is not possible due to overcommitment on the parts of the authors, I 
would be willing to be primary editor of the draft. This work should begin 
sooner, not after another implementation has started but gone in a different 
direction.

--Paul Hoffman
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
dns-jobs mailing list
https://lists.dns-oarc.net/mailman/listinfo/dns-jobs