Re: Service provider story about tracking down TCP RSTs

[Tarko Tikan]

hey,


But why did the TLS Hello has a TTL lower that the TCP Syn ?

Do you have any information on that ?


Consumer CPEs are typically some BCM reference design where initial TCP 
handshake is handled by linux kernel and everything following (including 
NAT) is handled in SOC.


I've seen those systems not decrement TTL at all, decrement TTL before 
checking if packet is destined to itself etc. This case is weird as 
typically the hardware part is faulty, not the kernel.


--
tarko

Re: Service provider story about tracking down TCP RSTs

[William Herrin]

On Sun, Sep 2, 2018 at 6:49 AM, Bjørn Mork  wrote:
> William Herrin  writes:
>> On Sun, Sep 2, 2018 at 6:06 AM, Bjørn Mork  wrote:
>>> William Herrin  writes:
  https://bill.herrin.us/network/anycasttcp.html
>>>
>>> I didn't see a security section in your document.  Did you consider the
>>> side effects of this sequence number abuse?
>>
>> In the "issues and criticisms" section.
>
> I can see the effect on syn cookies being disussed there, but I don't
> think that covers all concerns wrt more predicatable sequence numbers.
>
> See RFC6528, including its references.

Thanks Bjørn,

I've added several notes in "issues and criticisms" based on that information.

Regards,
Bill Herrin



-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Dirtside Systems . Web: 

Re: Service provider story about tracking down TCP RSTs

[Bjørn Mork]

William Herrin  writes:

> On Sun, Sep 2, 2018 at 6:06 AM, Bjørn Mork  wrote:
>> William Herrin  writes:
>>>  https://bill.herrin.us/network/anycasttcp.html
>>
>> I didn't see a security section in your document.  Did you consider the
>> side effects of this sequence number abuse?
>
> Hi Bjørn,
>
> In the "issues and criticisms" section.

I can see the effect on syn cookies being disussed there, but I don't
think that covers all concerns wrt more predicatable sequence numbers.

See RFC6528, including its references.


Bjørn

Re: Service provider story about tracking down TCP RSTs

[William Herrin]

On Sun, Sep 2, 2018 at 6:06 AM, Bjørn Mork  wrote:
> William Herrin  writes:
>>  https://bill.herrin.us/network/anycasttcp.html
>
> I didn't see a security section in your document.  Did you consider the
> side effects of this sequence number abuse?

Hi Bjørn,

In the "issues and criticisms" section.

Regards,
Bill Herrin


-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Dirtside Systems . Web: 

Re: Service provider story about tracking down TCP RSTs

[nanog]

On 09/02/2018 10:24 AM, James Bensley wrote:
> It is available via the NANOG list archives:
> https://mailman.nanog.org/pipermail/nanog/2018-September/096871.html

But why did the TLS Hello has a TTL lower that the TCP Syn ?

Do you have any information on that ?

Re: Service provider story about tracking down TCP RSTs

[Bjørn Mork]

William Herrin  writes:

> BTW, for anyone concerned about an explosion in state management
> overhead, the TL;DR version is: the anycast node which first accepts
> the TCP connection encodes its identity in the TCP sequence number
> where all the other nodes can statelessly find it in the subsequent
> packets.

I didn't see a security section in your document.  Did you consider the
side effects of this sequence number abuse?


Bjørn

Re: automatic rtbh trigger using flow data

[Baldur Norddahl]

I would redirect the packet to a VRF with one global drop UDP ACL. That
scales perfectly. There is probably many ways to implement such a feature.


søn. 2. sep. 2018 11.07 skrev Ryan Hamel :

> Baldur,
>
>
>
> Modifying the routing table with a next-hop change from a community, is
> different than having a line card filtering packets at layer 4, of course
> most if not all carriers will support it. Instead of doing normal TCAM
> route lookups, you’re getting into packet inspection territory, which is
> something completely different.
>
>
>
> Just quickly reading the ASR 9K documentation, it can only support 3K
> rules per system. Juniper – 8K, Alcatel-Lucent – 512. That’s pretty low
> considering I can put many /32s into a routing table very easily and
> without hassle.
>
>
>
> As I said before, no ISP is going to offer such filtering services for
> free when DDoS mitigation is a cash cow.
>
>
>
> Ryan Hamel
>
>
>
> *From:* NANOG  *On Behalf Of *Baldur Norddahl
> *Sent:* Sunday, September 02, 2018 1:42 AM
> *To:* nanog@nanog.org
> *Subject:* Re: automatic rtbh trigger using flow data
>
>
>
> This is not true. Some of our transits do RTBH for free. For example
> Cogent.
>
>
>
> They will not do FlowSpec. Maybe their equipment can not do it or for some
> other reason.
>
>
>
> However RTBH is a simple routing hack that can be implemented on any
> router. The traffic is dropped right at the edge and is never transported
> on the transit provider network. In that sense it also protects the transit
> network.
>
>
>
> RTBH only for UDP would also be a very simple hack on many routers.
>
>
>
> It might not be FlowSpec, but it may have most of the benefit, in a much
> simplified way.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>
>
>
> søn. 2. sep. 2018 02.39 skrev Ryan Hamel :
>
> No ISP is in the business of filtering traffic unless the client pays the
> hefty fee since someone still has to tank the attack.
>
>
>
> I also don’t think there is destination prefix IP filtering in flowspec,
> which could seriously cause problems.
>
>
>
> *From:* NANOG  *On Behalf Of *Baldur Norddahl
> *Sent:* Saturday, September 01, 2018 5:18 PM
> *To:* nanog@nanog.org
> *Subject:* Re: automatic rtbh trigger using flow data
>
>
>
>
>
> fre. 31. aug. 2018 17.16 skrev Hugo Slabbert :
>
>
>
> I would love an upstream that accepts flowspec routes to get granular
> about
> drops and to basically push "stateless ACLs" upstream.
>
> _keeps dreaming_
>
>
>
>
>
> We just need a signal to drop UDP for a prefix. The same as RTBH but only
> for UDP. This would prevent all volumetric attacks without the end user
> being cut off completely.
>
>
>
> Besides from some games, VPN and VoIP, they would have an almost
> completely normal internet experience. DNS would go through the ISP servers
> and only be affected if the user is using a third party service.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>
>

RE: automatic rtbh trigger using flow data

[Ryan Hamel]

Baldur,

Modifying the routing table with a next-hop change from a community, is 
different than having a line card filtering packets at layer 4, of course most 
if not all carriers will support it. Instead of doing normal TCAM route 
lookups, you’re getting into packet inspection territory, which is something 
completely different.

Just quickly reading the ASR 9K documentation, it can only support 3K rules per 
system. Juniper – 8K, Alcatel-Lucent – 512. That’s pretty low considering I can 
put many /32s into a routing table very easily and without hassle.

As I said before, no ISP is going to offer such filtering services for free 
when DDoS mitigation is a cash cow.

Ryan Hamel

From: NANOG  On Behalf Of Baldur Norddahl
Sent: Sunday, September 02, 2018 1:42 AM
To: nanog@nanog.org
Subject: Re: automatic rtbh trigger using flow data

This is not true. Some of our transits do RTBH for free. For example Cogent.

They will not do FlowSpec. Maybe their equipment can not do it or for some 
other reason.

However RTBH is a simple routing hack that can be implemented on any router. 
The traffic is dropped right at the edge and is never transported on the 
transit provider network. In that sense it also protects the transit network.

RTBH only for UDP would also be a very simple hack on many routers.

It might not be FlowSpec, but it may have most of the benefit, in a much 
simplified way.

Regards

Baldur


søn. 2. sep. 2018 02.39 skrev Ryan Hamel 
mailto:ryan.ha...@quadranet.com>>:
No ISP is in the business of filtering traffic unless the client pays the hefty 
fee since someone still has to tank the attack.

I also don’t think there is destination prefix IP filtering in flowspec, which 
could seriously cause problems.

From: NANOG mailto:nanog-boun...@nanog.org>> On Behalf 
Of Baldur Norddahl
Sent: Saturday, September 01, 2018 5:18 PM
To: nanog@nanog.org
Subject: Re: automatic rtbh trigger using flow data


fre. 31. aug. 2018 17.16 skrev Hugo Slabbert 
mailto:h...@slabnet.com>>:


I would love an upstream that accepts flowspec routes to get granular about
drops and to basically push "stateless ACLs" upstream.

_keeps dreaming_


We just need a signal to drop UDP for a prefix. The same as RTBH but only for 
UDP. This would prevent all volumetric attacks without the end user being cut 
off completely.

Besides from some games, VPN and VoIP, they would have an almost completely 
normal internet experience. DNS would go through the ISP servers and only be 
affected if the user is using a third party service.

Regards

Baldur

Re: automatic rtbh trigger using flow data

[Baldur Norddahl]

This is not true. Some of our transits do RTBH for free. For example Cogent.

They will not do FlowSpec. Maybe their equipment can not do it or for some
other reason.

However RTBH is a simple routing hack that can be implemented on any
router. The traffic is dropped right at the edge and is never transported
on the transit provider network. In that sense it also protects the transit
network.

RTBH only for UDP would also be a very simple hack on many routers.

It might not be FlowSpec, but it may have most of the benefit, in a much
simplified way.

Regards

Baldur


søn. 2. sep. 2018 02.39 skrev Ryan Hamel :

> No ISP is in the business of filtering traffic unless the client pays the
> hefty fee since someone still has to tank the attack.
>
>
>
> I also don’t think there is destination prefix IP filtering in flowspec,
> which could seriously cause problems.
>
>
>
> *From:* NANOG  *On Behalf Of *Baldur Norddahl
> *Sent:* Saturday, September 01, 2018 5:18 PM
> *To:* nanog@nanog.org
> *Subject:* Re: automatic rtbh trigger using flow data
>
>
>
>
>
> fre. 31. aug. 2018 17.16 skrev Hugo Slabbert :
>
>
>
> I would love an upstream that accepts flowspec routes to get granular
> about
> drops and to basically push "stateless ACLs" upstream.
>
> _keeps dreaming_
>
>
>
>
>
> We just need a signal to drop UDP for a prefix. The same as RTBH but only
> for UDP. This would prevent all volumetric attacks without the end user
> being cut off completely.
>
>
>
> Besides from some games, VPN and VoIP, they would have an almost
> completely normal internet experience. DNS would go through the ISP servers
> and only be affected if the user is using a third party service.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>

Re: Service provider story about tracking down TCP RSTs

[James Bensley]

On Sat, 1 Sep 2018 at 21:06, Garrett Skjelstad  wrote:
>
> I would love this as a blog post to link folks that are not nanog members.
>
> -Garrett

Hi Garrett,

It is available via the NANOG list archives:
https://mailman.nanog.org/pipermail/nanog/2018-September/096871.html

I've shared this story to non-list member using that URL.

Thanks for the write up Frank!

Cheers,
James.

Re: Service provider story about tracking down TCP RSTs

[Lee]

On 9/1/18, William Herrin  wrote:
> On Sat, Sep 1, 2018 at 6:11 PM, Lee  wrote:
>> On 9/1/18, William Herrin  wrote:
>>> On Sat, Sep 1, 2018 at 4:00 PM, William Herrin  wrote:
 Better yet, do the job right and build an anycast TCP stack as
 described here: https://bill.herrin.us/network/anycasttcp.html
>>
>> An explosion in state management would be the least of my worries :)
>> I got as far as your Third hook: and thought of this
>>   https://www.jwz.org/doc/worse-is-better.html
>
> Hi Lee,
>
> On a brief tangent: Geographic routing would drastically simplify the
> Internet core, reducing both cost and complexity. You'd need to carry
> only nearby specific routes and a few broad aggregates for
> destinations far away. It will never be implemented, never, because no
> cross-ocean carriers are willing to have their bandwidth stolen when
> the algorithm decides it likes their path better than a paid one. Even
> though the algorithm gets the packets where they're going, and does so
> simply, it does so in a way that's too often incorrect.
>
> Then again, I don't really understand the MIT/New Jersey argument in
> Richard's worse-is-better story.

The "New Jersey" description is more of a caricature than a valid description:
  "I have intentionally caricatured the worse-is-better philosophy to
   convince you that it is obviously a bad philosophy and that the
   New Jersey approach is a bad approach."

I mentally did a 's/New Jersey/Microsoft/' and it made a lot more sense.

> The MIT guy says that a routine
> should handle a common non-fatal exception. The Jersey guy says that
> it's ok for the routine to return a try-again error and expect the
> caller to handle it. Since its trivial to build another layer that
> calls the routine in a loop until it returns success or a fatal error,
> it's more a philosophical argument than a practical one. As long as a
> correct result is consistently achieved in both cases, what's the
> difference?

That it's not always a trivial matter to build another layer.
That your retry layer needs at least a counter or timeout value so it
doesn't retry forever & those values need to be user configurable, so
the re-try layer isn't quite as trivial as it appears at first blush.

> Richard characterized the Jersey argument as, "It is slightly better
> to be simple than correct." I just don't see that in the Jersey
> argument. Every component must be correct. The system of components as
> a whole must be complete. It's slightly better for a component to be
> simple than complete. That's the argument I read and it makes sense to
> me.

Yes, I did a lot of interpreting also.  Then I hit on s/New
Jersey/Microsoft/ and it made a lot more sense to me.

> Honestly, the idea that software is good enough even with known corner
> cases that do something incorrect... I don't know how that survives in
> a world where security-conscious programming is not optional.

Agreed.  I substituted "soft-fail or fail-closed: user has to retry"
for doing something incorrect.

>> I had it much easier with anycast in an enterprise setting.  With
>> anycast servers in data centers A & B, just make sure no site has an
>> equal cost path to A and B.  Any link/ router/ whatever failure & the
>> user can just re-try.
>
> You've delicately balanced your network to achieve the principle that
> even when routing around failures the anycast sites are not
> equidistant from any other site. That isn't simplicity. It's
> complexity hidden in the expert selection of magic numbers.

^shrug^ it seemed simple to me.  And it was real easy to explain,
which is why I thought of that "worse is better" paper.  I took the
New Jersey approach & did what was basically a hack. You took the MIT
approach and created a general solution .. which is not so easy to
explain :)

> Even were that achievable in a network as chaotic as the Internet, is it 
> simpler
> than four trivial tweaks to the TCP stack plus a modestly complex but
> fully automatic user-space program that correctly reroutes the small
> percentage of packets that went astray?

Your four trivial tweaks to the TCP stack are kernel patches - right?
Which seems not at all trivial to me, but if you've got a group of
people that can support & maintain that - good for you!

Regards
Lee