RE: MTU handling in 6RD deployments

[Templin, Fred L]

Hi again,

> Second (and more importantly) reassembly is not needed
> for packets of any size if the path can pass a 1500 byte ping packet.

I should have qualified this by saying that the mechanism still
works even if the BR responds to pings subject to rate limiting.

Thanks - Fred
fred.l.temp...@boeing.com

Re: MTU handling in 6RD deployments

[Anfinsen, Ragnar]

On 07.01.14 17:10, "jean-francois.tremblay...@videotron.com"
 wrote:


>> How many users use your 6rd BR (pr. BR if many)?
>
>50k on a pair of ASR1002-5G, but the second is mostly idle.

We have about 2K for the time being as we do opt-in.

> 
>
>> How does the rate-limiting (drops) numbers look at your side?
>
>Actually, it's quite high (over 50%). I gave up on reaching zero here.
>
>The nature of the traffic seems to be bursty enough that getting to zero
>will be nearly impossible.

As of right now, we have a IPv6 ICMP packet drop of 0,09%. If you want to
hit the 0% mark, you must disable rate-limiting. However, seen from an
operational point of view turning off ICMP rate-limiting makes your 6rd BR
vulnerable for ICMP ping attacks.

/Ragnar

RE: MTU handling in 6RD deployments

[Templin, Fred L]

Hi Tore,

> -Original Message-
> From: Tore Anderson [mailto:t...@fud.no]
> Sent: Tuesday, January 07, 2014 9:57 AM
> To: Templin, Fred L; IPv6 Ops list
> Subject: Re: MTU handling in 6RD deployments
> 
> * Templin, Fred L
> 
> > 6RD could use SEAL the same as any tunneling technology. SEAL makes
> > sure that packets up to 1500 get through no matter what, and lets
> > bigger packets through (as long as they fit the first-hop MTU) with
> > the expectation that hosts sending the bigger packets know what they
> > are doing. It works as follows:
> >
> >   - tunnel ingress pings the egress with a 1500 byte ping
> >   - if the ping succeeds, the path MTU is big enough to
> > accommodate 1500s w/o fragmentation
> >   - if the ping fails, use fragmentation/reassembly to
> > accommodate 1500 and smaller
> >   - end result - IPv6 hosts always see an MTU of at least 1500
> 
> In order for the BR to support reassembly it must maintain state. That's
> going to have a very negative impact on its scaling properties...

A couple of things about this. First, reassembly is used only for packets
in the range of 1280-1500 bytes (smaller and larger packets are passed
w/o fragmentation). Second (and more importantly) reassembly is not needed
for packets of any size if the path can pass a 1500 byte ping packet. So
(as Ole said a few messages back) if the 6rd domain MTU can be made to be
>=1520 the fragmentation and reassembly process is suppressed and only
whole packets are transmitted.

Thanks - Fred
fred.l.temp...@boeing.com

> Tore

Re: MTU handling in 6RD deployments

[Tore Anderson]

* Templin, Fred L

> 6RD could use SEAL the same as any tunneling technology. SEAL makes
> sure that packets up to 1500 get through no matter what, and lets
> bigger packets through (as long as they fit the first-hop MTU) with
> the expectation that hosts sending the bigger packets know what they
> are doing. It works as follows:
> 
>   - tunnel ingress pings the egress with a 1500 byte ping
>   - if the ping succeeds, the path MTU is big enough to
> accommodate 1500s w/o fragmentation
>   - if the ping fails, use fragmentation/reassembly to
> accommodate 1500 and smaller
>   - end result - IPv6 hosts always see an MTU of at least 1500

In order for the BR to support reassembly it must maintain state. That's
going to have a very negative impact on its scaling properties...

Tore

Re: MTU handling in 6RD deployments

[Jean-Francois . TremblayING]

> How many users use your 6rd BR (pr. BR if many)?

50k on a pair of ASR1002-5G, but the second is mostly idle. 

> How does the rate-limiting (drops) numbers look at your side?

Actually, it's quite high (over 50%). I gave up on reaching zero here. 
The nature of the traffic seems to be bursty enough that getting to zero 
will be nearly impossible. 

> We hit a problem where the default rate-limiting (100/10) where way to
> aggressive. Right now, we have reduced it to minimum (1/200), but we 
still
> see some drops. Next step would be to turn rate-limiting off, but from 
an
> operational perspective, it does not taste very good.

Agreed. We might actually try these numbers or even disabling it and see 
what happens CPU-wise. 

/JF

RE: MTU handling in 6RD deployments

[Templin, Fred L]

6RD could use SEAL the same as any tunneling technology. SEAL makes
sure that packets up to 1500 get through no matter what, and lets
bigger packets through (as long as they fit the first-hop MTU) with
the expectation that hosts sending the bigger packets know what they
are doing. It works as follows:

  - tunnel ingress pings the egress with a 1500 byte ping
  - if the ping succeeds, the path MTU is big enough to
accommodate 1500s w/o fragmentation
  - if the ping fails, use fragmentation/reassembly to
accommodate 1500 and smaller
  - end result - IPv6 hosts always see an MTU of at least 1500

http://tools.ietf.org/html/draft-templin-intarea-seal

Thanks - Fred
fred.l.temp...@boeing.com

> -Original Message-
> From: ipv6-ops-bounces+fred.l.templin=boeing@lists.cluenet.de 
> [mailto:ipv6-ops-
> bounces+fred.l.templin=boeing@lists.cluenet.de] On Behalf Of Tore Anderson
> Sent: Tuesday, January 07, 2014 3:38 AM
> To: IPv6 Ops list
> Subject: MTU handling in 6RD deployments
> 
> Hi list,
> 
> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
> Free, Swisscom, others?) are using to alleviate any problems stemming
> from the reduced IPv6 MTU? Some possibilities that come to mind are:
> 
> * Having the 6RD CPE lower the TCP MSS value of SYN packets as they
> enter/exit the tunnel device
> * Having the 6RD BR lower the TCP MSS value in the same way as above
> * Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options
> * Several (or all) of the above in combination
> 
> Also, given that some ISPs offer [only] Layer-2 service and expect/allow
> their customers to bring their own Layer-3 home gateway if they want
> one, I would find it interesting to learn if any of the most common
> off-the-shelf home gateway products (that enable 6RD by default) also
> implement any such tricks by default or not.
> 
> Tore

Re: MTU handling in 6RD deployments

[Simon Perreault]

Le 2014-01-07 10:18, Mark Townsley a écrit :


And generating stinkin' ICMPv6 too big messages ends up being perhaps the most 
significant scaling factor of a 6rd BR deployment...


The worst thing is a lot of content providers will simply ignore those 
too bigs you worked so hard to produce... *sigh*


Native IPv6 FTW.

Simon
--
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source--> http://ecdysis.viagenie.ca
STUN/TURN server   --> http://numb.viagenie.ca

Re: MTU handling in 6RD deployments

[Mark Townsley]

And generating stinkin' ICMPv6 too big messages ends up being perhaps the most 
significant scaling factor of a 6rd BR deployment...

- Mark

On Jan 7, 2014, at 3:59 PM, Simon Perreault wrote:

> Le 2014-01-07 08:46, jean-francois.tremblay...@videotron.com a écrit :
>> In the list of "tricks", you might want to add:
>> * Slightly raise the ICMPv6 rate-limit values for your 6RD BR (we do 50/20)
> 
> Yeah, this is really problematic. When IPv6 packets arrive at the BR from the 
> Internet, the BR needs to send too bigs so that the remote node can do PMTUD 
> correctly and figure out the 1480 MTU. If you rate-limit those too bigs, you 
> create black holes. You need to expect a lot of too bigs to be generated by 
> the BR in regular operation, even if the CPE uses tricks such as TCP MSS 
> adjustment or advertising 1480 in RA, because we still need to live with 
> non-TCP traffic and nodes that don't understand the MTU param in RAs.
> 
> Simon
> -- 
> DTN made easy, lean, and smart --> http://postellation.viagenie.ca
> NAT64/DNS64 open-source--> http://ecdysis.viagenie.ca
> STUN/TURN server   --> http://numb.viagenie.ca

Re: MTU handling in 6RD deployments

[Simon Perreault]

Le 2014-01-07 08:46, jean-francois.tremblay...@videotron.com a écrit :

In the list of "tricks", you might want to add:
* Slightly raise the ICMPv6 rate-limit values for your 6RD BR (we do 50/20)


Yeah, this is really problematic. When IPv6 packets arrive at the BR 
from the Internet, the BR needs to send too bigs so that the remote node 
can do PMTUD correctly and figure out the 1480 MTU. If you rate-limit 
those too bigs, you create black holes. You need to expect a lot of too 
bigs to be generated by the BR in regular operation, even if the CPE 
uses tricks such as TCP MSS adjustment or advertising 1480 in RA, 
because we still need to live with non-TCP traffic and nodes that don't 
understand the MTU param in RAs.


Simon
--
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source--> http://ecdysis.viagenie.ca
STUN/TURN server   --> http://numb.viagenie.ca

Re: MTU handling in 6RD deployments

[Anfinsen, Ragnar]

On 07.01.14 14:46, "jean-francois.tremblay...@videotron.com"
 wrote:


>In the list of "tricks", you might want to add:
>* Slightly raise the ICMPv6 rate-limit values for your 6RD BR (we do
>50/20)

How many users use your 6rd BR (pr. BR if many)?

>Too bigs remain quite common however...
>#sh ipv6 traffic | in too
>   11880 encapsulation failed, 0 no route, 3829023354 too big
>#sh ver | in upt 
>uptime is 2 years, 4 weeks, 5 days, 4 hours, 3 minutes

How does the rate-limiting (drops) numbers look at your side?

We hit a problem where the default rate-limiting (100/10) where way to
aggressive. Right now, we have reduced it to minimum (1/200), but we still
see some drops. Next step would be to turn rate-limiting off, but from an
operational perspective, it does not taste very good.

 
Best Regards 
Ragnar Anfinsen
 
Chief Architect CPE
IPv6 Architect
Netinfrastructure & Communication
Technology and Innovation
Altibox AS 

Phone: +47 51 90 80 00
Phone direct: +47 51 90 82 35
Mobile +47 93 48 82 35
E-mail: ragnar.anfin...@altibox.no 
Skype: ragnar_anfinsen
www.altibox.no 

Re: MTU handling in 6RD deployments

[Jean-Francois . TremblayING]

> De : Gert Doering 
> 
> "Have a higher IPv4 MTU between the 6rd tunnel endpoints"  sounds like
> a nice solution an ISP could deploy.

Docsis MTU is 1518 bytes, so that won't happen any time soon in the cable 
world. 
(Docsis 3.1 is higher at 2000 bytes, but that's years away)

/JF

Re: MTU handling in 6RD deployments

[Tore Anderson]

* Gert Doering

> "Have a higher IPv4 MTU between the 6rd tunnel endpoints"  sounds like
> a nice solution an ISP could deploy.

True, well, in theory anyway.

The reason I didn't include this in my list was that considering the
whole point of 6RD is to be able to bypass limitations of old rusty gear
that don't support fancy features like IPv6...the chances of that old
rusty gear being able to reliably support jumbo frames wasn't very high
either.

Tore

RE: MTU handling in 6RD deployments

[Jean-Francois . TremblayING]

Hi Tore. 

> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
> Free, Swisscom, others?) are using to alleviate any problems stemming
> from the reduced IPv6 MTU? Some possibilities that come to mind are:
> 
> * Having the 6RD CPE lower the TCP MSS value of SYN packets as they
> enter/exit the tunnel device
> * Having the 6RD BR lower the TCP MSS value in the same way as above
> * Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options
> * Several (or all) of the above in combination

Our managed CPEs (D-Links) send (IPv4 MTU) - 20 bytes in RAs, usually 
1480.

In the list of "tricks", you might want to add: 
* Slightly raise the ICMPv6 rate-limit values for your 6RD BR (we do 
50/20)

I haven't seen IPv6 MSS clamping in the wild yet (it was discussed on 
this list a year ago). 

> Also, given that some ISPs offer [only] Layer-2 service and expect/allow
> their customers to bring their own Layer-3 home gateway if they want
> one, I would find it interesting to learn if any of the most common
> off-the-shelf home gateway products (that enable 6RD by default) also
> implement any such tricks by default or not.

>From off-the-shelf, we see mostly D-Links and Cisco/Linksys/Belkin 
with option 212 support. A few Asus models started showing up in the 
stats in 2013 I believe. Last time I checked, all models supporting 
option 212 also reduced their MTU properly (YMMV here, that was almost a 
year ago).

Too bigs remain quite common however... 
#sh ipv6 traffic | in too
   11880 encapsulation failed, 0 no route, 3829023354 too big
#sh ver | in upt
uptime is 2 years, 4 weeks, 5 days, 4 hours, 3 minutes

If 6lab's data is right, roughly half of Canada's IPv6 users go through 
that box (50k users).

/JF

Re: MTU handling in 6RD deployments

[Gert Doering]

Hi,

On Tue, Jan 07, 2014 at 12:37:39PM +0100, Tore Anderson wrote:
> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
> Free, Swisscom, others?) are using to alleviate any problems stemming
> from the reduced IPv6 MTU? Some possibilities that come to mind are:

"Have a higher IPv4 MTU between the 6rd tunnel endpoints"  sounds like
a nice solution an ISP could deploy.

But I've long given up hope after everybody seems to agree that 1492
is just fine.

Gert Doering
-- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AGVorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14  Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444   USt-IdNr.: DE813185279

Re: MTU handling in 6RD deployments

[Mikael Abrahamsson]

On Tue, 7 Jan 2014, Mark Townsley wrote:

Note I've heard some ISPs consider running Jumbo Frames under the covers 
so that IPv4 could carry 1520 and 1500 would be possible for IPv6, but 
have not yet seen that confirmed to me in practice.


Unless this is done in a very controlled environment I'd say this is 
bordering on the impossible. There are so many failure points for a jumbo 
solution it's scary. Most of them is also silent failure of PMTUD, 
basically blackholing of traffic.


Yes, it can be done of course, but I'd say operationally it's easier to 
just drop the MTU to 1480 and known working, than the jumbo alternative.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: MTU handling in 6RD deployments

[Mark Townsley]

On Jan 7, 2014, at 12:56 PM, Emmanuel Thierry wrote:

> Hello,
> 
> Le 7 janv. 2014 à 12:37, Tore Anderson a écrit :
> 
>> Hi list,
>> 
>> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
>> Free, Swisscom, others?) are using to alleviate any problems stemming
>> from the reduced IPv6 MTU? Some possibilities that come to mind are:
>> 
>> * Having the 6RD CPE lower the TCP MSS value of SYN packets as they
>> enter/exit the tunnel device
>> * Having the 6RD BR lower the TCP MSS value in the same way as above
>> * Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options
> 
> For your information, i see an advertised mtu of 1480 on my WiFi interface 
> with the Free CPE.

Section 9.1 of RFC 5969:

   If the MTU is well-managed such that the IPv4 MTU on the CE WAN side
   interface is set so that no fragmentation occurs within the boundary
   of the SP, then the 6rd Tunnel MTU should be set to the known IPv4
   MTU minus the size of the encapsulating IPv4 header (20 bytes).  For
   example, if the IPv4 MTU is known to be 1500 bytes, the 6rd Tunnel
   MTU might be set to 1480 bytes.  Absent more specific information,
   the 6rd Tunnel MTU SHOULD default to 1280 bytes.

Note I've heard some ISPs consider running Jumbo Frames under the covers so 
that IPv4 could carry 1520 and 1500 would be possible for IPv6, but have not 
yet seen that confirmed to me in practice. 

- Mark

> 
>> * Several (or all) of the above in combination
>> 
>> Also, given that some ISPs offer [only] Layer-2 service and expect/allow
>> their customers to bring their own Layer-3 home gateway if they want
>> one, I would find it interesting to learn if any of the most common
>> off-the-shelf home gateway products (that enable 6RD by default) also
>> implement any such tricks by default or not.
>> 
> 
> Best regards
> Emmanuel Thierry
> 

Re: MTU handling in 6RD deployments

[Ole Troan]

> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
> Free, Swisscom, others?) are using to alleviate any problems stemming
> from the reduced IPv6 MTU? Some possibilities that come to mind are:
> 
> * Having the 6RD CPE lower the TCP MSS value of SYN packets as they
> enter/exit the tunnel device
> * Having the 6RD BR lower the TCP MSS value in the same way as above
> * Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options
> * Several (or all) of the above in combination

ensure the 6rd domain MTU is >=1520.

> Also, given that some ISPs offer [only] Layer-2 service and expect/allow
> their customers to bring their own Layer-3 home gateway if they want
> one, I would find it interesting to learn if any of the most common
> off-the-shelf home gateway products (that enable 6RD by default) also
> implement any such tricks by default or not.

cheers,
Ole



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: MTU handling in 6RD deployments

[Emmanuel Thierry]

Hello,

Le 7 janv. 2014 à 12:37, Tore Anderson a écrit :

> Hi list,
> 
> Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
> Free, Swisscom, others?) are using to alleviate any problems stemming
> from the reduced IPv6 MTU? Some possibilities that come to mind are:
> 
> * Having the 6RD CPE lower the TCP MSS value of SYN packets as they
> enter/exit the tunnel device
> * Having the 6RD BR lower the TCP MSS value in the same way as above
> * Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options

For your information, i see an advertised mtu of 1480 on my WiFi interface with 
the Free CPE.

> * Several (or all) of the above in combination
> 
> Also, given that some ISPs offer [only] Layer-2 service and expect/allow
> their customers to bring their own Layer-3 home gateway if they want
> one, I would find it interesting to learn if any of the most common
> off-the-shelf home gateway products (that enable 6RD by default) also
> implement any such tricks by default or not.
> 

Best regards
Emmanuel Thierry

Re: IPv6 broken on Fedora 20?

[Hannes Frederic Sowa]

On Tue, Jan 07, 2014 at 12:49:15PM +0100, Hannes Frederic Sowa wrote:
> Yes it is and I fixed that yesterday. I guess, I should ask that the patch
> should be pushed to stable.

Sorry, forgot the link:
https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=88ad31491e21f5dec347911d9804c673af414a09

Greetings,

  Hannes

Re: IPv6 broken on Fedora 20?

[Hannes Frederic Sowa]

On Tue, Jan 07, 2014 at 12:42:43PM +0100, Tore Anderson wrote:
> * Hannes Frederic Sowa
> 
> > It also had some affect of anycast address generation.
> > 
> >> But you are right, essentially it should work but some assumptions were
> >> made in the kernel which should have been checked first.
> > 
> > I guess they're switching back to 64 while suppressing automatically addding
> > prefix routes:
> > 
> >   
> > 
> > This feature should also be available in iproute then.
> 
> Could you elaborate on the anycast address generation problem?

Kernel did also install an subnet-all-router anycast address if the
prefixlen was 128. If you have NM and also e.g. libvirt, which may
enable ipv6 forwarding, the same /128 got installed as an anycast address
(see /proc/net/anycast6). I did not see any breakage, but it could defer ndisc
responses.

> Reason I'm asking is that even though the patch you linked to allow NM
> to return to adding /64s in the case of SLAAC, there's still DHCPv6
> IA_NA which are always /128, yet possibly in combination with arbitrary
> prefix length onlink routes (if PIO exists in RA with A=0, L=1). I'm
> thinking that perhaps this anycast address generation problem could be
> present in that case too?

Yes it is and I fixed that yesterday. I guess, I should ask that the patch
should be pushed to stable.

Greetings,

  Hannes

Re: IPv6 broken on Fedora 20?

[Tore Anderson]

* Hannes Frederic Sowa

> It also had some affect of anycast address generation.
> 
>> But you are right, essentially it should work but some assumptions were
>> made in the kernel which should have been checked first.
> 
> I guess they're switching back to 64 while suppressing automatically addding
> prefix routes:
> 
>   
> 
> This feature should also be available in iproute then.

Could you elaborate on the anycast address generation problem?

Reason I'm asking is that even though the patch you linked to allow NM
to return to adding /64s in the case of SLAAC, there's still DHCPv6
IA_NA which are always /128, yet possibly in combination with arbitrary
prefix length onlink routes (if PIO exists in RA with A=0, L=1). I'm
thinking that perhaps this anycast address generation problem could be
present in that case too?

Tore

MTU handling in 6RD deployments

[Tore Anderson]

Hi list,

Does anyone know what tricks, if any, the major 6RD deployments (AT&T,
Free, Swisscom, others?) are using to alleviate any problems stemming
from the reduced IPv6 MTU? Some possibilities that come to mind are:

* Having the 6RD CPE lower the TCP MSS value of SYN packets as they
enter/exit the tunnel device
* Having the 6RD BR lower the TCP MSS value in the same way as above
* Having the 6RD CPE advertise a lowered MTU to the LAN in RA Options
* Several (or all) of the above in combination

Also, given that some ISPs offer [only] Layer-2 service and expect/allow
their customers to bring their own Layer-3 home gateway if they want
one, I would find it interesting to learn if any of the most common
off-the-shelf home gateway products (that enable 6RD by default) also
implement any such tricks by default or not.

Tore

Re: IPv6 broken on Fedora 20?

[Hannes Frederic Sowa]

On Thu, Dec 19, 2013 at 07:14:24PM +0100, Hannes Frederic Sowa wrote:
> > Once you're doing that, it's probably easier to handle L=1 by simply
> > adding the on-link route directly, rather than adding the address as a
> > /64 and relying on the kernel to add the route for you. The two should
> > result in the same functionality, though, so I'm don't really understand
> > what's actually broken here.
> 
> I guess it breaks generation of privacy addresses.

It also had some affect of anycast address generation.

> But you are right, essentially it should work but some assumptions were
> made in the kernel which should have been checked first.

I guess they're switching back to 64 while suppressing automatically addding
prefix routes:

  

This feature should also be available in iproute then.

Greetings,

  Hannes