Re: contact with One & One ?

[Paul Stewart]

Yes and my understanding with ECMP on the network side is that this is exactly 
what’s happening … and that’s what Cloudflare blog entry is referring to as 
well …

I need to dig into this further - their code on Github for the fix I don’t 
believe will work in our network architecture…  although we are thinking of a 
redesign on that area so now would be a great chance to fix this too :)

> On Oct 14, 2016, at 8:17 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
> 
>> Up to now, every time I’ve seen this problem was just related to ICMPv6 
>> being filtered, as many folks do in IPv4 …
> 
> I know several cases where the problem was that the load balancer didn't 
> forward the ICMPv6 PTB to the correct host and didn't handle it itself. No 
> filtering, just bad vendor implementation or "oh, didn't think of that".
> 
> That's why I don't like people using the word "filtering", because this not 
> working isn't always intentional. "Filtering" implies intent.
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[Mikael Abrahamsson]

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

Up to now, every time I’ve seen this problem was just related to ICMPv6 
being filtered, as many folks do in IPv4 …


I know several cases where the problem was that the load balancer didn't 
forward the ICMPv6 PTB to the correct host and didn't handle it itself. No 
filtering, just bad vendor implementation or "oh, didn't think of that".


That's why I don't like people using the word "filtering", because this 
not working isn't always intentional. "Filtering" implies intent.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[JORDI PALET MARTINEZ]

Right I missed that too, and now reading the article instead of “quick review”, 
I think the solution is there:

https://github.com/cloudflare/pmtud


Saludos,
Jordi


-Mensaje original-
De:  en nombre de 
Paul Stewart 
Responder a: 
Fecha: viernes, 14 de octubre de 2016, 14:09
Para: Mikael Abrahamsson 
CC: , JORDI PALET MARTINEZ 

Asunto: Re: contact with One & One ?

You are correct - i misspoke on that … the reported issue from some 
visitors is site doesn’t load.  Sorry for the confusion - need more caffeine 
this morning :)

> On Oct 14, 2016, at 8:05 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 14 Oct 2016, Paul Stewart wrote:
> 
>> honestly we’ve never fixed it.  it works for lots of customer/visitors 
but breaks for others (and they fail back to IPv4) - we thought it was
> 
> Errr, how does this fallback work? I am not aware of any such mechanism.
> 
> Happy Eyeballs is done when the SYN+ACK gets back.
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se






**
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the use of the 
individual(s) named above. If you are not the intended recipient be aware that 
any disclosure, copying, distribution or use of the contents of this 
information, including attached files, is prohibited.

Re: contact with One & One ?

[Paul Stewart]

You are correct - i misspoke on that … the reported issue from some visitors is 
site doesn’t load.  Sorry for the confusion - need more caffeine this morning :)

> On Oct 14, 2016, at 8:05 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 14 Oct 2016, Paul Stewart wrote:
> 
>> honestly we’ve never fixed it.  it works for lots of customer/visitors but 
>> breaks for others (and they fail back to IPv4) - we thought it was
> 
> Errr, how does this fallback work? I am not aware of any such mechanism.
> 
> Happy Eyeballs is done when the SYN+ACK gets back.
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[Paul Stewart]

Thanks .. I meant to include link to that article - appreciate you doing so :)

We don’t filter ICMPv6 on those servers and the problem we are pretty confident 
is ECMP related (as per what we learned from the Cloudflare blog) … need to set 
up some time to look deeper though as internally and on our own servers we’ve 
never been able to replicate the issue

Cheers,
Paul

> On Oct 14, 2016, at 8:02 AM, JORDI PALET MARTINEZ 
>  wrote:
> 
> The issue here is that customers (the ones that browse the broken web sites), 
> don’t know about MTU, ICMP, etc.
> 
> So I guess is in your side as the “provider” of the content, who is the 
> interested party in making sure it works for “all” your possible customers.
> 
> Up to now, every time I’ve seen this problem was just related to ICMPv6 being 
> filtered, as many folks do in IPv4 …
> 
> 
> By the way, interesting article, I didn’t read it before:
> https://blog.cloudflare.com/path-mtu-discovery-in-practice/
> 
> 
> Saludos,
> Jordi
> 
> 
> -Mensaje original-
> De:  en nombre 
> de Paul Stewart 
> Responder a: 
> Fecha: viernes, 14 de octubre de 2016, 13:52
> Para: Mikael Abrahamsson 
> CC: , JORDI PALET MARTINEZ 
> 
> Asunto: Re: contact with One & One ?
> 
>At $$$job we run quite a bit of dual stack towards customers as an ISP 
> (mainly PPPoE) - our own public website fails the PTB test and quite honestly 
> we’ve never fixed it.  it works for lots of customer/visitors but breaks for 
> others (and they fail back to IPv4) - we thought it was only external tunnel 
> visitors but have found out otherwise… never fully understood what was going 
> on and I keep meaning to look at it .. 
> 
>NGINX front ends load balanced via anycast … pretty standard Ubuntu 
> 16.04LTS setup on the server side.  From what I’ve read it seems to be an 
> ECMP related problem like what CloudFlare published a blog about … 
> 
>Paul
> 
>> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson  wrote:
>> 
>> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
>> 
>>> I think is time to retire happy-eye-balls, it is the only way the people 
>>> will react to those issues!
>> 
>> Happy eyeballs doesn't solve PMTU blackhole.
>> 
>> So this is actually customer breakage occuring, but I imagine lots of ISPs 
>> are actually doing MSS re-write and/or announcing lower than 1500 MTU on the 
>> customer LAN, so even if a customer has PPPoE with 1492 MTU, they still 
>> won't see this problem.
>> 
>> I have seen swedish authorities websites with same "won't-respond-to-PTB", 
>> no answer there either to fault reports.
>> 
>> -- 
>> Mikael Abrahamssonemail: swm...@swm.pp.se
> 
> 
> 
> 
> 
> 
> **
> IPv4 is over
> Are you ready for the new Internet ?
> http://www.consulintel.es
> The IPv6 Company
> 
> This electronic message contains information which may be privileged or 
> confidential. The information is intended to be for the use of the 
> individual(s) named above. If you are not the intended recipient be aware 
> that any disclosure, copying, distribution or use of the contents of this 
> information, including attached files, is prohibited.
> 
> 
> 

Re: contact with One & One ?

[Mikael Abrahamsson]

On Fri, 14 Oct 2016, Paul Stewart wrote:

honestly we’ve never fixed it.  it works for lots of customer/visitors 
but breaks for others (and they fail back to IPv4) - we thought it was


Errr, how does this fallback work? I am not aware of any such mechanism.

Happy Eyeballs is done when the SYN+ACK gets back.

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[JORDI PALET MARTINEZ]

The issue here is that customers (the ones that browse the broken web sites), 
don’t know about MTU, ICMP, etc.

So I guess is in your side as the “provider” of the content, who is the 
interested party in making sure it works for “all” your possible customers.

Up to now, every time I’ve seen this problem was just related to ICMPv6 being 
filtered, as many folks do in IPv4 …


By the way, interesting article, I didn’t read it before:
https://blog.cloudflare.com/path-mtu-discovery-in-practice/


Saludos,
Jordi


-Mensaje original-
De:  en nombre de 
Paul Stewart 
Responder a: 
Fecha: viernes, 14 de octubre de 2016, 13:52
Para: Mikael Abrahamsson 
CC: , JORDI PALET MARTINEZ 

Asunto: Re: contact with One & One ?

At $$$job we run quite a bit of dual stack towards customers as an ISP 
(mainly PPPoE) - our own public website fails the PTB test and quite honestly 
we’ve never fixed it.  it works for lots of customer/visitors but breaks for 
others (and they fail back to IPv4) - we thought it was only external tunnel 
visitors but have found out otherwise… never fully understood what was going on 
and I keep meaning to look at it .. 

NGINX front ends load balanced via anycast … pretty standard Ubuntu 
16.04LTS setup on the server side.  From what I’ve read it seems to be an ECMP 
related problem like what CloudFlare published a blog about … 

Paul

> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
> 
>> I think is time to retire happy-eye-balls, it is the only way the people 
will react to those issues!
> 
> Happy eyeballs doesn't solve PMTU blackhole.
> 
> So this is actually customer breakage occuring, but I imagine lots of 
ISPs are actually doing MSS re-write and/or announcing lower than 1500 MTU on 
the customer LAN, so even if a customer has PPPoE with 1492 MTU, they still 
won't see this problem.
> 
> I have seen swedish authorities websites with same 
"won't-respond-to-PTB", no answer there either to fault reports.
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se






**
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the use of the 
individual(s) named above. If you are not the intended recipient be aware that 
any disclosure, copying, distribution or use of the contents of this 
information, including attached files, is prohibited.

Re: contact with One & One ?

[Paul Stewart]

At $$$job we run quite a bit of dual stack towards customers as an ISP (mainly 
PPPoE) - our own public website fails the PTB test and quite honestly we’ve 
never fixed it.  it works for lots of customer/visitors but breaks for others 
(and they fail back to IPv4) - we thought it was only external tunnel visitors 
but have found out otherwise… never fully understood what was going on and I 
keep meaning to look at it .. 

NGINX front ends load balanced via anycast … pretty standard Ubuntu 16.04LTS 
setup on the server side.  From what I’ve read it seems to be an ECMP related 
problem like what CloudFlare published a blog about … 

Paul

> On Oct 14, 2016, at 7:45 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:
> 
>> I think is time to retire happy-eye-balls, it is the only way the people 
>> will react to those issues!
> 
> Happy eyeballs doesn't solve PMTU blackhole.
> 
> So this is actually customer breakage occuring, but I imagine lots of ISPs 
> are actually doing MSS re-write and/or announcing lower than 1500 MTU on the 
> customer LAN, so even if a customer has PPPoE with 1492 MTU, they still won't 
> see this problem.
> 
> I have seen swedish authorities websites with same "won't-respond-to-PTB", no 
> answer there either to fault reports.
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[Mikael Abrahamsson]

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

I think is time to retire happy-eye-balls, it is the only way the people 
will react to those issues!


Happy eyeballs doesn't solve PMTU blackhole.

So this is actually customer breakage occuring, but I imagine lots of ISPs 
are actually doing MSS re-write and/or announcing lower than 1500 MTU on 
the customer LAN, so even if a customer has PPPoE with 1492 MTU, they 
still won't see this problem.


I have seen swedish authorities websites with same "won't-respond-to-PTB", 
no answer there either to fault reports.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[JORDI PALET MARTINEZ]

I don’t think it will help …

I’ve got several of their customers, several *months* ago, which opened a 
ticket, and they didn’t get a solution/response …

It may happen that the folks in the ticketing system don’t understand the 
problem or don’t scale it or whatever …

I think is time to retire happy-eye-balls, it is the only way the people will 
react to those issues!

That’s why, the ideal will be to have a direct contact with the team that is 
working on IPv6 …

Saludos,
Jordi


-Mensaje original-
De:  en nombre de 
Kurt Jaeger 
Responder a: 
Fecha: viernes, 14 de octubre de 2016, 12:58
Para: Mikael Abrahamsson 
CC: 
Asunto: Re: contact with One & One ?

Hi!

> > www.corso-kino.de
> 
> Thanks.
> 
> If it helps, point them to this website (still in development/beta):
> 
> https://ipv6alizer.se/
> 
> The result is (verifies what you said):
> 
> INFO:  server-mss 1440, result: pmtud-fail
> ERROR: http://www.corso-kino.de don't listen to PTB

Thanks. It's just around the corner, and I think I can
get them to open a ticket with 1und1 8-}

-- 
p...@opsec.eu+49 171 3101372 4 years to 
go !





**
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the use of the 
individual(s) named above. If you are not the intended recipient be aware that 
any disclosure, copying, distribution or use of the contents of this 
information, including attached files, is prohibited.

Re: contact with One & One ?

[Kurt Jaeger]

Hi!

> > www.corso-kino.de
> 
> Thanks.
> 
> If it helps, point them to this website (still in development/beta):
> 
> https://ipv6alizer.se/
> 
> The result is (verifies what you said):
> 
> INFO:  server-mss 1440, result: pmtud-fail
> ERROR: http://www.corso-kino.de don't listen to PTB

Thanks. It's just around the corner, and I think I can
get them to open a ticket with 1und1 8-}

-- 
p...@opsec.eu+49 171 3101372 4 years to go !

Re: contact with One & One ?

[Mikael Abrahamsson]

On Fri, 14 Oct 2016, Kurt Jaeger wrote:


www.corso-kino.de


Thanks.

If it helps, point them to this website (still in development/beta):

https://ipv6alizer.se/

The result is (verifies what you said):

INFO:  server-mss 1440, result: pmtud-fail
ERROR: http://www.corso-kino.de don't listen to PTB

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: contact with One & One ?

[JORDI PALET MARTINEZ]

There’re tons of them !



Here are a couple of PMTUD test:


tbit from 2001:df0:4:4000::1:115 to 2001:8d8:1001:238f:3cf1:2223:88f2:c80a
server-mss 1440, result: pmtud-fail
app: http, url: http://diskmakerx.com/
[  0.009] TX SYN 64  seq = 0:0
[  0.288] RX SYN/ACK 64  seq = 0:1
[  0.288] TX 60  seq = 1:1
[  0.298] TX233  seq = 1:1(173)
[  0.577] RX 60  seq = 1:174  
[  0.812] RX   1500  seq = 1:174(1440)
[  0.812] RX   1500  seq = 1441:174(1440)  
[  0.812] RX   1500  seq = 2881:174(1440)  
[  0.812] RX 69  seq = 4321:174(9)
[  0.812] RX   1500  seq = 4330:174(1440)  
[  0.812] RX   1500  seq = 5770:174(1440)  
[  0.812] TX PTB   1280  mtu = 1280
[  0.812] RX   1500  seq = 7210:174(1440)  
[  0.816] RX   1500  seq = 8650:174(1440)  
[  0.822] TX 60  seq = 174:1  
[  0.883] RX   1500  seq = 10090:174(1440)
[  0.892] RX   1500  seq = 11530:174(1440)
[  1.651] RX   1500  seq = 1:174(1440)
[  1.651] TX PTB   1280  mtu = 1280
[  3.335] RX   1500  seq = 1:174(1440)
[  3.335] TX PTB   1280  mtu = 1280
[  6.703] RX   1500  seq = 1:174(1440)
[  6.703] TX PTB   1280  mtu = 1280
[ 13.439] RX   1500  seq = 1:174(1440)


tbit from 2001:df0:4:4000::1:115 to 2001:8d8:1000:d2ea:95d2:30d0:d4ad:9357
server-mss 1440, result: pmtud-fail
app: http, url: http://www.legalveritas.es/
[  0.009] TX SYN 64  seq = 0:0
[  0.285] RX SYN/ACK 64  seq = 0:1
[  0.285] TX 60  seq = 1:1
[  0.297] TX238  seq = 1:1(178)
[  0.572] RX 60  seq = 1:179  
[  0.810] RX   1492  seq = 1:179(1432)
[  0.810] TX PTB   1280  mtu = 1280
[  0.825] RX   1500  seq = 1433:179(1440)  
[  0.825] RX   1500  seq = 2873:179(1440)  
[  0.825] RX   1500  seq = 4313:179(1440)  
[  0.825] RX   1500  seq = 5753:179(1440)  
[  0.825] RX   1500  seq = 7193:179(1440)  
[  0.825] RX   1500  seq = 8633:179(1440)  
[  0.825] RX   1500  seq = 10073:179(1440)
[  0.825] RX   1500  seq = 11513:179(1440)
[  0.825] RX   1500  seq = 12953:179(1440)
[  1.636] RX   1492  seq = 1:179(1432)
[  1.636] TX PTB   1280  mtu = 1280
[  3.296] RX   1492  seq = 1:179(1432)
[  3.296] TX PTB   1280  mtu = 1280
[  6.616] RX   1492  seq = 1:179(1432)
[  6.616] TX PTB   1280  mtu = 1280
[ 13.248] RX   1492  seq = 1:179(1432)




Saludos,
Jordi


-Mensaje original-
De:  en nombre de 
Mikael Abrahamsson 
Organización: People's Front Against WWW
Responder a: 
Fecha: viernes, 14 de octubre de 2016, 12:32
Para: JORDI PALET MARTINEZ 
CC: 
Asunto: Re: contact with One & One ?

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:

> Hi,
>
> I’ve discovered, several months ago already, that all the 1&1 web sites 
> with IPv6 support enabled are broken, because they filter PMTUD, so any 
> residential customer with has a reduced MTU because PPP or any other 
> encapsulation/tunnel, etc., is not reaching them.

Do you have an example of a website they host that I can test against?

-- 
Mikael Abrahamssonemail: swm...@swm.pp.se



**
IPv4 is over
Are you ready for the new Internet ?
http://www.consulintel.es
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the use of the 
individual(s) named above. If you are not the intended recipient be aware that 
any disclosure, copying, distribution or use of the contents of this 
information, including attached files, is prohibited.

Re: contact with One & One ?

[Torbjörn Eklöv]

14 okt. 2016 kl. 12:38 skrev Kurt Jaeger 
mailto:ipv6-...@c0mplx.org>>:

Hi!

I've discovered, several months ago already, that all the 1&1 web sites
with IPv6 support enabled are broken, because they filter PMTUD, so any
residential customer with has a reduced MTU because PPP or any other
encapsulation/tunnel, etc., is not reaching them.

Do you have an example of a website they host that I can test against?

www.corso-kino.de

Yes, it fails PTB test

https://ipv6alizer.se?address=http://www.corso-kino.de

/Tobbe



--
p...@opsec.eu+49 171 3101372  
   4 years to go !

Re: contact with One & One ?

[Kurt Jaeger]

Hi!

> > I've discovered, several months ago already, that all the 1&1 web sites 
> > with IPv6 support enabled are broken, because they filter PMTUD, so any 
> > residential customer with has a reduced MTU because PPP or any other 
> > encapsulation/tunnel, etc., is not reaching them.
> 
> Do you have an example of a website they host that I can test against?

www.corso-kino.de

-- 
p...@opsec.eu+49 171 3101372 4 years to go !

Re: contact with One & One ?

[Mikael Abrahamsson]

On Fri, 14 Oct 2016, JORDI PALET MARTINEZ wrote:


Hi,

I’ve discovered, several months ago already, that all the 1&1 web sites 
with IPv6 support enabled are broken, because they filter PMTUD, so any 
residential customer with has a reduced MTU because PPP or any other 
encapsulation/tunnel, etc., is not reaching them.


Do you have an example of a website they host that I can test against?

--
Mikael Abrahamssonemail: swm...@swm.pp.se